In the previous post, we have introduced Linux observability and how eBPF technology contributes to observability. Finally, we introduced BCC as a toolset. So, in this post we will focus on how to use these tools.
It should be noted that we will be using RHEL8 as the runtime environment for the BCC tools. In fact, the RHEL operating system has introduced eBPF as well as the BCC toolset since RHEL 7.6.
BCC Installation
BCC is very easy to install, just run the following command after configuring the yum source:
# yum install bcc-toolsIn fact, the bcc-tools package is included in the ISO file, so you can install bcc-tools in an offline environment by configuring the local yum source.
Once installed, all the executable tools are located in the /usr/share/bcc/tools/ directory.
All bcc-tools are written in python for the user space and in C for the kernel space, so the source code is easy to view with a text tool. We can easily find out what options each tool has and what they mean. We can also get a deeper understanding of their code logic.
CPU Subsystem
When encountering problems such as computational performance bottlenecks, or high cpu utilization, you can use the following tools:
execsnoop
This is a tracing tool. It can dynamically output the spawning of new processes in real time on a system wide basis by tracing the execve() syscall event. We can use this tool if we want to know if the cpu usage is high because of a large number of new application processes being spawned on the system.
[root@rhel8 tools]# ./execsnoop -xTU
TIME UID PCOMM PID PPID RET ARGS
01:10:52 0 sshd 1279 775 0 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cb -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2- -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-grou -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-h -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v0 -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-ce -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rs -R
01:10:55 0 unix_chkpwd 1281 1279 0 /usr/sbin/unix_chkpwd root nonull
01:10:55 0 unix_chkpwd 1282 1279 0 /usr/sbin/unix_chkpwd root chkexpiry
01:10:56 0 bash 1284 1283 0 /bin/bash
01:10:56 0 id 1286 1285 0 /usr/bin/id -un
01:10:56 0 hostname 1288 1287 0 /usr/bin/hostnameThe -x option outputs the records including when execve() failed. The -T option outputs the time of the record. The -U option outputs the UID of the process.
The meaning of most of the columns in the above output is already obvious and will not be explained again. But the RET column refers to the return value of execve(), which we can use to figure out what went wrong; for example, when the nproc limit of resource limit is reached, we can make a judgment based on the return value of this output.
Before this tool, we usually used strace and ltrace to trace a process’s syscall event, perf-trace and audit to trace system-wide syscall events, and ps/top to see a sampling of process list metrics.
exitsnoop
This is a tracing tool. It gets key information about the lifetime of a process or thread, as well as exit codes, signal number, etc. by tracing kernel tracepoint sched_process_exit, or in other words, and more essentially, by tracing do_exit() kernel function.
[root@rhel8 tools]# ./exitsnoop -t --per-thread
TIME-CST PCOMM PID PPID TID AGE(s) EXIT_CODE
03:13:45.830 ps 1726 1404 1726 0.13 0
03:13:45.831 grep 1727 1404 1727 0.12 0
03:13:54.106 sleep 1725 1284 1725 15.03 signal 15 (TERM)
03:14:15.520 systemctl 1730 1481 1730 0.04 0
03:14:15.564 systemctl 1731 1481 1731 0.03 0
03:14:15.575 tuned 1481 1 1609 2172.05 0
03:14:16.587 tuned 1481 1 1610 2172.92 0
03:14:16.815 tuned 1481 1 1606 2173.43 0
03:14:16.818 tuned 1481 1 1611 2173.13 0
03:14:16.890 tuned 1481 1 1481 2175.44 0
03:14:16.945 systemd-cgroups 1733 2 1733 0.05 0
03:14:17.643 ldconfig 1737 1734 1737 0.01 0
03:14:17.682 id 1739 1738 1739 0.01 0
03:14:17.691 which 1741 1740 1741 0.00 0
03:14:17.694 virt-what 1740 1738 1740 0.01 0
03:14:17.701 virt-what-cpuid 1742 1738 1742 0.01 0
03:14:17.710 dmidecode 1744 1743 1744 0.00 0
03:14:17.714 virt-what 1743 1738 1743 0.01 0
03:14:17.729 uname 1746 1745 1746 0.01 0
03:14:17.735 sed 1748 1745 1748 0.01 0
03:14:17.736 sed 1747 1745 1747 0.01 0
03:14:17.737 virt-what 1745 1738 1745 0.02 0
03:14:17.744 virt-what 1749 1738 1749 0.00 0
03:14:17.750 grep 1750 1738 1750 0.01 code 1
03:14:17.760 virt-what 1751 1738 1751 0.01 0The -t option indicates the exit timestamp. The — per-thread outputs the exit record of the threads. If we only want to output records where the exit code is non-zero, i.e. a failure, we can use the -x option.
As an extra note, the AGE(s) column indicates the lifetime of the process or thread, not the on-cpu running time.
In the above output example, you can see the exit of tuned’s multiple threads, and you can also see the sleep process being sent signal 15 and exiting.
The exit code of a process or thread, as well as the received signal, is the very important information for auditing and troubleshooting.
Conclusion
Today we have covered the tools execsnoop and exitsnoop in the cpu subsystem, and I will cover the use of more bcc tools in the series in the future.
