Linux Performance: Existing methods and tools
Table of Contents
- 1 Abstract
- 2 Introduction
- 3 A Primer of existing tools
- 3.1 ftrace:
- 3.2 perf:
- 3.3 oprofile:
- 3.4 lttng:
- 3.5 flamegraphs
- 3.6 coldgraphs
- 3.7 differential graphs
- 3.8 systrace
- 3.9 trace-cmd
- 3.10 atrace
- 3.11 strace
- 3.12 dtrace
- 3.13 sar
- 3.14 iostat
- 3.15 vmstat
- 3.16 iosnoop
- 3.17 sysfs - a treasure house
- 3.18 procfs - another treasure house
- 3.19 ltrace
- 3.20 systemtap
- 3.21 ktap
- 3.22 mpstat
- 3.23 sysstat
- 3.24 collectl
- 3.25 top
- 3.26 htop
- 3.27 ps
- 3.28 free
- 3.29 pidstat
- 3.30 sar
- 4
1 Abstract
A number of tools and methods exist to measure performance on Linux with varying capabilities and applications. From user space applications to kernel or even bare metals hardware evaluation. We propose to explore, to varying depths, a broad array of tools available. In particular, we go into the details of a handful of a few tools listed below. This paper lists the applications of each of these tools and their relevance to different problems at hand.
2 Introduction
In this section, we provide a brief motivation for using such tools in a modern Linux system and attempt to convince the reader that such tools can tremendously help reduce the overhead in debugging issues with a system, tuning a system to maximize performance etc.
3 A Primer of existing tools
3.1 ftrace:
Ftrace is a kernel functionality implemented by Steve Rostedt.
- How is ftrace implemented?
Add a detailed paragraph about the implementation specifics of the ftrace implementation.- ftrace is a kernel tracing functionality that allows to trace the order of execution of functions along with information such as the pid whose context the current function is being called from, timestamp of the entry etc.
- There are similar functionalities in the user land for debugging via the strace. ftrace provides an ordered tracing of the functions called in the kernel and can log them in the trace buffer. As long as the support for tracing exists in kernel traces can be logged and disabled dynamically. This allows the user to log traces on events. Events are nothing but individual traces.
- ftrace provides an invaluable insight into a running system by capturing functions/events in the order of their execution.
3.2 perf:
- Perf feature was added to the mainline kernel around the 2.6.31 version. This was added to track the values of hardware counters maintained by the processor to keep track of events such as cache misses (instruction and data), branch mispredictions etc. It was also possible to track software events such as page faults and such. Perf started as a simple low-overhead functionality to keep track of Linux performance. With time, several events were added that perf is able to keep track of.
- Perf utility collects various counters from the hardware, including CPU specific events such as the cache hits, cache misses, branch mis-predictions, and other similar events.
- Perf top provides a nice interface like the top functionality of the Linux system. Especially if the kptrrestrict is set to 0 then the perf utility grabs the kernel's symbols and dereferences the addresses in run time to show the functions that are currently being executed. So you have a flashing screen that shows the list of functions currently being executed on the system with some statistics such as the number of samples, time spent in the function etc.
- list:
- top:
- record:
- report:
- script:
- stat:
- -rNNN:
- hardware counters: