A simple introduction to linux perf tool. For a comprehensive description please read this post.

Install(Ubuntu)

sudo apt install linux-tools-common

Run perf command, then you can be required to install additional packages:

WARNING: perf not found for kernel 4.15.0-36

You may need to install the following packages for this specific kernel:
linux-tools-4.15.0-36-generic
linux-cloud-tools-4.15.0-36-generic

You may also want to install one of the following packages to keep up to date:
linux-tools-generic
linux-cloud-tools-generic

sudo apt install linux-tools-4.15.0-36-generic

Basic Usage

demo program

#include <iostream>
#include <thread>

using namespace std;

#define MAX_LOOP (20 * 10000 * 10000)

void func1()
{
    for(int i = 0; i < MAX_LOOP; i++);
}

void func2()
{
    for(int i = 0; i < MAX_LOOP; i++);
}

int main(int argc, char **argv)
{
    thread t1(func1);
    thread t2(func1);
    thread t3(func2);

    t1.join();
    t2.join();
    t3.join();
    return 0;
}

Assume we compile this demo as demo.

record data

Priviliged permissions may be needed to record data.

• start a program with perf command and record data
  perf record -F 1000 ./demo
• attach a existing process(pid 1828) and record data
  perf record -F 1000 -p 1828

You will get a file named perf.data.

Some useful options:

• -g
  Enables call-graph (stack chain/backtrace) recording.
• -s or --stat
  Record per-thread event counts. Use it with perf report -T to see the values.

generate report

Run perf report to inspect collected data interactively. Press ? key to get help.
Use --demangle option to get demangled symbols but this doesn’t work on my Ubuntu 18.04. Instead try perf report --stdio | c++filt to get demangled results. The drawback is that you lose the convenience of viewing the collected data interactively.

generate flame graph

To understand the collected perf data more easily, it’s best to generate a flame graph to help you find the cpu load distribution among lots’of functions.

git clone https://github.com/brendangregg/FlameGraph  # or download it from github
cd FlameGraph
perf record -F 99 -a -g -- sleep 60
perf script | ./stackcollapse-perf.pl > out.perf-folded
./flamegraph.pl out.perf-folded > perf-kernel.svg

Advanced Usage

Again, please read brendangregg’s post to learn details of linux perf tool.

• Command perf record -e <event type> can trace various kinds of events.
• Or use perf stat -e <event type> to just get event counters.
• Use perf probe --add <user defined tracepoint> to add your customized tracepoints.