notes on tracepoints

TRACE_EVENT macro declares event types and handler function inside kernel. by default, event is disabled. once enabled, it emits some info into the ftrace ring buffer. they appear in the 'trace' file.

scripts to format function traces

There is a nice script in kernel scripts/draw_functrace.py

it can draw function traces without using graph_function type built-in tracing.

# echo function > /sys/kernel/debug/tracing/current_tracer
$ cat /sys/kernel/debug/tracing/trace_pipe > ~/raw_trace_func
$ scripts/draw_functrace.py < raw_trace_func > draw_functrace

Kernel Tracing (Notes from LWN)

Tracepoints
http://lwn.net/Articles/291091/
http://lwn.net/Articles/330402/
http://lwn.net/Articles/379903/
http://lwn.net/Articles/381064/
http://lwn.net/Articles/383362/
http://lwn.net/Articles/346470/
also look at the samples in source

Config options

CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE
CONFIG_KALLSYMS

One
of the most powerful tracers of Ftrace is the function tracer. It uses
the -pg option of gcc to have every function in the kernel call a
special function "mcount()". When CONFIG_DYNAMIC_FTRACE is configured,
the call is converted to a NOP at boot time to keep the system running
at 100% performance. During compilation the mcount() call-sites are
recorded. That list is used at boot time to convert those sites to NOPs.
Since NOPs are pretty useless for tracing, the list is saved to convert
the call-sites back into trace calls when the function (or function
graph) tracer is enabled.


- task -
make your own tracepoint/event
trace it with the perf command... should be really easy to accomplish.
- end -

- available filers -
[tracing]# cat available_filter_functions | head -8
[tracing]# wc -l available_filter_functions
[tracing]# grep sched available_filter_functions > set_ftrace_filter
[tracing]# echo 'set*' > set_ftrace_filter
[tracing]# echo ':mod:tg3' > set_ftrace_filter
[tracing]# echo '!*lock*' >> set_ftrace_filter -- The '!' symbol will remove functions listed in the filter file.

Important Links on Kernel Tracing/Debugging

setup KVM to use console mode debugging (remotely)
    (link 1)

get hands dirty on kernel tracing (work on more debugging stuff, function tracing)
     kgtp kprobe tracepoints trave_events kgdb
     Marging KDB and KGDB (link)
     kgdb kdb wiki (link)
     kdb (link)
     kernel debugging (link)

     LJ - Debugging Memory on Linux (link)
     LJ - Kernel Debugging Techniques (link) [from book Embedded Linux Primer]
     LJ - Kprobe (link)
     LJ - Remote debugging of LKM with KGDB (link)

    Collecting trace data from using klog (link)
    Build kernel faster (link) (patchworks)

Setting up KVM for remote debugging of guest

Change in GRUB2
copy relevant entry from /boot/grub/grub.cfg into /etc/grub.d/40_custom
do grub-mkconfig -o /boot/grub/grub.cfg
check which one to make default (initial was 0)
change relevant entry in /etc/default/grub to point to the default
do grub-mkconfig -o /boot/grub/grub.cfg
reboot, hold 'shift' or hit some 'arrow' keys to see the boot menu in kvm... for some reason it goes off very fast.

Kernel Parameters for Serial Console
... $console kgdbwait kgdboc=ttyS0,115200n8 selinux=0
i also tried to redirect console using ... console=ttyS0,115200n8
so, guest inside KVM uses /dev/ttyS0 for regular serial port.
outside KVM, kvm pipes serial port into /dev/pts/X (see host config)
using gdb or gtkterm etc and using port /dev/pts/X, it is easy to capture/push data stream though virtual serial port in host... easy :)

Compiling Kernel outside KVM
do kernel compilation as usual
copy configXX, System-mapXX and vmliuzXX into guest /boot
also copy /lib/modules/XX directory from host into guest /lib/modules/
inside guest, do mkinitramfs   -o initrd-img-XX    XX
update grub as earlier
reboot, select new kernel