QEMU Networking

Summary:

{/etc/qemu-ifup}
#!/bin/sh
#switch=$(/sbin/ip route list | awk '/^default / { print $5 }') -- error/remove
switch=virbr0 -- correct
/sbin/ifconfig $1 0.0.0.0 up -- keep
/usr/sbin/brctl addif ${switch} $1 -- keep

{/etc/qemu-ifdown}
#!/bin/sh
#switch=$(/sbin/ip route list | awk '/^default / { print $5 }') -- error/remove
swtich=virbr0 -- correct
/usr/sbin/brctl delif $switch $1 -- keep
/sbin/ifconfig $1 0.0.0.0 down -- keep

add these parameters to qemu: -net nic,vlan=0 -net tap,vlan=0

Previous Writing:
Write 2 network scripts as below: ( don't forget the chmod +x on each file. )

[root@sitedesign ~]# cat /etc/qemu-ifdown
#!/bin/sh
/sbin/ifconfig virbr0 down
/sbin/ifconfig down $1
/sbin/ifup eth0

[root@sitedesign ~]# cat /etc/qemu-ifup
#!/bin/sh
/sbin/ifconfig $1 0.0.0.0 promisc up
/usr/sbin/brctl addif virbr0 $1

Then to start the VM:

as root:

#~> qemu-kvm -net nic,vlan=0 -net,tap,vlan=0 -hda winxp.img -hdb winxp_disk2.img -usb -usbdevice tablet -localtime -daemonize

That should start you up with a connection to the default virbr that gets made by fedora at boot time.

(original link)

using CSCOPE

1. build database: cscope -b -q -k -R
2. invoke: cscope -d ... does not create the database again

3: invoke command line:
      find all functions that calls bad_page()
          cscope -d -3 bad_page -L
      find all functions that bad_page() calls
          cscope -d -2 bad_page -L

exit: ctrl_d

Using QEMU for Direct Debugging (INCOMPLETE)

Install kvm-pxe first

kernel build: two places
     inside the vm
     in in the host

make initrd inside vm, copy it to host in the qemu disk directory
copy vmlinux, sysmap and config into qemu disk directory

then run the kernel:
qemu -smp 4 -m 384 -hda ubswapper.img -kernel vmlinuz-2.6.38.2 -initrd initrd.img-2.6.38.2 "console=ttyS0,115200n8 root=/dev/sda1 text" -serial stdio > trace

Tracing In Brief

cd /sys/kernel/debug/tracing

# viewing functions
functions are listed in available_filter_function
grep shrink available_filter_function

# viewing tracers
cat available_tracers
function_graph, function are important

# setting function and graph
root@musfiq-Dell-DM051:/sys/kernel/debug/tracing# echo shrink_zone > set_ftrace_filter
root@musfiq-Dell-DM051:/sys/kernel/debug/tracing# echo shrink_zone > set_graph_function

# set tracer
root@musfiq-Dell-DM051:/sys/kernel/debug/tracing# echo function_graph > current_tracer
root@musfiq-Dell-DM051:/sys/kernel/debug/tracing# echo 1 > tracing_on

root@musfiq-Dell-DM051:/sys/kernel/debug/tracing# echo 0 > tracing_on

Console Over Serial

in kvm, add another serial connection
in kernel boot parameters, add: console=ttyS1
now, all kernel messages will be redirected to ttyS1
if kgdboc=ttyS0, then run gdb on that port.
that's it!

so, now you can debug kernel on one serial port while get the kernel messages on another serial port.

Remote Kernel Compilation: Custom Installation Path

make a directory myinstall

#> make modules_install INSTALL_MOD_PATH=myinstall
#> make install INSTALL_PATH=myinstall
#> make headers_install INSTALL_HDR_PATH=myinstall

everything will be neatly under myinstall directory :-)

now make the ramdisk by
mkinitramfs -o initrd.img-3.1.6 -r (myinstall) -v 3.1.6

on the remote machine,
copy myinstall/lib into /lib and fix the build and source symbolic links to actual source code directory. if there is no source directory, then only copy the headers. into /lib/modules/<version>/build/include...

/lib/modules/<version>/kernel contains all the .ko files
/lib/modules/<version>/build and source points to the headers, or you can keep the actual source files here... all it cares is the include directory inside build :-)

copy myinstall/config, map and vmlinuz into /boot and configure grub.

that's how we transfer a kernel from remote machine to target machine. do not forget to copy the .config file from target to remote before building kernel on remote.

Breakpoints in modules

I am working in modframe

Inside Host, connect gdb via pts to VM. 

Inside VM,
put the module in /root/Desktop/modexperiments/mf
make the module with all debugging info
    add this line in Makefile... EXTRA_CFLAGS += -ggdb3
modframe.ko will be created

insert the module. then it will have a /sys/modules/modframe/sections/.text entry.
track that 0xf8031000
or try this
grep function_name /proc/kallsyms ... track that address

Inside Host,

copy mf directory into /root/Desktop/modexperiments/ ... so that the files are there.

Now we will add breakpoints. 

Inside VM,

echo g > /proc/sysrq-trigger
This will hand transfer to gdb in host

Inside Host,
fire up gdb in host,
  # add-symbol-file modframe.ko 0x########
               -s .data 0x########
               -s .bss   0x########
you can view listing, or put a break point b *0xf8031000

Inside VM,
or remove the modules... that will hit the breakpoint :)
Other interesting utilities
objdump is another utility (use -h flag)

 $ nm modframe.ko -- view symbols 

or

 $ objdump -t modframe.ko            (all sections)
 $ objdump -t -j data modframe.ko    (data section)
 $ objdump -t -j bss modframe.ko     (BSS section)  

Ref:

https://www.linux.com/learn/linux-career-center/33991-the-kernel-newbie-corner-

kernel-and-module-debugging-with-gdb 

http://linux-hacks.blogspot.com/2009/07/using-gdb-for-debugging-kernel-modules.html

Commands Saturday Oct 15 2011

viewing information
info program, info sttck, info registers, info break

setting a breakpoint
break <function name>
break <line number>

clearing breakpoint
clear <function_name>
delete <breakpoint number>

viewing paramters
print <name>

Compile Kernel in a Remote Machine

compile remote kernel
http://web.archive.org/web/20000823075731/http://www.mindspring.com/~rgolan/lula/moving_kernels.html

preparing kvm ubuntu

install ubuntu mini in kvm VM.

then install ssh, gcc, vim, libncurses5-dev

then build and install the kernel

links link links

http://people.freebsd.org/~jhb/papers/bsdcan/2008/article/article.html
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html
http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html

loading kernel in gdb

compile kernel with everything about tracing and kernel hacking (DYNAMIC, FTRACE, OPTIMIZE)

then vmlinux file is copied from remote to host

now start gdb/ddd
    > file vmlinux
    > target remote /dev/pts/X

that's it

Sysrq

echo g > /proc/sysrq-trigger

enable/disable sysrq

echo 0 > /proc/sys/kernel/sysrq

echo 1 > /proc/sys/kernel/sysrq 

Finding a symbol
grep alloc_pages /proc/kallsyms -- more symbols because of the loadable modules
or grep alloc_pages /boot/System.map-XXX -- only built-in symbols


kgdb ref:
http://elinux.org/Debugging_The_Linux_Kernel_Using_Gdb

notes on tracepoints

TRACE_EVENT macro declares event types and handler function inside kernel. by default, event is disabled. once enabled, it emits some info into the ftrace ring buffer. they appear in the 'trace' file.

scripts to format function traces

There is a nice script in kernel scripts/draw_functrace.py

it can draw function traces without using graph_function type built-in tracing.

# echo function > /sys/kernel/debug/tracing/current_tracer
$ cat /sys/kernel/debug/tracing/trace_pipe > ~/raw_trace_func
$ scripts/draw_functrace.py < raw_trace_func > draw_functrace

Kernel Tracing (Notes from LWN)

Tracepoints
http://lwn.net/Articles/291091/
http://lwn.net/Articles/330402/
http://lwn.net/Articles/379903/
http://lwn.net/Articles/381064/
http://lwn.net/Articles/383362/
http://lwn.net/Articles/346470/
also look at the samples in source

Config options

CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE
CONFIG_KALLSYMS

One
of the most powerful tracers of Ftrace is the function tracer. It uses
the -pg option of gcc to have every function in the kernel call a
special function "mcount()". When CONFIG_DYNAMIC_FTRACE is configured,
the call is converted to a NOP at boot time to keep the system running
at 100% performance. During compilation the mcount() call-sites are
recorded. That list is used at boot time to convert those sites to NOPs.
Since NOPs are pretty useless for tracing, the list is saved to convert
the call-sites back into trace calls when the function (or function
graph) tracer is enabled.


- task -
make your own tracepoint/event
trace it with the perf command... should be really easy to accomplish.
- end -

- available filers -
[tracing]# cat available_filter_functions | head -8
[tracing]# wc -l available_filter_functions
[tracing]# grep sched available_filter_functions > set_ftrace_filter
[tracing]# echo 'set*' > set_ftrace_filter
[tracing]# echo ':mod:tg3' > set_ftrace_filter
[tracing]# echo '!*lock*' >> set_ftrace_filter -- The '!' symbol will remove functions listed in the filter file.

Important Links on Kernel Tracing/Debugging

setup KVM to use console mode debugging (remotely)
    (link 1)

get hands dirty on kernel tracing (work on more debugging stuff, function tracing)
     kgtp kprobe tracepoints trave_events kgdb
     Marging KDB and KGDB (link)
     kgdb kdb wiki (link)
     kdb (link)
     kernel debugging (link)

     LJ - Debugging Memory on Linux (link)
     LJ - Kernel Debugging Techniques (link) [from book Embedded Linux Primer]
     LJ - Kprobe (link)
     LJ - Remote debugging of LKM with KGDB (link)

    Collecting trace data from using klog (link)
    Build kernel faster (link) (patchworks)

Setting up KVM for remote debugging of guest

Change in GRUB2
copy relevant entry from /boot/grub/grub.cfg into /etc/grub.d/40_custom
do grub-mkconfig -o /boot/grub/grub.cfg
check which one to make default (initial was 0)
change relevant entry in /etc/default/grub to point to the default
do grub-mkconfig -o /boot/grub/grub.cfg
reboot, hold 'shift' or hit some 'arrow' keys to see the boot menu in kvm... for some reason it goes off very fast.

Kernel Parameters for Serial Console
... $console kgdbwait kgdboc=ttyS0,115200n8 selinux=0
i also tried to redirect console using ... console=ttyS0,115200n8
so, guest inside KVM uses /dev/ttyS0 for regular serial port.
outside KVM, kvm pipes serial port into /dev/pts/X (see host config)
using gdb or gtkterm etc and using port /dev/pts/X, it is easy to capture/push data stream though virtual serial port in host... easy :)

Compiling Kernel outside KVM
do kernel compilation as usual
copy configXX, System-mapXX and vmliuzXX into guest /boot
also copy /lib/modules/XX directory from host into guest /lib/modules/
inside guest, do mkinitramfs   -o initrd-img-XX    XX
update grub as earlier
reboot, select new kernel