The Linux Kernel
Blueprints for World Domination
 
Gary Lawrence Murphy
Principle Consultant
TCI Ontario
garym@linux.ca
Richard Sevenich 
rsevenic@penguin.sirti.org
Tim Waugh
Red Hat UK
twaugh@redhat.com
Juan-Mariano de Goyeneche
jmseyas@dit.upm.es
Manuel J. Petit de Gabriel
mpetit@dit.upm.es
Tom Lees
Debian GNU/Linux
tom@debian.org
Richard West
Georgia Tech
 west@cc.gatech.edu
Christian Poellabauer
Georgia Tech
chris@cc.gatech.edu
Donald Jeff Dionne
uCLinux
jeff@uclinux.org
Michael Durrant
RT Control
 mdurrant@rt-control.com
Kurt Garloff
SuSE Neurnberg, FRG
garloff@suse.de
Alolita Sharma
cWare Consulting
asharma@cwareco.com
Nilayan Sharma
cWare Consulting
nsharma@cwareco.com
Bob Adkins
cWare Consulting
radkins@cwareco.com
Bertrand Lee
Creative Labs
Bertrand_Lee@climail.cli.creaf.com
Edited by
Michael Johnson
William Brown

The Linux Kernel: Blueprints for World Domination
by Gary Lawrence Murphy
by 
Richard Sevenich 
rsevenic@penguin.sirti.org
Tim Waugh
Red Hat UK
twaugh@redhat.com
Juan-Mariano de Goyeneche
jmseyas@dit.upm.es
Manuel J. Petit de Gabriel
mpetit@dit.upm.es
Tom Lees
Debian GNU/Linux
tom@debian.org
Richard West
Georgia Tech
 west@cc.gatech.edu
Christian Poellabauer
Georgia Tech
chris@cc.gatech.edu
Donald Jeff Dionne
uCLinux
jeff@uclinux.org
Michael Durrant
RT Control
 mdurrant@rt-control.com
Kurt Garloff
SuSE Neurnberg, FRG
garloff@suse.de
Alolita Sharma
cWare Consulting
asharma@cwareco.com
Nilayan Sharma
cWare Consulting
nsharma@cwareco.com
Bob Adkins
cWare Consulting
radkins@cwareco.com
Bertrand Lee
Creative Labs
Bertrand_Lee@climail.cli.creaf.com
Edited by Michael Johnson
Edited by William Brown
First Edition Edition 
Published $Date: 2000/05/25 18:30:13 $
Copyright © 2000 by Macmillan Computer Publishing

 
Professional Linux Kernel Programming is released under the Open Publication Licence (OPL); for more information, please visit OpenContent.Org (http://www.opencontent.org/publ/)

Dedication
This book is dedicated to the countless developers, testers, implementors, reviewers and users of the Linux kernel; these are those upon whose shoulders we stand and those who have assured us all the right stand upon those shoulders.

Table of Contents
Preface	000
The Pioneers had it Easy	000
Who is it?	000
Styles and Conventions	000
About the Authors	000
Acknowledgements	000
I. The Linux Kernel	000
1. Linux is not Unix (20)	000
A Brief History of Linux	000
From Multics to the Unix Wars	000
From Task Scheduling to Open Source	000
The Holy Grail of Portability	000
Monoliths and Microkernels	000
Designing to a 'Reasonable Architecture'	000
Open Sources, Open Minds	000
Linux and the GPL	000
Linux Development Model	000
Lines in the Sands over GPL	000
Unix Foundations	000
Hardware Abstraction	000
Filesystems	000
Multitasking	000
Kernel Space vs User Space	000
Virtual Memory	000
System Time	000
New Features in Linux	000
Linux 2.4	000
Planned features for Linux 2.5	000
2. The Reasonable CPU (15)	000
Portability and Reasonable Assumptions	000
From Altair to Pentium	000
The CPU	000
Registers	000
Memory Address Spaces	000
Interrupts and Interrupt Controllers	000
Timers	000
Busses, Controllers and Peripherals	000
3. Tour of the Linux Kernel (15)	000
Torvalds' Design Goals	000
Overview of the Kernel Subsystems	000
Conceptual Architecture	000
Component Collaborations	000
Initialization	000
Interprocess Communications	000
Memory Management	000
File Systems	000
Networking Layer	000
Kernel Library	000
4. The Kernel Sources (15) Richard Sevenich	000
How to Get the Kernel Sources	000
Source Directory Structure	000
Documentation/	000
arch/	000
drivers/	000
fs/	000
include/	000
init/	000
ipc/	000
kernel/	000
lib/	000
mm/	000
modules/	000
net/	000
scripts/	000
Kernel Configuration and Compiling	000
Installation via lilo	000
Where to put the new image	000
Modifying /etc/lilo.conf	000
Modifying /boot/message	000
Backing up the master boot record	000
Running lilo	000
Rebooting	000
5. Tools for Kernel Programmers	000
GCC, EGCS and GNU Assembler	000
Linux 2.4 and egcs	000
Using GNU Assembler	000
Assembler Caveats	000
Intel2GAS Assembler Translator	000
Navigational Aids	000
Code Visualization	000
Rigi Visual Software Understanding Tool	000
PBS: The Portable Bookshelf	000
Source Code Navigation	000
The Linux Source Navigator	000
LXR: The Linux Cross-Reference project	000
Installing the LXR	000
Using LXR	000
In-Editor Source Browsing	000
Global Source Tags	000
Obtaining Global	000
Building Kernel GTags	000
XRef-Speller for EMACS	000
Obtaining XRef	000
Using kernel-doc	000
parport_set_timeout	000
Development Tools	000
Rapid Application Development in Kernel Programming	000
Linux Driver Development Kit	000
Debugging and Kernel Probes	000
Using ptrace	000
Using the GNU Symbolic Debugger (gdb)	000
Built-in Kernel Debugger	000
Kernel Sandboxes	000
User Mode Kernel (UMK)	000
CPU Emulators	000
Useful Utilities	000
System and Process Monitors	000
Analyzing Core Dumps	000
Kernel Profiling	000
Profiling with gprof	000
Linux Trace Toolkit	000
IKD Debugging Tools Patch	000
6. Contributing to the Kernel Project (20)	000
Participating in Linux Development	000
Points of Contact	000
Mailing List Ettiquette	000
Joining the Inner Circle	000
Programming Style	000
Coding Style Guide	000
Comment Blocks	000
Using GNU Emacs	000
Using GNU indent	000
Asserts and Exceptions	000
Other Programming Guidelines	000
Preparing a Patch	000
Bug Reporting	000
Kernel First-Aid	000
Reporting a Bug	000
Tracing from a Kernel OOPS	000
II. Kernel Subsystems	000
7. Speaking Kernel: Global Structures and Methods (45)	000
Kernel DEFINEs	000
Common Data Structures	000
Library Services	000
Kernel Memory Allocation	000
Timers and Sleep States	000
Block Requests	000
Signals and Interrupts	000
/proc Filesystem	000
8. Kernel Initialization	000
Boot Loader Sequence	000
Subsystem Initializations	000
9. Process Scheduler (30)	000
Bottom-Half Handler	000
Task Queues	000
Timers	000
Wait Queues	000
Spin Locks	000
10. Interprocess Communications (IPC) (20)	000
Signals	000
Pipes	000
Sockets	000
SysV IPC	000
Message Queues	000
Semaphores	000
Shared Memory	000
11. Memory Management (20) Thomas Strohm	000
Hardware Support	000
Page Table Abstraction	000
System Initialization	000
The Page Allocator	000
Kernel Memory Allocation	000
kmalloc()	000
The VM Allocator	000
Memory Allocation in Userland	000
Demand Paging	000
The C Library Interface	000
brk()	000
mm*()	000
12. Resource Management	000
Resource Allocation	000
Reporting	000
PCI Card support	000
APM and ACPI support (here?)	000
ISA PnP	000
I2O	000
13. Linux File Systems (30) Manuel J. Petit de Gabriel	000
Ext2fs	000
Virtual File Systems	000
Shared Memory VFS	000
Device Special Files	000
devfsd and the DevFS Option	000
The Venus Interface	000
Executable Files	000
Raw Disk Access (here or own chapter?)	000
14. Networking Layer(35) Alolita Sharma, Bob Adkins, Nilayan Sharma	000
TCP/IP Networking Layers	000
BSD Sockets	000
Packet Socket MMAP	000
INET Socket Layers	000
Creating a Connection	000
Listening on an INET BSD Socket	000
Data Fragmentation	000
Address Resolution (ARP)	000
IP Layer	000
IP Routing	000
Multicast	000
IP Accounting	000
IP Filters and Firewalls	000
Packet Filtering in 2.4	000
Network Address Layer (NAT)	000
Connection Tracking	000
Forwarding via FIB	000
IPv6	000
DECNet/ARCNet (here or in devices?)	000
Kernel HTTPD (here or in devices?)	000
Driver Status Codes	000
III. Kernel Services	000
15. Designing Linux Modules (15) Juan-Mariano de Goyeneche	000
The Generic Module	000
Loadable Modules	000
Kerneld vs KMod	000
Using Boot Parameters and conf.modules	000
Module Debugging	000
16. Device Drivers (35)	000
Common Characteristics	000
Dynamic Configuration	000
Polling vs Interrupt Drivers	000
Direct Memory Access (DMA)	000
Kernel Memory	000
Kernel Interfaces	000
Extending Makefiles and Config.in	000
17. Character Devices (20)	000
Registering Character Devices	000
ioctl Message Handling	000
Predefined Messages	000
Dereferencing Pointers	000
Blocking I/O	000
Asynchronous I/O	000
IRQ Sharing in PCI I/O	000
USB Devices (here or own chapter?)	000
New Character Devices (here or own chapter?)	000
Speech Device Support	000
18. Block Devices (20)	000
Registering Block Devices	000
API Changes in 2.4	000
ATA66	000
ioctl Message Handling	000
Predefined Messages	000
Request Handling	000
Interrupt Handling	000
Block Device Partitions	000
Logical Volumn Manager (here or in FS?)	000
Removable Media	000
19. Network Devices (20) Alolita Sharma, Bob Adkins, Nilayan Sharma	000
Telecommunications Links	000
Protocols	000
Parallel Link Internet Protocol (PLIP)	000
Serial Link Internet Protocol (SLIP)	000
Serial Link Relay Protocol (SLiRP)	000
Point to Point Protocol (PPP)	000
Point to Point over Ethernet (PPPoE)	000
Interfaces	000
The `Null Modem' Connection	000
Analogue Modems (POTS)	000
Integrated Services Digital Network (ISDN)	000
Asynchronous Digital Subscriber Line (ADSL)	000
Asynchronous Transfer Method (ATM)	000
Wired networking	000
Ethernet	000
Token Ring	000
Wireless Networking	000
Amateur Packet Radio (AX.25)	000
WaveLAN	000
Kernel HTTP Support	000
20. PCI (20)	000
PCI Addressing	000
PCI/ISA Bridging	000
I/O and Memory Windows	000
Configuration Cycles	000
Initializing PCI	000
pseudo device driver	000
BIOS	000
PCI Fixup	000
PCI Data Structures	000
Pseudo-Device Sequence	000
Bridge Numbering	000
PCI Bios	000
PCI Fixup	000
21. SCSI Devices (20) Kurt Garloff	000
Overview of the SCSI subsystem	000
scsi-core	000
Upper-level Drivers	000
Lower-level Drivers	000
22. Parallel Ports (20) Tim Waugh	000
Design goals	000
Standard transfer modes	000
Structure	000
Device driver's view	000
Port drivers	000
UDMA	000
The printer driver	000
User-level device drivers	000
23. Video Systems (30)	000
Linux and X11R6	000
Memory-Mapped Video IO	000
Frame Buffers	000
Direct Rendering Manager	000
VGA video modes	000
24. Video for Linux: TV and Radio Interface (15)	000
Overview of Video4Linux	000
Adapting the Kernel for Video	000
Reserving Video Memory	000
Creating the Video Devices	000
Capture Buffers	000
Video Devices	000
Video Capture Interface	000
AM/FM Radio Devices	000
Teletext Interface	000
Raw BI Data	000
Querying Capabilities	000
Reading Images	000
RDS Streams	000
I2C	000
25. Sound Systems (20)	000
Sound Cards	000
Audio Devices	000
Digital Signal Processors (DSP)	000
Synthesizers	000
Mixers	000
OSS-Free	000
SoftOSS Virtual Wave Table	000
ALSA	000
26. SMP (20) Alolita Sharma, Bob Adkins, Nilayan Sharma	000
SMP and Linux	000
SMP in 2.4	000
Multiple IO-APIC	000
Inter Processor Communications	000
Managing the SpinLocks	000
27. Plug'n'Play (15) Tom Lees	000
P'n'P Hardware Implementations	000
ISA	000
BIOS	000
PCI	000
PC Card	000
P'n'P Specifications	000
Kernel 2.3 ISA support	000
Driver Architecture	000
Using the ISA driver	000
/proc Interface	000
User-space tools	000
Supported Hardware	000
PCI Support	000
Sound Cards	000
Network Cards	000
IV. Advanced Topics	000
28. Porting Drivers from NT (35)	000
Paradigm shift from Windows to Unix	000
Port and Memory Access	000
Timing Issues	000
Writing Portable Drivers	000
29. Porting Linux to Other Platforms (35)	000
Overview of Porting Issues	000
Subsystems Issues	000
30. Embedded Linux (30) Donald Jeff Dionne and Michael Durrant	000
31. Real-Time Linux (30) Richard West and Christian Poellabauer	000
Real Time Requirements	000
Linux in Real Time	000
Implementing a Custom Scheduler	000
Alternative Scheduling Algorithms	000
RT-Linux on Intel	000
Related Projects	000
A. Overview of Linux Processors (20)	000
Intel x86	000
PIII Extensions	000
Strong-ARM	000
HP Alpha AXP	000
Motorola 68k	000
Sparc64	000
Itanium (ia64)	000
IBM S/390	000
SuperH	000
B. Kernel Programming Resources (10)	000
Linux Documentation	000
Online Kernel Resources	000
LDP Kernel Guides	000
linux-kernel Mailing List	000
Further Reading	000
Bibliography	000
Index	000

List of Figures
5-1. Screenshot of an LXR page	000

I. The Linux Kernel
Table of Contents
1. Linux is not Unix (20)	000
2. The Reasonable CPU (15)	000
3. Tour of the Linux Kernel (15)	000
4. The Kernel Sources (15) Richard Sevenich	000
5. Tools for Kernel Programmers	000
6. Contributing to the Kernel Project (20)	000

Chapter 4. The Kernel Sources (15) Richard Sevenich
How to Get the Kernel Sources
 The kernel versions follow a numerology consisting of a dotted triad, X.Y.Z, e.g. 2.3.14 or 2.2.12. If the second number (Y) is an odd integer, then the kernel is a development version; whereas, if Y is even, the kernel is a stable version. The usual Linux distributions ship with stable versions. Kernel developers will often use both the developmental and stable series. 
 The average user rarely has motivation for installing a new kernel. A major kernel update with desired new features or the correction of an inconvenient flaw might provide such motivation. However, many programmers who work in kernel related areas need to update routinely. For example, a kernel module developer may have several hard drive partitions occupied by various kernels from both the stable and development trees. Such a person may keep track of current kernel activity by watching http://www.kt.opensrc.org/ and occasionally downloading another kernel. 
 To download kernel sources, aim your browser at http://www.kernel.org/pub/, which allows you to choose a convenient mirror site from which to download. As of this writing one may choose from two possible formats, either *.tar.gz or *.tar.bz2, with file sizes in excess of 10 MB. The latter format requires that you have the bzip2/bunzip2 package on your system. We'll next work through a specific example showing what to do once the desired 'new' kernel has been downloaded. 
 For concreteness, let's assume that you have downloaded the file named linux-2.2.12.tar.gz. Switch to root and move the file to its conventional location, /usr/src. [CAUTION!] Before doing the obvious decompression and unarchiving of the tarball, you must inspect the status of this directory and possibly make some preparatory changes as described in the next paragraph. 
If there is already kernel source in this directory, it will be in a subdirectory named 'linux'. If this exists, create a backup tarball by entering 'tar cvfz linux.old.tgz'. If you have recompiled the existing kernel recently, you should also backup the related configuration material which is in the file, '/usr/src/linux/include/linux/autoconf.h'. Next make room for the new source hierarchy by entering mv linux linux.old. Once that is accomplished, decompress and unarchive the tarball with the command 'tar xvfz linux-2.2.12.tar.gz'. This will create a source tree hierarchy starting from the node, /usr/src/linux. The structure of this hierarchy is the subject of the next section. 
Source Directory Structure
 For version 2.2.Z of the kernel, the first level of directories below the node /usr/src/linux is: 
•	Documentation/
•	arch/
•	drivers/
•	fs/
•	include/
•	init/
•	ipc/
•	kernel/
•	lib/
•	mm/
•	modules/
•	net/
•	scripts/
 We'll now discuss each of these directories in turn. 
Documentation/
This directory contains a wealth of information, much of it specific to this particular kernel. For example, when moving from the 2.0.Z to the 2.2.Z kernel series, one can examine The 'Changes' file to find important details such as the minimal requirements necessary to make the upgrade. Another example - the 2.2.Z series replaces the 'kerneld' functionality by a new daemon, 'kmod'. This is described in the 'kmod.txt' file. Perusing this directory is very worthwhile. 
arch/
One of the strengths of Linux is that it runs on a variety of architectures. This subdirectory contains architecture specific code and a typical set of its subdirectories might be: 
•	alpha/
•	arm/
•	i386/
•	m68k/
•	mips/
•	ppc/
•	sparc/
•	sparc64/
 Clearly, with this number of architectures represented, there is a strong incentive to design and implement portable code in the rest of the kernel. 
drivers/
This contains existing drivers. Among the various subdirectories, perhaps the most central are those for the three important Linux driver classifications: 
•	block/
•	char/
•	net/
 The device driver developer finds this area of the source code full of useful examples e.g. the many drivers contributed by network guru, Donald Becker. 
fs/
This contains file system related source code for the ext2 file system, but also for 'alien' file systems. In fact, looking at the subdirectories of this node, we see the names of the aliens including, but not limited to: 
•	minix/
•	msdos/
•	ntfs/
 It is common in dual boot systems to work in the Linux partition and have reason for mounting the filesystem (e.g. vfat) of the other partition. 
include/
Common includes in the kernel source code are in the style: 
#include< linux/fs.h> or	
#include< asm/io.h>	

 which assume that the pathnames are descended from /usr/src/linux/include. The #include's which begin with 'linux' are intended to be architecture independent; whereas, those starting with 'asm' involve architecture specific headers. The 'asm' subdirectory is a soft link to the appropriate architecture e.g. asm-i386 at the same level. 
As a developer you may find yourself doing significant detective work in these include files, particularly when major kernel revisions occur. For example, in the change from linux kernel series 2.0.Z to 2.2.Z there were substantive changes visited upon device driver developers. Some detective work was necessary for those not paying close attention to a site such as 'kernel traffic' at http://kt.linuxcare.com. In general, a useful kernel detective's tool is 'cross referencing Linux' at http://lxr.linux.no/source/. 
init/
This contains the kernel's 'main.c' which consists of initialization code - beyond the architecture specific code needed to get the system to this point. This is a possible starting point for those who wish to discover the inner workings of the kernel. 
ipc/
Here we find the source code for the kernel's interprocess communication. this would include primitives to support message passing, semaphores, and shared memory. 
kernel/
Most portions of the kernel are architecture independent and are located here. Architecture dependent specifics are elsewhere e.g. arch/i386/kernel. Device drivers are also elsewhere as mentioned earlier. 
lib/
The architecture independent part of the kernel library is located here; whereas the architectural specifics would be somewhere like arch/i386/lib, adjusted for the architecture of interest. 
mm/
The architecture independent part of the memory management code is located here; whereas the architectural specifics would be somewhere like arch/i386/mm; i.e. adjusted for the architecture of interest. 
modules/
When one recompiles the kernel, the actual compilation is preceded by configuration. In configuring the various options, a typical choice is to include the option in the kernel, to exclude it, or to include it as a module. This directory holds the object files for those oprions chosen as modules during the configuration. This would be empty for your system if the kernel was configured with no options chosen as modules. 
net/
This contains the networking code for the kernel, not including the drivers. Among the many subdirectories we find, for example: 
appletalk/ 	
ax25/ 	
decnet/ 	
ethernet/ 	

scripts/
The scripts in this area are used to configure and generate new kernels. 
Kernel Configuration and Compiling
Here we will consider configuring and installing a new kernel. In the /usr/src/linux directory you will find a README file which gives step by step directions for the configuration and compilation. Read the entire file before proceeding. Also, before proceeding, read the file /usr/src/linux/Documentation/Changes. This latter file may require you to do other preparatory work. If not, carefully follow the README directions. After configuration at the compilation step you will have the option of producing an image or a compressed image. For the typical 'workstation' user, the latter is appropriate. It's also wise to subsequently create a floppy disk image via make bzdisk 
When done performing the README instructions, you will have the file /usr/src/linux/arch/i386/boot/bzImage. This is to be installed as your new bootable kernel while retaining the ability to alternatively boot into the old kernel. This is discussed in the next section. 
Installation via lilo
We assume here that your system already boots by using the 'lilo' ( Linux loader ) mechanism which typically writes the boot code in the master boot record. We also assume that you have a boot floppy or rescue disk available in case the installation attempt goes awry - this is hopefully not needed, but could be crucially important. Lilo is both powerful and flexible. We'll not detail all its capabilities here, but rather describe a typical configuration. For in-depth documentation on lilo, see the lilo manual in /usr/doc/lilo. 
Examine the lilo configuration file '/etc/lilo.conf'. You may find something like this: 
        boot=/dev/hda
        verbose=2
        compact
        install=/boot/boot.b
        map=/boot/map
        vga=normal
        prompt
        message=/boot/message

        image=/vmlinuz
        root=/dev/hda2
        label=linux
        read-only                    
 
The idea now is to carry out the following sequence of steps taking care to perform them in order, with special care not to reboot before the last step: 
•	/put the new image in an appropriate place 
•	modify the previous configuration file to allow booting the new or old kernel 
•	modify the '/boot/message' file to give the user an informative boot time prompt 
•	make a backup copy of the master boot record 
•	run 'lilo' from the command line to create the new master boot record, reflecting these changes 
•	reboot 
 In the next subsections, we'll go through each of these steps. 
Where to put the new image
The new image is in /usr/src/linux/arch/i386/boot and has name bzImage. This is to be moved to a convenient location, conventionally to /boot or to just /. You might look at your system to determine which of those two places is already used and continue with that style. To figure this out, look at your /etc/lilo.conf. In the example /etc/lilo.conf given earlier we see the line image=/vmlinuz - so the old image is at / and has name vmlinuz. We'll go with that convention and start by renaming the old image via something such as: mv /vmlinuz /vmlinuz.old. Then we'll copy the new image by: cp /usr/src/linux/arch/i386/boot/bzImage /vmlinuz. However, we'll not be ready to reboot until after some more preparation. 
Modifying /etc/lilo.conf
To boot the new kernel via lilo we start by modifying /etc/lilo.conf. We'll proceed by using our example from earlier. Consider the fragment: 
                   image=/vmlinuz
                     root=/dev/hda2
                     label=linux
                     read-only                    
                   
 Cut and paste a copy of that just below the original and modify it to read: 
                   image=/vmlinuz.old
                     root=/dev/hda2
                     label=linux.old
                     read-only                    
                   
 Now our earlier example /etc/lilo.conf would become: 
                      boot=/dev/hda
                   verbose=2
                   compact
                   install=/boot/boot.b
                   map=/boot/map
                   vga=normal
                   prompt
                   message=/boot/message

                   image=/vmlinuz
                     root=/dev/hda2
                     label=linux
                     read-only                    
                          
                   image=/vmlinuz.old
                     root=/dev/hda2
                     label=linux.old
                     read-only                    

 
Modifying /boot/message
Create or modify the /boot/message file so it looks something like this: 
                   To boot the replacement linux kernel,
                   enter 'linux' at the boot prompt.
                   
                   To boot the previous linux kernel,
                   enter 'linux.old' at the boot prompt.

Backing up the master boot record
Use a floppy diskette to store a copy of the master boot record e.g. dd if=/dev/hda of=/fd/MBR bs=512 count=1 Subsequently, if there is reason to restore the old master boot record, you could do so via dd if=/fd/MBR of=/dev/hda bs=512 count=1 
Running lilo
We are now ready to write the master boot record. The preparatory steps allow us to return to the prior kernel if desired/necessary. Run lilo at the command prompt by simply entering 'lilo' at the command line. 
Rebooting
Now reboot the system. If all goes well you will soon see your new boot message on the screen: 
                   To boot the replacement linux kernel,
                   enter 'linux' at the boot prompt.
                   
                   To boot the previous linux kernel,
                   enter 'linux.old' at the boot prompt.

                   boot:
                   
 Enter 'linux', as prompted. If the new kernel configuration is proper, the system should now successfully reboot the system. If it does not, reboot once again and enter 'linux.old' in response to the boot message. If success completely eludes you and the system hangs upon any hard drive reboot, then reboot with the boot floppy or rescue disk and restore the master boot record as described earlier. Failure indicates that you may have done something incorrectly. You can then embark on a second attempt, if you consider it appropriate. 

Chapter 5. Tools for Kernel Programmers
The kernel programing kitbox starts with the C compiler, but writing and compiling code is only part of the story. If you want to code something relatively simple, and need to get up and running quickly, there are prototyping RAD tools, and to code effectively, you will want debugging, trace and profiling tools to ensure your work does what you expect. Finally, there is no magic bullet to comprehending where your modifications should fit into Linux architecture, but there are some very good reverse engineering tools to help you “call before you dig”.
Note: This overview of programming tools, like most discussions on Linux, charts a moving target. New tools and updates are added almost daily; if your needs go beyond what is outlined here, do a fresh hunt through one of the Linux kernel support sites.
Because it is essentially an embedded application, working with the kernel means abandoning many comfort tools of the programmer's toolkit. You cannot use standard libc functions or logging tools; you have only what is provided by the compiler and what is included in the kernel. Fortunately, there are other ways to get inside the box: Through careful analysis of the source files and, with minor restrictions, the use of debuggers and profilers, you can get some inside understanding of what your code is doing. 
GCC, EGCS and GNU Assembler
The foundation of kernel programming is the GNU Compiler Collection (GCC). Linux is defined by the the interpretations of the C standard as they appear in GCC; as a general rule, Linux is portable to any platform where the Free Software Foundation has ported the GNU compiler. There are exceptions, but this is the official relationship between Linux and GNU: All patches and all bug reports are verified through the official, Linux-approved version of the GCC.
Linux programming may begin with GCC, but not just any GCC: At the time of writing, Documentation/Changes lists version 2.7.2.3 as the only compiler where the kernel is expected to compile. The sources are not neccessarily incompatible with other versions or even with other C compilers, but before you can report suspected kernel or compiler bugs, you must first verify your results using the 2.7.2.3 GNU compiler.
Note: Many core developers will test the kernel build with proprietary compilers (for example, the IRIX and Solaris C compilers) and with the Experimental GNU Compiler System (EGCS), and each build will be tested in different configurations and platforms; unless you find problems specifically mentioned in the KernelNotes or kernel-list archives, the chances for success with another ANSI standard C compiler are reasonably good.
Linux 2.4 and egcs
Why not EGCS? After all, most Linux distributions released after the summer of 1999 bundle the EGCS version of GNU C. This is frequently debated in kernel circles. Yes, EGCS is the “official” GNU C, but no, it is not the same as GCC. The key word in “EGCS” is “Experimental”.
Some explaination: After the release of gcc2 in 1997, the Free Software Foundation halted further development of the compiler and the EGCS group was formed to carry the project forward, this time as a true open source initiative rather than as the cloistered workgroup model used for the earlier versions. In late April 1999, the Free Software Foundation formally appointed EGCS as official maintainers of the compiler. As a result, EGCS, because it is “new technology”, is the most widely distributed flavour of the GNU C compiler in modern distributions. It is still "new technology" and the merger of the old version 2 GCC code and the newer EGCS code is not yet complete. There are subtle differences.
Strange things can happen with egcs, often very subtle and hard to debug things. The most significant difference between these compilers is in optimization, especially with inline code. Prior to Linux 2.2, the differences were stark: The more aggressive optimizations on the Intel version caused wildly incorrect compilations. Since the primary goal of the 2.0/2.2 kernels was stability and EGCS presented too many unknowns, the kernel group voted to stay with 2.7.2.3 until these issues could be resolved.
This does not mean you cannot use the newer compiler. Many problematic sections have been rewritten and the most gross EGCS complications have vanished. Still, absence of evidence is not the same as evidence of absence: As recently as 2.3.18, optimization problems again pointed fingers at EGCS for miscompilations of deeply nested macros. It is like the dentist who tells patients they need not floss all their teeth, only those they wish to keep: You can use EGCS, but if you require stability, you reduce your unknowns with the official, albeit less clever compiler. 
Using GNU Assembler
If you feel the urge to use assembler, get some friends to talk you out of it. Most tasks are completely practical using pure C, and much of the assembler code in the Linux kernel has migrated to C to ease portability (or EGCS conflict) issues. If you absolutely must have close coupling with your hardware and assembler is your only option, then, just as with the C compiler, the official assembler is the GNU Assembler (GAS) in the binutils package. Once again, Documentation/Changes lists the official and minimum version as binutils-2.9.1.0.7; other assemblers may be possible, and many are mentioned in the Assembly HOWTO, but the same caveat applies: Before you can report a bug, you must verify your results with the official toolset.
Assembler can be included inline in the C sources or in stand-alone files; both use the same assembler, in one case invoked by gcc, the other by make rules. The results and the rules are the same. Detailed information on mixing inline-assembler with C can be found in the GCC info pages under C Extensions::®Extended Asm:: with details about Intel-platform conventions under Invoking GCC::®Submodel Options::®i386 Options::. 
Inline Assembler and Optimization
Be very careful with compiler optimization flags when using inline assembly language. GCC will not inline assembler functions unless optimization -O or -On flags are added; without optimization, the link stage will hang looking for unresolved references to any assembly routines that may have been embedded in extern inline functions. You can mix -g and -O flags for debugging, but you must include optimization for correct inline compilation.

If you know any other assembler, you will find few surprises in GAS. GNU assembler uses standard AT&T syntax, a little odd to Intel users, but completely familiar to those accustomed to the m68k assembler. 
Assembler Caveats
The Assembler-HOWTO offers the following checklist on using GNU assembler:
•	Register names are always prefixed with the percent (%) sign, as in %eax; this makes it possible to mix external C symbols without introducing underscore notation.
•	The order of operands is reversed from the Intel convention; gas expects operand pairs as (source, destination)
•	Operand datatypes can be disambiguated by appending a single-letter type identifier to the name of the operator, for example, 
movw %ax,
                  %dx
 would move a 16-bit (word) value; the default behaviour will deduce the word-size from the operands or use 32-bit (long) values.
•	Immediate operands are identified by a $ prefix
•	Where there is no prefix, variable identifiers are taken as memory references, for example $var is the address of var whereas just var denotes the value stored at that location.
•	Parenthesis around a variable name denote indirection or indexing, for example 17(%ebp) is an offset of 17 bytes from the address stored in the base pointer.
Intel2GAS Assembler Translator
In some instances, assembler code from other platforms and from other assemblers can be machine translated to GAS format using Mikko Tiihonen's Intel2GAS (ttp://hermes.terminal.at/intel2gas/) translator. Originally designed to translate NASM files to the GNU Assembler, this project has evolved into a general purpose system to translate several ix86 assemblers into GAS and can be scripted to translate between almost any two assembler syntax conventions.
Navigational Aids
You may understand the compiler and have a good notion what your code must accomplish. Next, you need to know where and how it will fit into the kernel architecture. Simply adding a new kernel device module may have an obvious best-location, but if your project spans module or subsystem boundaries, you need the big picture. Even if you are only extending existing code or adding a new minor service, without a current and accurate map, you run a risk of re-inventing services which may have been available had you chosen some other location for your code. 
Any project of 1.6 million lines is going to have some tangles no matter how you slice it. Telling you to “start in init/main.c” is not much help, and even a printed reference such as this book is at best only a snapshot conceptual map through the labyrinth; the developers will emplore you to “just read the source” but in a project this size, finding your way around is a non-trivial problem.
 Fortunately, several excellent open source packages exist for reverse-engineering large projects. There are also many commercial toolkits but start you thinking about architecture, I don't want to invite the excuse that you cannot afford them; this section will focus only on those tools which are freely available, with a preference for open source.
Code Visualization
Code visualization is not about pretty printouts of control structures in a source file. It is useful to have such a display for large and complex functions, but such code is not very common in the kernel sources (see StyleGuide). In our everyday life, we expect streets to be straight with periodic intersections and rows of driveways on either side; we don't buy maps to place our mailbox in front of the house, but we do buy maps and surveys to see the neighbourhood and we use architectural plans to understand any large construction project. Even if those plans are not accurate (they rarely are) we can work out the differences as we explore. This is the purpose of code visualization.
 Unfortunately, most open source projects have no functional specifications and no architectural plans. At best, there are some loose guiding principles as decided by the steering committees. Linux is no different. Contrary to the best advice of software engineering, most open source projects are invented on the fly, made up as project members encounter each new design problem. Open source teams move organically with no master plan, and overall, the method works very well ... unless you arrived late.
I'm not advocating design process be thrust upon open source; not only does the open source “massively parallel solution space random walk” work, but Apache, Linux and many other large projects prove that it works very well, as good or better than the traditional software methodologies. I'm only suggesting that, for any non-trivial changes, we need to understand the current architecture, and this is especially important in a fast-moving project like Linux. Due to the great popularity of the O/S, we have limited contact with core developers, and a rapid pace of kernel development renders existing documents obsolete faster than they are written. Even for the seasoned team member, the escalating pace and scope of Linux development ensures, whatever conceptual model they may have had when they started, the reality is probably very different, and there is no one we can “call before we dig”: On-the-fly reverse engineering techniques are practically essential.
Reverse engineering is the process of deriving a map directly from the territory using an automated system to gather relational data, and a graph editor to help distill the jumble of call graphs and misinformation of directory structures into comprehensible interactions of well-defined subsystems. While common in object-oriented programming, the practice is relatively unknown in the open source community: Most projects have processes for moving conceptual consensus into source code, but there are virtually no reverse repair practices ensuring documents match the source reality.
Rigi Visual Software Understanding Tool
Rigi is a code visualization suite from the University of Victoria in British Columbia. The package has been ported to Linux and also runs on Sun SPARCstations (SunOS), IBM RISC System 6000 (AIX) workstations, and on Windows 95/NT. Rigi can extract and abstract structural information from the source code and display results in a variety of high-level views; the system is bundled with parsers for C, C++ and COBOL (a Y2K-boom legacy?) and can be extended using the Tcl interface.
Rigi includes two main components, a command-line source parsing rigiparse for creating the Rigi Standard Format (RSF) data tupples, and the Motif-based regiedit diagram editor for the graph manipulations, analysis and report generation. The kit also includes tools for massaging raw data produced by the parser to remove redundant information and for generating hypertext displays
The regiedit display window initially shows the project as an inverted tree where leaf nodes correspond to functions and structures, and intermediate nodes are assigned as preliminary subsystem boundaries. Selecting a subsystem will show contained elements with the call and reference arcs. You then use the editor to alter this projection and re-group nodes to disclose the underlying structure of the software.
While it is easy enough to manipulate source code graphs using rigiedit, populating the graph with the relational data gleened from source code is not so straightforward. For an application the size and complexity of Linux, this requires considerable massaging of data before you can get a meaningful result; the built-in RCL scripting language allows you to automate much of the process, but creating a useful map of the entire Linux kernel will require some effort and experiment.
To prepare the database for the editor, the basic procedure is
	1.	Create a shell script wrapper around  gcc to substitute the preprocessor output and  for all the compiler invokations in the Linux make files.
	2.	Recompile Linux setting the  CC to your script wrapper. This  will create corresponding regiparse  output files for every C-language file.
	3.	After compilation, the results can be collected  together, sorted and filtered, or processed into  hypertext information (for a hypertext navigation)
	4.	Once loaded into the editor, the built-in  scripting system can also be used to filter the  dataset for redundant or misleading links before  displaying the project graph.
The process does not require changes to the Linux source tree and will only map those options which were selected in the kernel .config file.
It is worth the effort. Once loaded into the graph editor, you have a very powerful tool for distilling the source files into manageable bits. For example, you can generate forward dependency maps out from main() or any other function to see the call or reference scope of that item.
 A very useful feature of Rigi is the ability to reframe any data structure into an object oriented view: The structure is portrayed as the attributes of a virtual class with the functions involving this structure as its methods. The whole virtual class definition is grouped as a single subsystem and folding this back into the overall graph will allow you to project higher-order forward dependency and interactions maps. This process can be repeated to reduce the entire kernel into the subsystems listed in Part II.
For your documentation and presentation purposes, Rigi will also produce graph view screen dumps which can be massaged through postscript tools such as gv or xv or included in DocBook files; Rigi's output may be somewhat rustic, but the program more than compensates for aesthetics by being a solid piece of technical wizardry.
The package also includes scripted demos of the abstraction features using a simple application, a ray tracer and a portion of the IBM SQL/DS sources. The latter is a dramatic demonstration of distilling a nightmarish soup of a flat call graph into an elegant and instantly comprehensible display. Other examples of applying Rigi to very large projects and a collection of research papers can be found on the Rigi Homepage (http://www.rigi.csc.uvic.ca) at the University of Victoria.
PBS: The Portable Bookshelf
Conceptually and functionally similar to Rigi, the Portable Book Shelf (PBS) from Ric Holt's group at the University of Waterloo is another reverse-engineering Swiss Army Knife for gathering and presenting structural information from a large software project. PBS has also been ported to Solaris, Linux and NT and can be obtained from the PBS Homepage (http://www.turing.toronto.edu/pbs/index.html) at the University of Toronto; this site also sports a pre-built demo (and downloadable datasets) using the 2.0 Linux kernel as well as many excellent software architecture research documents describing PBS as specifically applied to Linux.
(sidebar) Do we need reverse engineering?
Do we need elaborate schemes to tell us what is already outlined in other kernel documentation without the benefit of complicated tools like Rigi or PBS? Yes, other documentation does divide the kernel into these same subsystems, and we even depended on that knowledge in distilling our concrete model, but what is most interesting is where we differ: Our empirical model shows many interactions and dependencies not reported in other documents or even in the source code comments. By appealing to reality, we discover non-obvious (or forgotten) interactions such as the kmod dependencies between the network subsystem and IPC.
This should not imply our empirical view is any less prone to omissions or errors; our tools can and do produce artifacts which must be verified in the source code logic, and because these tools work on the kernel being built (using our current .config file) we cannot be certain other configurations will not include other unseen relationships. What we can do is gain a more accurate map of what is known and to illustrate Richard Feynman's famous remark “"...it doesn't matter how beautiful your theory is, it doesn't matter how smart you are -- if it doesn't agree with experiment, it's wrong.”

Source Code Navigation
Once you understand kernel structure, and while you do your actual coding, you need to manage that tangle of symbols and function calls. Souce code navigation tools give you various methods for hypertext views of source files; you need to know where you are going, but for stepping through static source structures, these products are essential
These tools can be divided into two camps, the HTML browser-based systems and the ctags editor-extension systems. Each method has its advantages and disadvantages, but all have the common objective of point-and-click navigation through cross-references between many source files spread over many directories.
The Linux Source Navigator
One of the first systems to render Linux in hypertext is a lone Metalab UNC webpage called simply “the Linux Source Navigator”. There are no mentions of authorship, available source code or any explanation why this site ceased to follow Linux at some unspecified incarnation of the 2.0 kernel; the only things I can say about the the Linux Source Navigator (http://metalab.unc.edu/navigator-bin/navigator.cgi?) are that it is fast and that it probably also served as an inspiration for more useful projects such as LXR.
LXR: The Linux Cross-Reference project
The Source Navigator was the seed of an idea to use HTML as a means to present Linux source code. LXR takes this further, wrapping the idea into a useful and elegant package for presenting multiple versions and multiple branches of a large project.
LXR is a suite of Perl5 scripts which, when coupled with a search engine such as Glimpse or Ht/Dig, lets you navigate kernel sources by searching directories, searching on keywords, identifiers, strings and perl regular expressions (regexps), and even just plain old hyperlink browsing. The display renders C code with all project identifiers and function names as HTML anchors linking to the definitions or to cross reference lists on other uses of the symbol.
If you have a reasonably fast Internet connection, you can enjoy the benefits of LXR through the LXR Homepage (http://lxr.linux.no/). This site sports a full demonstration of the software, complete with Glimpse-search capabilities, and shows the very latest Linux sources for both the production and the development kernels. Other demos at the LXR site include source browsing of the GIMP, BSD and several other large-scale open source projects.
Installing the LXR
If, on the other hand, you need rapid and/or local access to the display, a full installation of the LXR search engine is simple to do, but the installation is large: The identifier indexes alone for the 2.3 Linux kernel total 30Mb and take nearly a half hour to generate on a Celeron 300 PC. The LXR documents are sparse, but not impossible to follow if taken completely literally. An overview of the installation goes as follows:
	1.	Install the LXR web files in your webserver  document tree and set the webserver config to recognize  the CGI programs
	2.	create the index directories outside of the  document tree, typically in  /usr/local/lxr and create index  subdirectories for each kernel version to be indexed;  these kernel versions must also be listed in the  versions text file in the base  directory of the indexes
	3.	change to the version-specific index directory and  run the genxref program giving the full path of the  kernel source tree as the target of the command.
This installs the basic system to browse the source code and switch between alternate versions, plus it will let you do basic identifier lookups and to navigate the hyperlinks LXR will embed into the source-code display
The next-level of LXR use requires a search engine, and for this purpose, the LXR documents recommend the Glimpse search engine, but they concede that any text-based search engine will do just as well; the LXR config allows for any CGI program to be called in response to the full-text search form1. Given this backend, the system is able to answer just about any pattern-search query on the kernel in a matter of seconds.
Using LXR
There are no surprises in using LXR. The screen display shows the current page and lists all installed kernel versions and architectures (see Figure 5-1). As you would expect, while viewing a file for one version or architecture, selecting the option for an alternate source tree will display the corresponding file (if available) in that new context. This makes LXR an indispensable tool for making source comparisons, for example, when adapting instructions for previous kernels to the situation in a more modern source tree.
Figure 5-1. Screenshot of an LXR page

In-Editor Source Browsing
A source browser such as LXR may be useful for large sweeps of your sources or for identifier or comment searching via the ad-hoc perl regexps, but it is read-only and browser based, and not very practical during your hands-on coding. To integrate source navigation into the practice of writing software, many programmers already use code navigation tools such as ctags in vi or etags in EMACS but these tools are not quite up to indexing tags lists as large and complex as those of the Linux kernel.
Global Source Tags
You can think of Shigio Yamaguchi's Global as ctags on steroids: Global is a portable source code tag system for C, YACC and Java source files and can be integrated into vi or Emacs, or used with a stand-alone hypertext generator to quickly locate identifiers, object references and even preprocessor symbols. 
Global tags may be invoked within your editor or from the command line. Its features include
•	Reference scope can be limited to directories allowing duplicate identifiers within different source areas. 
•	Search space can include library paths
•	Tag files are platform independent and can be moved or shared freely between different architectures (for example, tag files may be shared via NFS)
•	An optional compact format can be used to reduce diskspace requirements
Obtaining Global
This is one of those gem applications where development has all but ceased only because the current version is about as good as it gets. The most recent version dates from 1996 and is available at the Global Homepage (http://wafu.netgate.net/tama/unix/global.html); it is also available through large toolkit sites such as MetaLab.
Building Kernel GTags
The advanced uses of the Global system for generating HTML output, leveraging the Extended VI and EMACS specific features or extending the parsers are beyond the scope of this chapter; the documents which accompany Global are unusually complete and well written. All you need know here is, to the end user, preparing a complete GTAGS index for even a large project such as Linux is no more complicated than the traditional CTAGS; all the complexity of the Global btree indexing is completely and delightfully hidden.
The command-line use of Global is similarly straight forward and with no surprises. Using the included global utility from a shell, you can quickly locate identifiers and references ... try this the next time a kernel patch leaves you without some symbol defined! It gets even better for the Kernel developer: By creating separate indexes in each of your linux-VERSION directories and a third index in the parent /usr/src directory, searching done in the context of one version will be limited to that scope while a search done from the root directory will locate all references or definitions in all your installed versions.
There's more good news: Global does not require a complete rebuild of the database every time there is a change to a few files. There is an initial hit to build your tags index, but thereafter, tag files can be updated incrementally by changing to the parent directory of the changed or patched files and issuing the -i command option. Solidly cool.
XRef-Speller for EMACS
Similar in intent to Global under the EMACS editors, XRef-Speller also provides in-editor tracing of identifiers and function calls in C, C++ and Java source code. XRef extends this basic lookup function to include other interesting features such as the ability to highlight a segment of code within a function and to auto-generate a new function to factor out that code, or to do the opposite and adjust a function to be inserted into some other function. Like the Source Code Browser, XRef can also be used to generate hypertext (HTML) editions of source code files to cross reference symbols to the definitions and other uses; their website includes a demo of this using the 2.2.5 Linux sources (http://guma.ii.fmph.uniba.sk/linux.html).
Obtaining XRef
XRef is distributed through the XRef-Tech Homepage (ttp://www.xref-tech.com/) and while demonstration versions are available for download, XRef is commercial software and is licenced either in binary or in source-code form.
Using kernel-doc
Understanding a large program requires grasping the high-level design and navigating the token zoo, but it also requires understanding the source code itself and the intentions of the developers. While not followed throughout the Linux kernel (it wouldn't be any fun if there were rules), many kernel developers, and everyone reading this book, are adopting the kernel-doc conventions for embedded comments; Tim Waugh and Michael Zucchi have contributed a perl script, scripts/kernel-doc, to parse these comments into DocBook, HTML, man page and text formats. This utility will also let you single out specific functions in the C code to be included or excluded.
parport_set_timeout illustrates the man-page like output returned from the kernel-doc command perl scripts/kernel-doc -docbook -function parport_set_timeout drivers/parport/ieee1284.c.
The kernel-doc is the core application behind the new Documentation/DocBook project headed by Tim Waugh and Alan Cox. This directory uses a simple template system to insert kernel-doc generated reference pages into DocBook sources. A few “Kernel Books” can be found in the Documentation/DocBook directory where there is also a Makefile for generating PDF and Postscript output 
Note: Programmer guidelines for coding kernel-doc comment blocks can be found in the section called Comment Blocks in Chapter 6.
parport_set_timeout
Name
parport_set_timeout — set the inactivity timeout for a device
Synopsis
long parport_set_timeout (struct pardevice * dev, long inactivity);
Arguments
dev
 	device on a port
inactivity
 	inactivity timeout (in jiffies)
Description
This sets the inactivity timeout for a particular device on a port. This affects functions like parport_wait_peripheral. The special value 0 means not to call schedule while dealing with this device.
The return value is the previous inactivity timeout.
Any callers of parport_wait_event for this device are woken up.
Description
This sets the inactivity timeout for a particular device on a port. This affects functions like parport_wait_peripheral. The special value 0 means not to call schedule while dealing with this device. 
The return value is the previous inactivity timeout.
Any callers of parport_wait_event for this device are woken up.
Development Tools
In a perfect world, you have all the time you need to carefully craft blemish-free software with a carte blanche budget. I don't know about your world, but in mine, my clients want things yesterday, and they are more likely to approve a project budget after they can see something tangible, only rarely before. Some readers may be raptured in a labour of love working on kernel code, but others may need that extra boost of development tools. Fortunately, there is help.
Development tools can be divided into three domains: RAD tools, performance evaluation, and proofing simulators. The first get you coding quickly, the second let you verify and probe your program, and the last group give you a test-harness that avoids the bother and risk of continually rebooting and lets you observe your code on almost any interface instead of crossing your fingers and inflicting it on a live machine.
Linux offers the most in the middle category, in software probes and debugging tools; there is a simple RAD kit for developing new driver modules, but there is only one. There is also only one 'sandbox' application to run your kernel in userspace. You can probably tell from these counts where the average programmer spends most of their time, and it is not in either initial coding or in the final proofing of their code. 
Rapid Application Development in Kernel Programming
A kernel RAD tool needs to capture knowledge of (or extract) current kernel APIs, structures and function calls. It would know what is valid and in what sequence, and it would build source and document templates for common design patterns. In actual practice, however, you are not likely to create many new kernel services in your kernel programming career, and you are much more likely to spend most of your time using source code navigation tools such as Global. This probably explains why there are so few (exactly one) kernel development kits: There is no demand. It may also simply be an application who's time has not yet come.
Linux Driver Development Kit
Let's suppose you have some new piece of hardware which will sit on some I/O or network port awaiting some handshake sequence of events; this is the scenario of the Linux Driver Development Kit. A project of Linux Labs, the LDDK simplifies creating device drivers by generating the basic module source code from a a driver description file and corresponding code-templates.
The toolkit currently contains only one program, the command-line based compiler for their Driver Definition Language (DDL); other tools such as a GUI and a language editor are planned, but at the time of writing, the recommended path is to just use SWIG (see below) to add widget access to the compiler. Given your hand-crafted DDL file, the ddl2c compiler will generate a complete driver project source tree, complete with all your makefiles, libraries and online documentation, all ready to compile into a new kernel module.
While limited, the LDDK is still very useful for creating (or just bootstrapping) global devices (single major number) or simple submodules within a single hardware device (minor numbered devices). The generated files automatically include /proc pseudo-file reporting, /proc/sys Sysctl I/O, ioctl() functions and the user libraries. Generated drivers also include definitions for multi-level debugging code which can be manipulated via insmod command line parameters.
The LDDK package includes example code for simple and submodule drivers, block devices, memory mapping, /proc functions and a guide to passing signals between the kernel and user applications. The full documentation and download information for the LDDK is available from the Linux Labs Software Page (http://www.llp.fu-berlin.de/pool/software/dutil/); Tcl/Tk, Perl, Phyton or Guile support for your can also be added through the SWIG wrapper generator from http://www.cs.utah.edu/~beazley/SWIG/ (http://www.cs.utah.edu/~beazley/SWIG/) or from the Linux Lab Project.
Debugging and Kernel Probes
Linus Torvalds
"I'm afraid that I've seen too many people fix bugs by looking at debugger output, and that almost inevitably leads to fixing the symptoms rather than the underlying problems."
Torvalds may be right, but you can also have bugs which cannot be discovered through static analysis of the source code. Very often, especially in real-time or multithreaded applications, the only way to gather information about system behaviour is to watch it. Put another way, software is never under any obligation to behave the way it was intended to behave. 
Working on the operating system itself presents some obvious problems for debugging. Because the kernel is what provides the basic I/O services for all applications software, we are not able to stand outside of its space in the same way we can stand outside of another application; outside of the kernel, there is no place to stand. Fortunately, this is not precisely true: We are able to gain some knowledge about the running kernel through the kernel's own I/O services such as printk() or sysctl(), or we can indeed stand outside the kernel by standing in the I/O space of some other kernel either on the same machine or on some other box.
When printk() fails
Simple diagnostics and tracing can usually be done by peppering suspect code with the kernel library printk() function. As expected, printk() provides a 'printf'-like facility to log messages through syslogd. Where it works, and where calling the function is allowed, this technique works quite well, but as we all know, like any instrumentation, print statements inserted into code do not monitor the code you wrote, they measure new code with print statements inserted into it. The difference can lead to code which may work when traced, but cease to work when the statements are removed.
The same warning holds true for the use of assert() macros to add pre- and post-conditions to your functions: While wonderful in theory, when these macros are enabled they can fundamentally change your program. I have seen cases where inserting assert() shifted stack addresses enough to prevent the bug from occuring and could as easily caused new instabilities to appear. 

Sometimes, the only way to understand why our code behaves the way it does is to strap it to the lab bench and stuff it full of probes and monitors. This section describes some methods in all of these scenarios, from the very low-level probes to ways to enable full symbolic debugging and port taps on a running kernel image.
Using ptrace
 Unless you are creating custom debugging tools or extending the facilities of the existing tools, you are unlikely to work directly with this function. ptrace() is the standard unix basis of all breakpoint debugging and is used to manipulate the runtime state of a process through the protocol of signals defined in the header files. For example, a controlling process can attach another using the PTRACE_ATTACH request and initiate a single instruction step with PTRACE_SINGLESTEP or manipulate data within the child process or change the register values within a stopped process.
Most Linux distributions will include ample documentation for the ptrace system call easily accessible through man ptrace; in Linux systems, <sys/ptrace.h>> defines the interface to all the process trace and manipulations functions located in Linux/arch/ARCH/kernel/ptrace.c. 
Using the GNU Symbolic Debugger (gdb)
Rather than manipulating the kernel directly through ptrace(), if you must monitor or correct a running kernel image, you will probably do this using gdb. The good news is the GNU symbolic debugger can easily monitor a running Linux kernel; the bad news is the access is read-only; you cannot set breakpoints or modify values. 
To prep the kernel for debugging, 
	1.	Add debugging symbols to your object  files by manually inserting the -g  compiler option into  arch/i368/Makefile.
	2.	Enable the /proc filesystem  in the kernel options; this will be your window into  the kernel.
	3.	Issue a make clean and then  rebuild and install the new kernel image
 With these changes, the new kernel can be monitored through the /proc/kcore memory image pseudo-file using the command
gdb vmlinux --core /proc/kcore
 If you need more control over the runtime state or you need to monitor a kernel that is going to crash and burn (taking your debugger with it), gdb can be run from a second machine and connected to your test box through a serial port; to facilitate this connection and to enable you to attach to the kernel as early in the boot process as possible, Silicon Graphics offers a kernel patch which adds the gdb option to the LILO prompt; this patch will also add the option to define CONFIG_GDB in the kernel config so the feature can be turned off when no longer required. The SGI kGDB patch can be downloaded from the kGDB Project Homepage (http://oss.sgi.com/projects/kgdb/). 
Built-in Kernel Debugger
SGI people just don't know when to quit. If your kernel is among the list of supported versions, they also offer the KDB patch (see oss.sgi.com (http://oss.sgi.com/projects/kdb/)) which will add configuration options for CONFIG_KDB and CONFIG_KDB_FRAMEPTR. When enabled, these options provide symbolic debugging services directly in the kernel, without the need for a remote gdb process; the KDB patch kit also includes a Documentation/kdb directory with copious release and usage information.
Kernel Sandboxes
What if your changes may damage your hardware, or involve handshake sequences you would like to test through a regression test suite and a device stub? If your changes kernel fit within the capabilities of a simulator, it may make sense to test your code a user-space virtual machine, or 'sandbox', to observe its behaviour. In many cases, the same tests can be done through a debugger or even by simply tracing loading and unloading your kernel module, but in some circumstances, you may want to verify your code before you inflict it on a real machine and to proof your new kernel in a nice safe bubble.
When the kernel is running in user-space, it is much easier to use standard debugging and verification tools. Debugging with gdb or profiling your code using gprof is no more involved than any other application. Other diagnostics, for example branch testing and other automated regression tests, can also be applied.
User Mode Kernel (UMK)
The only existing "Kernel in a Sandbox" system, the "User Mode Kernel" by Jeff Dike, started out as an experiment to prove doing such madness was actually possible; it was later recognized to have real debugging value and has since evolved into a major project and a major voice behind mainstream kernel facilities such as ptrace().
When run in on a console or in a terminal window, the UMK will appear just like any kernel boot, but with some devices adjusted around the host machine. For example, during the boot sequence, the new kernel may report serial ports as 
serial line 0 assigned pty /dev/ptyp7
; to connect to this running kernel, you can either login at the prompt in your window or connect via the serial port. In our example this would be by pointing minicom or some other termal program at /dev/ptyp7; when you exit from this session, the user mode kernel will close that port and open a new one, for example, on /dev/ttyp6
The UMK is available for Linux 2.2 but is primarily aimed at the 2.3 kernels; if you only need to test loadable modules, you can also fetch a pre-compiled kernel image. The software supports network emulation, but does not (yet) simulate multi-processors and many hardware devices are in need of a hero to code them in. Your installation will require the UMK patches or the binary and a root filesystem for the new runtime kernel; this can be built on any existing filesystem using the scripts provided. More information on UMK and all the necessary files can be fetched from Jeff Dike's UML2 website (http://www.mv.com/ipusers/karaya/uml/uml.html).
CPU Emulators
Depending on your application and its device requirements, it may be possible to boot your experimental kernel within a ix86 (or other architecture) emulator such as VMWare or FreeVMWare/Bochs, although the stark division between the operating space of such emulators will severely limit the amount of information you can extract from the test; considering the extra cost of the software and of the machine requirements needed to run the virtual machine, it is probably easier, cheaper and more fruitful to purchase a cheap beater-box and to use the remote debugging technique.
Useful Utilities
You have your new custom kernel up and running, it gets safely past initialization and now you need to know if it is doing what it should and is not doing what it shouldn't. In this stage of development, you may not need the bother of a symbolic debugger to know your program has gone astray (or to prove that it hasn't) and in some cases, a debugger would be giving information too low level or too specific to be useful.
While it is true that there is not much you can do about the behaviour of your changes while the kernel is booting, most often it is enough to just watch the progress of your code and the state of your machine using the traditional means of the programmers' toolkit: Since most devices are readily implemented as dynamically loaded modules, there are relatively few applications outside of inner kernel subsystems development where plain old system monitor tools are not enough to diagnose the problem ... and as expected, it is those deep core procedures which are the most intractable to identify, but let's try to stay positive: 90% of everyday kernel diagnosis will be obvious from the system behaviour. For that other 10%, the following sections describe some of the available options available to Linux programmers.
System and Process Monitors
Assuming your modified kernel will boot far enough to be accessible, and also assuming your changes are amiable to being tracked by the usual system monitor tools. These include the normal arsenal of unix commands such as top, free and the direct reading of the /proc pseudo files these utilities reference. Very recently, other standard unix tools for process accounting, programs such as sar, have been ported to Linux and will generally give a more accurate picture of what the machine was doing while it ran your code.
Analyzing Core Dumps
It ran for hours, the process monitors all looked normal and then ... ooops. There are several means available to deduce the final state of the kernel after an oops, ranging in their diagnostic detail from showing only where the program failed, to being able to recreate the state of the machine.
 At one end of the post-mortem spectrum, the ksymoops utility, which was included in the Linux source tree up until the 2.2 distribution, will map the output of the kernel oops to the System.map file to show the name fo the function where the failure occured (See the section called Tracing from a Kernel OOPS in Chapter 6. Very often, knowing the location of a failure is enough to diagnose a problem, but for other situations, there is no substitute for a full backtrace and snapshot of your machine at the time of death. In the traditional user-space programming toolkit, this is the domain of the gdb --core command; thanks to the contributions of the Linux Kernel Crash Dump project at Silicon Graphics, it is now also possible to apply this analysis to the Linux kernel
The LKCD kit includes kernel patches and user-space tools to capture a core dump of a failing kernel to a dedicated SCSI partition, and it can also perform some analysis on the generated core dump or to reload the kernel state when the machine reboots.
In its simplest form, LKCD will store the core file snapshot to /dev/vmdump, and then, on reboot, it will fetch this image and generate a report on /var/log/vmdump. RPM files and the necessary kernel patches can be downloaded from the LKCD homepage (http://oss.sgi.com/projects/lkcd/).
Kernel Profiling
Your program does not oops, and it doesn't exhibit any gross behaviours recognizable during process monitoring, but it still fails to make the proper handshake or just needs to go faster; this is the domain of profiling.
The stock Linux kernel offers very basic facilities for gathering timing statistics on certain subsystems such as the SCSI parallel interface and the network layer, but all such code is limited in resolution by the 100Hz kernel clock. As a result, the standard kernel profiling is pretty much useless. Several authors have contributed patches to allow fine grain profiling but in all cases, these patches are highly version-specific and are not be available for all kernels; the code is often portable to other kernel versions, and the authors are almost always helpful, but if you are tinkering with this stuff, be prepared for surprises (which is to say, don't promise the results to anyone by any specific deadline date).
Profiling with gprof
If you have been following this chapter from the start, you may have already concluded that Silicon Graphics is a big player in the realm of kernel debugging; kernel profiling is yet another area made possible by the kind contributions from SGI engineers. The gprof patches provide detailed trace information through the /proc filesystem; once the system is prepared, kernprof is used to set the sampling rates and to toggle the data collection and this data is then analyzed by running gprof on the kernel image and the generated data.
Prepping the kernel is done by applying the SGI patch to the source tree to collect the call arc data; unfortunately, due to a bug in all known editions of GCC, including the EGCS compiler, gcc must also be modified using the patches found in the EGCS patch archive (http://egcs.cygnus.com/ml/egcs-patches/2000-04/msg00332.html). If you are unable to modify your compiler, an alternative fix is to remove all FASTCALL support in the kernel sources; the SGI patch kit includes a patch to do this removal but keep in mind that this will change the behaviour of all systems which might use the FASTCALL API. 
As of this writing, there is no FTP archive for this patch other than in the mailing list archives. If you are looking for it in some other archive, the gprof patch was posted to the Linux Kernel Mailing List (http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/2383.html) by Dimitris Michailidis on the 14th of May, 1999.
Linux Trace Toolkit
Karim Yaghmour's Trace Toolkit is another kernel instrumentation package designed to catalog CPU and allocation events down to microsecond resolution. Like other instrumented-kernel patch sets, LTT adds a trace capability option to the standard kernel configuration script and lets you choose the level of detail and the specifics on the events to be monitored.
Because the LTT tracks the system over a specific time period, it is particularly useful for debugging I/O latency and synchronization problems. Generated reports include event flow, process analysis and raw event logs; these reports are all available in user-space via the /proc interface and can be read as text tables or analyzed with the included GUI reporting module.
The kit consists of the kernel instrumentation patches, a kernel module and trace daemon to handle the data collection, and a GUI report reader which will translate the trace output into a readable report. Source files and documentation can be downloaded from the Linux Trace Toolkit Homepage (http://www.info.polymtl.ca/~karym/trace/)
IKD Debugging Tools Patch
For the Linux 2.2.x kernels, Andrea Archangelli compiled a single patch to bundle together the most common kernel debugging facilities. The included tools cover
•	Debug kernel stack overflows
•	Detect software lockups
•	Procedure flow tracing
•	Set breakpoints and single step
•	Print-EIP on video ram
•	Stack monitor
•	Fine-grain profiling
Unfortunately, all of these services are highly version dependent and while the patch may be portable to other kernel versions, Andrea tends to only support a few as the needs arise.
The latest versions of the IKD patch are available from e-mind.com through Andrea's FTP area (ftp://e-mind.com/pub/andrea/ikd/)
Notes
1. 	In 1999, the Glimpse search engine was pulled from the free software world and re-released as a commercial product with the usual restrictions on for-profit use. Before you install Glimpse and its companion WebGlimpse, read over the new license to ensure your use will comply with their requirements.
2. 	While Jeff refers to his system as the User-Mode Kernel, he most often abbreviates this as UML (user-mode linux); I have chosen the short form of UMK to avoid confusion with the Universal Modelling Language

Chapter 6. Contributing to the Kernel Project (20)
Your particular use of the Linux kernel may be intended for wide release or you may be looking to modify portions for a specific application in your own business or research projects. Of course, any changes you may make to the kernel project are your own business, that is the beauty of working on a GPL project. Still, if you are like most people who become involved in Linux, as time goes on, you will want to share what you've done to help advance the project. 
Participation can take many forms. Your first step will probably be to join the discussion groups on USENET or on IRC to make contact with other developers working in your area. Later you may want to take a more active role, first by contributing your own results with kernel patches and later by even taking or sharing ownership of kernel subsystems. In all these instances, the community has evolved some groundrules for participation. This section is a brief guide to how this is done, how to join in and participate in Linux.
Participating in Linux Development
Your first act of participation is in choosing Linux, and if you have come this far, you probably already know about resources such as the Linux Documentation Project and the documentation available in the kernel linux/Documentation directory; both of these collections contain everything you need to know about who is who in the kernel project.
Points of Contact
Most developers in the kernel project are more than approachable. The official list of developers and the subsystems under their wing is listed in the MAINTAINERS file in the root directory of the kernel source tree. When looking at this list, though, you need to keep in mind there are millions of Linux users, but only a few dozen developers listed in the file; these contacts should be considered as the project managers and as such may not be the best people to contact about trivial problems or novice questions.
When you do have questions about the kernel sources, try to exhaust all other options before you ask directly. It may seem more expedient to you to send a single email rather than sift twenty or thirty webpages, but do consider what it would be like to be on the other end of an email address which has been posted publically as an expert before an audience of millions. Linux has been around for many years, and each new revision is tested and probed by thousands of people. Chances are very good your question has been asked before. Your first stop should always be the kernel FAQ (http://www.tux.org), then the mailing list archives search page or Deja.com (http://www.deja.com), and only then should you approach the list or the contributors directly.
Instead of dealing directly with the top people, a better approach is to first join the Linux Kernel mailing list at Tux.org (http://www.tux.org) or to browse the archives to see who on the list is involved in your particular area of interest. Summary sites for this list such as the Kernel Notes (http://www.kernelnotes.org) regularly publish summaries of the week's postings where you can also get a feel for who may be best able to answer your questions.
Mailing List Ettiquette
The usual rules of nettiquette apply in your communications with the kernel development group; I don't need to tell you this. There are, however, other guidelines to keep in mind in your communications:
•	Kernel programmers are primarily employed on their own projects; they are not paid technical support people for Linux but are paid by their employer to get real work done on their company's agenda, not yours.
•	The Golden Rule of kernel programming is that, if you absolutely need some feature added for your own use, you should be prepared to add it yourself. If it is a great idea that has wide appeal, the developers may agree and change their focus to accommodate you, but they are under no obligation to do so. If you need it for a deliverable, your proper course of action is to roll up your sleeves and build it yourself, or to arrange to hire someone on your own initiative. Kernel programmers are not hard to find (a pleasant side effect of an open source project) but the kernel developers themselves are largely not available for hire.
•	The linux-kernel mailing list is a very busy newsgroup which should have been factored into many subgroups years ago. As it stands, it already has far too much noise and its members, largely due to the above points, are not very tolerant of people adding more. If you have a valid comment or something of value to contribute, you are more than welcome to join, but please try to avoid "me too" postings or postings that stray from the charter of the group.
Keeping all those points of ettiquette in mind, it is also very true that the kernel project needs your input; your "kernel oops" report, even if the cause eludes you, is very important to them as they work to ensure Linux continues to be both stable and widely useful.
Joining the Inner Circle
The structure of the Linux development team is such that all 'official' additions to the kernel sources will eventually find their way to Linus Torvalds before they are folded into the master sources. Within each subsystem, the developers listed in the contributors file may work on their own private source tree and collaborate with their core group and the public to produce what they believe to be the best possible technical solution, but at some point, their patches must go through the Torvalds gateway. 
Add to this that Linux is exploding. You may be reading this years after my writing but I have no doubt that statement will be as true in your time as it is today. As the achievements of a few enable ever more advances by ever more people who in their turn enable even more advances and participants, there is no absolute limit such growth. To maintain sanity, the Linux team structure has needed to evolve over time, not only to adjust for people who, for whatever reason, were forced to leave the project, but because the number of patches and additions vying for inclusion have continued to increase in frequency.
To accommodate this explosive growth, by Linux 2.0, Linus had opened the inner circle to admit others, and today, there are three or four people who hold the keys to the gateway. I'm not for a moment suggesting you may be thinking of joining that prestigious group, although it is certainly not unthinkable for the right talent, but I say this to give you a picture of how the flow of development is structured. In more practical terms, it is very much within reach for the right people with the right mix of people skills, telework skills and programming savvy to find themselves the primary contact for a particular subsystem; it is all part of my initial statement that the Linux team members are all very approachable, and they know a valuable player when they meet one.
Programming Style
If you search through the kernel mailing list or the kernel sources, you will find several discussions about coding style with no real resolution. Some authors will ask that you emulate the style used in the module where you are working, but such advice really only ascerbates the situation. In the Linux 1.2 kernel, Torvalds set forth his first all for a common coding style for use by all developers and his advice has since been collected and expanded into the linux/Documentation/CodingStyle document. That document starts out with the following advice:“First off, I'd suggest printing out a copy of the GNU coding standards, and NOT read it. Burn them, it's a great symbolic gesture.”
Coding Style Guide
The basic advice on coding style is to follow the traditional K&R rules set forth at the dawn of C; formatting programs such as GNU indent can be used, but need commandline modifiers to conform to the kernel conventions. Put briefly, these conventions are
•	Use 8 character tabs
•	Place opening braces at the end of the line, not on a line by themselves
•	An exception to the above rule is to place opening braces on a new line only when the brace begins a function block.
•	Closing braces are on a line by themselves unless they are followed by a continuing clause such as with an else or with the while in a do-while block.
•	Variable names should be descriptive while not becoming verbose; some C compilers are only sensitive to the first 32 characters of an identifier, and long variable names only rarely add useful information.
•	Mixed-case variable names are frowned upon. Multi-word variable names should be seperated by underscores.
•	Global variables must be used sparingly and should have very clearly unique and descriptive names.
•	Local variables should be short, lowercase tokens. Loop counters have been called "i" since Fortran days and no one ever got hurt ... providing your functions do not grow out of control.
•	Functions should be short and single-purpose; the source should fit into one or two screens of text. The best assurance of code correctness is having code structures which can be instantly comprehended. If your function requires more than 10 local variables, you may want to reconsider your design. GNU C can always inline your functions if you need to save the overhead of a function call.
•	Comments are good. Everyone likes comments, but only if those comments explain what the code should be doing and why; code which renders the algorithm in natural language is only rarely useful. Use small comments to warn future developers, but otherwise avoid plaing comments inside the function body; comments should be placed in a block above the function declaration. 
Comment Blocks
New to the 2.4 kernel, Tim Waugh and Alan Cox are promoting GNOME-like comment block conventions; Tim has also provided a script utility, kernel-doc to format these comments into DocBook, HTML, man pages and text; all kernel contributors are encouraged to annotate their code with these blocks so new developers can generate accurate API and design docs in much the same way that Perl and Java programmers use perldoc and javadoc.
Comment blocks follow the a simple template:
 /**
  * function_prototype() (- short description)  Œ
  * @parameter_x: (description of parameter x)  �
  *                                                                                     Ž
  * Description: (Description of function)              
  * section header: (section description)               �
  **/
Œ	These elements are optional
Ž	Function prototype and parameter descriptions are separated froom the description and section header with a blank line.
��	These elements are may be repeated
The header template will be familiar to Java and GNOME programmers. Kernel-Doc will also replace special markup for environment variables, structure names and contants according to the following patterns:
funcname()	 function	$ENVVAR	 environmental variable	&struct_name	 name of a structure (can include “struct”)	@parameter	 name of a parameter	%CONST	 name of a constant.	

Details on the use of the kernel-doc script can be found in the section called Using kernel-doc in Chapter 5.
Using GNU Emacs
If you are using GNU Emacs or one of its variants, you can modify your c-mode support to follow the kernel guidelines, either globally for all your C language coding or to limit this style to just the kernel sources. To enable this within the scope of kernel files, you can add the following code to your .emacs file: 
(defun linux-c-mode ()
   "C mode with adjusted defaults for use with the Linux kernel."
   (interactive)
   (c-mode)
   (c-set-style "K&amp;R")
   (setq c-basic-offset 8))
 
(setq auto-mode-alist
       (append '(("/usr/src/linux.*/.*\\.[ch]$" . linux-c-mode)) 
                           auto-mode-alist))
Using GNU indent
If you are using some other editor or if you get tangled about using Emacs, you can recover the proper formatting of your C language files with the indent command. In its basic mode, indent will backup your source files and produce new copies; indent can also be used to transform source files between different coding styles or to reformat your comment blocks (see man indent). For our purposes, the command to reformat a file into Linux kernel style is
indent
                -kr -i8 <sourcefile>
Asserts and Exceptions
As a general rule, avoid assert() statements; where they work, they are bloat, and where they don't they are trouble. Besides, Linus does not like them.
Other Programming Guidelines
Your very best source on C language programming style for working with any highly-portable and large-team-effort project is to appeal to "K&R", the classic "Programming in C" by Kernighan and Richie, and a final word on Linux programming guidelines belongs to legendary open source programmer, Henry Spencer and his oft-cited Ten Commandments of C Programming (http://www.ee.ryerson.ca:8080/~elf/hack/The10.html). 
Preparing a Patch
Thomas Jefferson
The patch should be co-measurate with the hole.
Final say on what is and is not included into the Linux kernel rests with Linus Torvalds, although as the size, complexity and popularity of Linux grows, Linus is shifting some of that burden to able contributors such as Alan Cox. Promising candidate patches are often incorporated into Alan's pre-release patches before becoming part of the canonical source. This does not mean your brilliant hack should be sent directly to either of these two; although roles do change, every module has a maintainer listed in the linux/MAINTAINERS file, and any patches intended for general distribution should go to the relevent people on that list.
The ease with which the maintainers can apply your patch is directly related to the speed and likelihood of the patch being included. Patches should always be submitted as a context diff, the output from the diff -c command. A convenient way to generate these patches is to keep your local Linux sources in a versioning repository such as CVS and to generate the patch with the cvs diff command.
Bug Reporting
Bug reports are of crucial importance to Linux developers, but only when the reports include relevent information. Linux runs on many platforms and many configurations; simply saying that “SMP crashes” is about as useful as including the weather report. Context is everything, and developers bestow special blessings on those who can supply a short, repeatable example.
Don't let me disuade you from submitting bug reports on sporadic problems, and for many past bugs, the most obtuse observation has been critically useful, but where possible, try to provide as much information about your machine, the tasks it was doing and any other hardware or process information relevent to the bug. If the crash includes an “OOPS:” message, follow the procedure in the section called Tracing from a Kernel OOPS to create a more useful description of your machine and the state of the kernel when the problem occurred. If possible, create a test case which will reliably trip the OOPS condition; a repeatable example of a problem is worth far more than any amount of diagnostic data.
Kernel First-Aid
Before reporting any suspected bug, check the obvious causes.
•	If you are using EGCS or any other unofficial development tools, verify the bug using the recommended toolkit as described in Chapter 5.
•	A frequent cause of trouble is a mismatch between the kernel and the supporting software itemized in Documentation/Changes; the script file scripts/ver_linux will generate a partial list of version numbers to check against the recommended list.
•	Check symlink paths for /usr/include/linux and other relevant headerfiles, libraries and binaries to ensure you do not have version mis-matching or older versions masking out newer versions on the search path. If possible, verify the bug on another machine.
Reporting a Bug
Detailed reporting is very important; kernel developers want to help, and in most cases can help very quickly, providing they are given enough information to localize and identify the problem code. When preparing a bug report, include the following information:
	1.	One-line description of the problem
	2.	A more detailed description including any  processes which were going on at the time, any events  or odd hardware which may have triggered the bug and the  sequence of events which lead to the bug.
	3.	If you are posting to the mailing list, include  keywords to help identify the relevent maintainer, for  example, “modules, networking, NFS”
	4.	The kernel version reported by  /proc/version or reported when the  kernel boots.
	5.	If the bug results in an OOPS, include the output  with the symbolic information resolved (see the section called Tracing from a Kernel OOPS).
	6.	If you can repeat the problem conditions, include  a shell script or the sequence of steps that will evoke  the bug.
In addition to the bug synopsis, list a summary of your operating environment:
	1.	Software versions as reported by  scripts/ver_linux
	2.	Output from /proc/cpuinfo
	3.	Active modules listed from  /proc/modules or  lsmod
	4.	Current hardware and driver information output  from
•	/proc/ioports
•	/proc/interrupts
•	/proc/iomem
	5.	Where relevent, include the PCI information from  lspci -vvv (  lspcidrake on Mandrake  machines), SCSI details from  /proc/scsi/scsi and any other  /proc reports which may shed some  light on the state of your machine.
	6.	Any other notes, patches or workarounds. If you  can include the fix for this bug as a context-diff  against the current kernel sources, expect a very warm  welcome.
The completed bug report should be sent to the current maintainer for the relevent kernel component as listed in the MAINTAINERS file; bugs specific to any one module should be sent to the technical lead listed in that file, or, where the bug seems specific to a particular source file, reports can be sent to the author listed in the file header. As a last resort, or for bugs which appear to transcend module boundaries, reports can be sent to the Linux-kernel mailing list (http://www.tux.org/lkml) at <linux-kernel@vger.rutgers.edu>. 
Tracing from a Kernel OOPS
The main trick is having 5 years of experience with those pesky oops messages
Linus
On those rare occasions where the kernel does fail, the dying gasp lets out a register dump. The addresses in this dump can be resolved against the current /proc/ksyms using the ksymoops program; because this utility does not depend on the version of the kernel, ksymoops is no longer bundled with the Linux source code and now lives with many other kernel debugging and reportint tools at the OCS FTP site (ftp://ftp.ocs.com.au/pub/ksymoops).
Depending on the cause and severity of the OOPS condition, the register dump may have been read by klogd and recorded by syslogd in the /var/log/messages log. If klogd fails, the message may be available using dmesg or can be read directly from the kernel buffers using cat /proc/kmsg; be aware that the latter method is reading a continuously generated source and must be interrupted with Ctrl-C.
If the kernel OOPS happens during the kernel boot or is otherwise unavailable using any of the above methods, you have three options:
	1.	Configure your system for a serial console and  capture the output on a second machine.
	2.	Transcribe the message off the screen by hand.
	3.	Apply one of the non-standard “crash  dump” patches such as  kmsgdump, lkcd  or oops+smram to enable saving  kernel messages to a floppy disk, video RAM or the swap  partition. (See the section called Analyzing Core Dumps in Chapter 5
Given the output from ksymoops, the next step is to load the errant code into the debugger and to compare the disassembled binary with the register values given in the panic message to gain some insight into what should be happening vs what is actually happening. Running gdb vmlinux (ie, on the uncompressed kernel image), the gdb disassemble command will generate a source listing of the given function; the given offset into that function will pin-point the failing code.
Generating a Reference Disassembly: In Documentation/kernel-oops.txt, Linus provides two tips for generating assembly code to compare with the code generated with the gdb disassembly command:
	1.	make fs/buffer.s (or whatever  the errant file is) will stop the compilation after  generating the GNU assembler. If the kernel bug  report and the current kernel were created using the  same development environment, this will provide a  reference for comparison with the disassembled code. 
	2.	The Code: values given in the  panic message can be translated into assembler by  creating a short C sourcefile, compiling it with  gcc -g and running it through the  gdb disassemble. Linus recommends an  empty main() and a static  char str[] set to the code values;  he suggests using cut and paste to place the code  numbers into the C source and replacing the spaces in  the pasted string with  \x:
        char str[] = "\xXX\xXX\xXX...";
        main(){}
 
Linus
Now, if somebody gets the idea that this is time-consuming and requires some small amount of concentration, you're right. Which is why I will mostly just ignore any panic reports that don't have the symbol table info etc looked up: it simply gets too hard to look it up (I have some programs to search for specific patterns in the kernel code segment, and sometimes I have been able to look up those kinds of panics too, but that really requires pretty good knowledge of the kernel just to be able to pick out the right sequences etc..)
Sometimes it happens that I just see the disassembled code sequence from the panic, and I know immediately where it's coming from. That's when I get worried that I've been doing this for too long ;-) 

II. Kernel Subsystems
Table of Contents
7. Speaking Kernel: Global Structures and Methods (45)	000
8. Kernel Initialization	000
9. Process Scheduler (30)	000
10. Interprocess Communications (IPC) (20)	000
11. Memory Management (20) Thomas Strohm	000
12. Resource Management	000
13. Linux File Systems (30) Manuel J. Petit de Gabriel	000
14. Networking Layer(35) Alolita Sharma, Bob Adkins, Nilayan Sharma	000

III. Kernel Services
Table of Contents
15. Designing Linux Modules (15) Juan-Mariano de Goyeneche	000
16. Device Drivers (35)	000
17. Character Devices (20)	000
18. Block Devices (20)	000
19. Network Devices (20) Alolita Sharma, Bob Adkins, Nilayan Sharma	000
20. PCI (20)	000
21. SCSI Devices (20) Kurt Garloff	000
22. Parallel Ports (20) Tim Waugh	000
23. Video Systems (30)	000
24. Video for Linux: TV and Radio Interface (15)	000
25. Sound Systems (20)	000
26. SMP (20) Alolita Sharma, Bob Adkins, Nilayan Sharma	000
27. Plug'n'Play (15) Tom Lees	000

IV. Advanced Topics
Table of Contents
28. Porting Drivers from NT (35)	000
29. Porting Linux to Other Platforms (35)	000
30. Embedded Linux (30) Donald Jeff Dionne and Michael Durrant	000
31. Real-Time Linux (30) Richard West and Christian Poellabauer	000

Appendix A. Overview of Linux Processors (20)
Intel x86
PIII Extensions
Strong-ARM
HP Alpha AXP
Motorola 68k
Sparc64
Itanium (ia64)
IBM S/390
SuperH

Appendix B. Kernel Programming Resources (10)
Linux Documentation
Online Kernel Resources
LDP Kernel Guides
linux-kernel Mailing List
Further Reading

Bibliography
[Beck97] Michael Beck, Harald Böohme, and Robert Magnus, Linux Kernel Internals, Addison-Wesley, 1997.
[Card98] Remy Card, Eric Dumas, and Franck Mevel, Linux Kernel Programming, John Wiley & Sons, 1998.
[Rubini98] Alessandro Rubini and Andy Oram, Linux Device Drivers, O'Reilly Publishers, 1998.

Index