path: root/rklogd.8

RKLOGD(8)                          Linux System Administration                          RKLOGD(8)

NAME
       rklogd - Kernel Log Daemon

SYNOPSIS
       rklogd  [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [ -p ] [ -s ] [ -k fname ] [ -v
       ] [ -x ] [ -2 ]

DESCRIPTION
       rklogd is a system daemon which intercepts and logs Linux kernel messages.

OPTIONS
       -c n   Sets the default log level of console messages to n.

       -d     Enable debugging mode.  This will generate LOTS of output to stderr.

       -f file
              Log messages to the specified filename rather than to the syslog facility.

       -i -I  Signal the currently executing rklogd daemon.  Both of these switches  control  the
              loading/reloading  of  symbol  information.   The  -i  switch signals the daemon to
              reload the kernel module symbols.  The -I switch signals for a reload of  both  the
              static kernel symbols and the kernel module symbols.

       -n     Avoid  auto-backgrounding.   This is needed especially if the rklogd is started and
              controlled by init(8).

       -o     Execute in 'one-shot' mode.  This causes rklogd to read and log  all  the  messages
              that  are  found  in the kernel message buffers.  After a single read and log cycle
              the daemon exits.

       -p     Enable paranoia.  This option controls when rklogd loads kernel module  symbol  in-
              formation.   Setting this switch causes rklogd to load the kernel module symbol in-
              formation whenever an Oops string is detected in the kernel message stream.

       -s     Force rklogd to use the system call interface to the kernel message buffers.

       -k file
              Use the specified file as the source of kernel symbol information.

       -v     Print version and exit.

       -x     Omits EIP translation and therefore doesn't read the System.map file.

       -2     When symbols are expanded, print the line twice.  Once with addresses converted  to
              symbols, once with the raw text.  This allows external programs such as ksymoops do
              their own processing on the original data.

OVERVIEW
       The functionality of rklogd has been typically incorporated into other versions of syslogd
       but  this  seems to be a poor place for it.  In the modern Linux kernel a number of kernel
       messaging issues such as sourcing, prioritization and resolution of kernel addresses  must
       be addressed.  Incorporating kernel logging into a separate process offers a cleaner sepa-
       ration of services.

       In Linux there are two potential sources of kernel log information: the /proc file  system
       and  the  syscall  (sys_syslog)  interface, although ultimately they are one and the same.
       Klogd is designed to choose whichever source of information is the most  appropriate.   It
       does  this  by first checking for the presence of a mounted /proc file system.  If this is
       found the /proc/kmsg file is used as the source of kernel log information.   If  the  proc
       file  system is not mounted rklogd uses a system call to obtain kernel messages.  The com-
       mand line switch (-s) can be used to force rklogd to use the system call interface as  its
       messaging source.

       If  kernel  messages are directed through the syslogd daemon the rklogd daemon, as of ver-
       sion 1.1, has the ability to properly prioritize kernel messages.  Prioritization  of  the
       kernel  messages was added to it at approximately version 0.99pl13 of the kernel.  The raw
       kernel messages are of the form:

              <[0-7]>Something said by the kernel.

       The priority of the kernel message is encoded as a single numeric  digit  enclosed  inside
       the  <>  pair.   The  definitions of these values is given in the kernel include file ker-
       nel.h.  When a message is received from the kernel the rklogd daemon reads  this  priority
       level  and  assigns  the appropriate priority level to the syslog message.  If file output
       (-f) is used the prioritization sequence is left pre-pended to the kernel message.

       The rklogd daemon also allows the ability to alter the presentation of kernel messages  to
       the  system console.  Consequent with the prioritization of kernel messages was the inclu-
       sion of default messaging levels for the kernel.  In a stock kernel the the  default  con-
       sole  log  level is set to 7.  Any messages with a priority level numerically lower than 7
       (higher priority) appear on the console.

       Messages of priority level 7 are considered to be 'debug' messages and will thus  not  ap-
       pear  on the console.  Many administrators, particularly in a multi-user environment, pre-
       fer that all kernel messages be handled by rklogd and either directed to a file or to  the
       syslogd  daemon.   This  prevents 'nuisance' messages such as line printer out of paper or
       disk change detected from cluttering the console.

       When -c is given on the commandline the rklogd daemon will execute a system  call  to  in-
       hibit all kernel messages from being displayed on the console.  Former versions always is-
       sued this system call and defaulted to all kernel messages except  for  panics.   This  is
       handled differently nowardays so rklogd doesn't need to set this value anymore.  The argu-
       ment given to the -c switch specifies the priority level of messages  which  will  be  di-
       rected  to  the  console.  Note that messages of a priority value LOWER than the indicated
       number will be directed to the console.

              For example, to have the kernel display all messages with a  priority  level  of  3
              (KERN_ERR) or more severe the following command would be executed:

                   rklogd -c 4

       The  definitions  of the numeric values for kernel messages are given in the file kernel.h
       which can be found in the /usr/include/linux directory  if  the  kernel  sources  are  in-
       stalled.   These  values parallel the syslog priority values which are defined in the file
       syslog.h found in the /usr/include/sys sub-directory.

       The rklogd daemon can also be used in a 'one-shot' mode for  reading  the  kernel  message
       buffers.  One shot mode is selected by specifying the -o switch on the command line.  Out-
       put will be directed to either the syslogd daemon or to an alternate file specified by the
       -f switch.

              For example, to read all the kernel messages after a system boot and record them in
              a file called krnl.msg the following command would be given.

                   rklogd -o -f ./krnl.msg

KERNEL ADDRESS RESOLUTION
       If the kernel detects an internal error condition a general protection fault will be trig-
       gered.   As part of the GPF handling procedure the kernel prints out a status report indi-
       cating the state of the processor at the time of the fault.  Included in this display  are
       the  contents  of  the  microprocessor's registers, the contents of the kernel stack and a
       tracing of what functions were being executed at the time of the fault.

       This information is EXTREMELY IMPORTANT in determining what caused the internal error con-
       dition.   The  difficulty  comes when a kernel developer attempts to analyze this informa-
       tion.  The raw numeric information present in the protection fault  printout  is  of  very
       little  use to the developers.  This is due to the fact that kernels are not identical and
       the addresses of variable locations or functions will not be the same in all kernels.   In
       order  to  correctly  diagnose  the cause of failure a kernel developer needs to know what
       specific kernel functions or variable locations were involved in the error.

       As part of the kernel compilation process a listing is created which specified the address
       locations  of important variables and function in the kernel being compiled.  This listing
       is saved in a file called System.map in the top of the kernel directory source tree.   Us-
       ing  this  listing a kernel developer can determine exactly what the kernel was doing when
       the error condition occurred.

       The process of resolving the numeric addresses from the protection fault printout  can  be
       done manually or by using the ksymoops program which is included in the kernel sources.

       As a convenience rklogd will attempt to resolve kernel numeric addresses to their symbolic
       forms if a kernel symbol table is available at execution time.  If you require the  origi-
       nal  address  of  the symbol, use the -2 switch to preserve the numeric address.  A symbol
       table may be specified by using the -k switch on the command line.  If a  symbol  file  is
       not explicitly specified the following filenames will be tried:

       /boot/System.map
       /System.map
       /usr/src/linux/System.map

       Version  information is supplied in the system maps as of kernel 1.3.43.  This version in-
       formation is used to direct an intelligent search of the list of symbol tables.  This fea-
       ture is useful since it provides support for both production and experimental kernels.

       For  example  a production kernel may have its map file stored in /boot/System.map.  If an
       experimental or test kernel is compiled with the sources in  the  'standard'  location  of
       /usr/src/linux  the  system  map  will be found in /usr/src/linux/System.map.  When rklogd
       starts under the experimental kernel the map in /boot/System.map will be bypassed in favor
       of the map in /usr/src/linux/System.map.

       Modern  kernels  as of 1.3.43 properly format important kernel addresses so that they will
       be recognized and translated by rklogd.  Earlier kernels require a source  code  patch  be
       applied to the kernel sources.  This patch is supplied with the sysrklogd sources.

       The  process  of analyzing kernel protections faults works very well with a static kernel.
       Additional difficulties are encountered when attempting to diagnose errors which occur  in
       loadable kernel modules.  Loadable kernel modules are used to implement kernel functional-
       ity in a form which can be loaded or unloaded at will.  The use  of  loadable  modules  is
       useful from a debugging standpoint and can also be useful in decreasing the amount of mem-
       ory required by a kernel.

       The difficulty with diagnosing errors in loadable modules is due to the dynamic nature  of
       the  kernel  modules.  When a module is loaded the kernel will allocate memory to hold the
       module, when the module is unloaded this memory will be returned back to the kernel.  This
       dynamic  memory allocation makes it impossible to produce a map file which details the ad-
       dresses of the variable and functions in a kernel loadable module.  Without this  location
       map it is not possible for a kernel developer to determine what went wrong if a protection
       fault involves a kernel module.

       rklogd has support for dealing with the problem of diagnosing protection faults in  kernel
       loadable modules.  At program start time or in response to a signal the daemon will inter-
       rogate the kernel for a listing of all modules loaded and the addresses in memory they are
       loaded at.  Individual modules can also register the locations of important functions when
       the module is loaded.  The addresses of these exported symbols are also determined  during
       this interrogation process.

       When  a  protection  fault occurs an attempt will be made to resolve kernel addresses from
       the static symbol table.  If this fails the symbols from the currently loaded modules  are
       examined  in  an attempt to resolve the addresses.  At the very minimum this allows rklogd
       to indicate which loadable module was responsible for  generating  the  protection  fault.
       Additional information may be available if the module developer chose to export symbol in-
       formation from the module.

       Proper and accurate resolution of addresses in kernel modules requires that rklogd be  in-
       formed  whenever  the kernel module status changes.  The -i and -I switches can be used to
       signal the currently executing daemon that symbol information be reloaded.  Of most impor-
       tance  to proper resolution of module symbols is the -i switch.  Each time a kernel module
       is loaded or removed from the kernel the following command should be executed:

       rklogd -i

       The -p switch can also be used to insure that module symbol information  is  up  to  date.
       This switch instructs rklogd to reload the module symbol information whenever a protection
       fault is detected.  Caution should be used before invoking the program in 'paranoid' mode.
       The  stability of the kernel and the operating environment is always under question when a
       protection fault occurs.  Since the rklogd daemon must execute system calls  in  order  to
       read the module symbol information there is the possibility that the system may be too un-
       stable to capture useful information.  A much better policy is to insure  that  rklogd  is
       updated  whenever  a  module  is  loaded  or unloaded.  Having uptodate symbol information
       loaded increases the probability of properly resolving a protection fault if it should oc-
       cur.

       Included  in  the  sysrklogd  source  distribution is a patch to the modules-2.0.0 package
       which allows the insmod, rmmod and modprobe utilities to automatically signal rklogd when-
       ever  a  module is inserted or removed from the kernel.  Using this patch will insure that
       the symbol information maintained in rklogd is always consistent with the  current  kernel
       state.

SIGNAL HANDLING
       The  rklogd  will respond to eight signals: SIGHUP, SIGINT, SIGKILL, SIGTERM, SIGTSTP, SI-
       GUSR1, SIGUSR2 and SIGCONT.  The SIGINT, SIGKILL, SIGTERM and SIGHUP  signals  will  cause
       the daemon to close its kernel log sources and terminate gracefully.

       The  SIGTSTP  and SIGCONT signals are used to start and stop kernel logging.  Upon receipt
       of a SIGTSTP signal the daemon will close its log sources and spin in an idle loop.   Sub-
       sequent receipt of a SIGCONT signal will cause the daemon to go through its initialization
       sequence and re-choose an input source.  Using SIGSTOP and SIGCONT in combination the ker-
       nel log input can be re-chosen without stopping and restarting the daemon.  For example if
       the /proc file system is to be un-mounted the following command sequence should be used:

            # kill -TSTP pid
            # umount /proc
            # kill -CONT pid

       Notations will be  made  in  the  system  logs  with  LOG_INFO  priority  documenting  the
       start/stop of logging.

       The  SIGUSR1  and  SIGUSR2 signals are used to initiate loading/reloading of kernel symbol
       information.  Receipt of the SIGUSR1 signal will cause the kernel  module  symbols  to  be
       reloaded.  Signaling the daemon with SIGUSR2 will cause both the static kernel symbols and
       the kernel module symbols to be reloaded.

       Provided that the System.map file is placed in an appropriate location the signal of  gen-
       erally  greatest  usefulness is the SIGUSR1 signal.  This signal is designed to be used to
       signal the daemon when kernel modules are loaded/unloaded.  Sending  this  signal  to  the
       daemon  after  a  kernel module state change will insure that proper resolution of symbols
       will occur if a protection fault occurs in the address space occupied by a kernel module.

FILES
       /proc/kmsg
              One Source for kernel messages rklogd
       /var/run/rklogd.pid
              The file containing the process id of rklogd
       /boot/System.map, /System.map, /usr/src/linux/System.map
              Default locations for kernel system maps.

BUGS
       Probably numerous.  Well formed context diffs appreciated.

AUTHOR
       The rklogd was originally written by Steve Lord (lord@cray.com), Greg Wettstein made major
       improvements.

       Dr. Greg Wettstein (greg@wind.enjellic.com)
       Enjellic Systems Development

       Oncology Research Divsion Computing Facility
       Roger Maris Cancer Center
       Fargo, ND 58122

Version 1.13.5 (devel)                     21 JUNE 2007                                 RKLOGD(8)