Debugging ChibiOS/RT applications

ChibiOS/RT offers several mechanisms that can help in the debug phase of the development cycle.

What this guide does not cover

This guide assumes knowledge in following areas:

• General knowledge of embedded development.
• RTOS concepts.
• Setup of your specific target hardware and toolchain.
• Knowledge of your toolchain. The guide will explain what you need to do, not how it is done using you specific debugger, compiler, JTAG probe and target hardware.

Helpful debugging configuration settings

There are several settings in your kernel configuration file (see templates/chconf.h) that you may want to enable during debugging and in general during the whole development process.

• CH_OPTIMIZE_SPEED=FALSE, this disables inlining into the kernel code and makes it easier to debug using your debugger, you may also want to reduce or disable compiler optimizations (-O0 using GCC).
• CH_DBG_ENABLE_CHECKS=TRUE, this setting enables the checks on the API parameters, useful to understand if you are passing wrong parameters to the OS functions.
• CH_DBG_ENABLE_ASSERTS=TRUE, this setting enables the OS internal consistency checks, this can trap several kind of errors in the user code (or in the kernel itself).
• CH_DBG_ENABLE_STACK_CHECK=TRUE, this setting enables checks on threads stack overflow. Note that this option is not available in all ports, check your port documentation. If not supported then it is silently ignored, see also the article Stacks and stack sizes.
• CH_DBG_FILL_THREADS=TRUE, this setting enables the threads workspace filling, this can help examining the stack usage from your debugger.

Note that all the failed checks lock the kernel into the port_halt() function. In order to assess what triggered the lock the global variable panic_msg must be inspected using the debugger, the variable is a pointer to an error message (a zero terminated string), the pointer may contain NULL if the lock was triggered by a stack overflow.

Common errors and symptoms

There are some common errors while using an RTOS, use the following table as a check list, if your problem is not a generic programming error then probably it is one of the following common RTOS/embedded related mistakes:

• Insufficient stack allocated to one or more threads.
  Common symptoms:
  • Target instability.
  • Target locked into the port_halt() function.
  • Target trapped into an exception handler (architecture dependent).
  • Target apparent self reset (not real resets usually).
• Insufficient stack allocated to the IRQ stack (in those architectures that have a separate IRQ stack, ARM as example).
  Common symptoms:
  • Target instability.
  • Target trapped into an exception handler (architecture dependent).
  • Target apparent self reset (not real resets usually).
• Use of a non reentrant function from within an interrupt handler, as example most C runtime functions.
  Common symptoms:
  • Target instability.
  • Unexpected application behavior.
• Missing use of a mutual exclusion mechanism to protect data (or non reentrant code) shared among multiple threads and/or threads and interrupt handlers, see also the article Mutual Exclusion guide.
  Common symptoms:
  • Target instability.
  • Unexpected application behavior.
• Use of S-class or I-class APIs outside a proper lock state, see the Kernel Concepts article, specifically the API Names Suffixes and System States sections.
  Common symptoms:
  • Target instability.
  • Target trapped into an exception handler (architecture dependent).
  • Target apparent self reset (not real resets usually).
• Use of a non I-class API from an interrupt handler, see the Kernel Concepts article, specifically the API Names Suffixes and System States sections.
  Common symptoms:
  • Target instability.
  • Target trapped into an exception handler (architecture dependent).
  • Target apparent self reset (not real resets usually).
• Wrong threads priority assignment. One of the most critical things to do when designing an RTOS based application is to assign correct priorities to the threads in the system.
  Common symptoms:
  • Excessive or unpredictable response times.
  • Threads that appear to be never executed (CPU intensive threads at higher priority).

General suggestions

For the less expert users, there are several things you may do in order to minimize the need for debugging:

• Read carefully the documentation first.
• Try to find a code examples for things are you going to do, good sources are: the documentation, the test code, under "./test" you will find examples for almost any API in the ChibiOS/RT kernel and most common RTOS related tasks, under "./testhal" there are examples regarding the various device drivers, the various demos contain good code samples too).
• Start your application from an existing demo, add things one at a time and test often, if you add too many things at once then finding a small problem can become a debugging nightmare. Follow the cycle: think, implement, test, repeat.
• If you are stuck for too much time then consider asking for advice.
• Report bugs and problems, bugs can be fixed, problems can become new articles in the documentation (this and other documentation articles spawned from questions in the forum or in the tracker).
• Never give up :-)