Testing Strategy

Description

Most of the ChibiOS/RT demos link a set of software modules (test suite) in order to verify the proper working of the kernel, the port and the demo itself.

Strategy by Component

The OS components are tested in various modes depending on their importance:

• Kernel. The kernel code is subject to rigorous testing. The test suite aims to test all the kernel code and reach a code coverage as close to 100% as possible. In addition to the code coverage, the kernel code is tested for functionality and benchmarked for speed and size before each stable release. In addition to the code coverage and functional testing a batch compilation test is performed before each release, the kernel is compiled by alternatively enabling and disabling all the various configuration options, the kernel code is expected to compile without errors nor warnings and execute the test suite without failures (a specific simulator is used for this execution test, it is done automatically by a script because the entire sequence can take hours).
  All the tests results are included as reports in the OS distribution under ./docs/reports.
• Ports. The port code is tested by executing the kernel test suite on the target hardware. A port is validated only if it passes all the tests. Speed and size benchmarks for all the supported architectures are performed, both size and speed regressions are monitored.
• HAL. The HAL high level code and device drivers implementations are tested by use in the various demos and/or by users.
• Various. The miscellaneous code is tested by use in the various demos and/or by users.
• External Code. Not tested, external libraries or components are used as-is or with minor patching where required, problems are usually reported upstream.

Kernel Test Suite

The kernel test suite is divided in modules or test sequences. Each Test Module performs a series of tests on a specified kernel subsystem or subsystems and can report a failure/success status and/or a performance index as the test suite output.
The test suite is usually activated in the demo applications by pressing a button on the target board, see the readme into the various demos directories. The test suite output is usually sent through a serial port and can be examined by using a terminal emulator program.

Kernel Test Modules

• Threads and Scheduler test
• Dynamic APIs test
• Messages test
• Semaphores test
• Mutexes test
• Events test
• Mailboxes test
• I/O Queues test
• Memory Heap test
• Memory Pools test
• Kernel Benchmarks