Tracing and Sampling for Real-Time partially simulated Avionics Systems
Tracers using dynamic tracepoints, including DTrace, SystemTap and GDB tracepoints, have been optimized for ad hoc tracing with no overhead when not tracing. Tracepoints are then dynamically inserted using trap instructions and require a costly context switch to handle the trap in most cases. Furthermore, using traps may impose additional constraints on the context (e.g. not in NMI) in which tracepoints can be inserted. Despite the lower performance, TuningFork uses SystemTap to get system events.
LTTng is a tracer based on static tracepoints. Other Linux kernel tracers, Perf and Ftrace, derive to some extent from LTTng for static tracepoints mechanisms. These tracers have been optimized for low overhead and low disturbance, insuring that they can be used to trace the behavior of heavily loaded multi-core online systems. LTTng in particular uses per CPU buffers, lockless per CPU atomic operations, zero copy buffering and transport, and Read Copy Update (RCU) synchronization. These advanced tracing algorithms need to be extended and complemented in order to operate properly in a real-time environment. In addition, tracing data often integrates information traditionally associated with sampling techniques such as performance counters or program counter value at regular interval. The collection of such information is very useful but also needs to be adapted to real-time operation.
As part of milestone D1.1, the specific context of Real-Time Avionics Systems, with partial or complete simulation, running on distributed multi-core systems, will be studied. This will determine the extent of the complement required for the tracing tools. Then, in milestone D1.2, each of the operations required for tracing (jumping to the probe, reading the trace context, getting the timestamp, writing the event in the buffers, switching between buffers, sending the buffer content to disk or to the network, changing the trace configuration) will be studied to review the associated synchronization required and the corresponding average and maximum latency. New algorithms will be required for some of these operations as well as additional functionality. For example, to insure that a buffer switch or a buffer full situation does not happen while in a tight real-time deadline, the tracer and application may need to cooperate in order to adjust the buffer size and synchronize the buffer transfer operations with the idle time in the real-time loop.
These new algorithms will then be implemented and prototyped in real-time avionics applications at CAE and Opal-RT, with an iterative process of performance measurements and refinements, as part of milestone D1.3. The new algorithms, resulting in improved performance and functionality, will then be transfered to be incorporated into the Linux Tracing Tools by the Linux kernel and LTTng communities.
A similar process will be applied to make available hardware preformance counters, sampling data and other hardware tracing facilities for analyzing real-time avionics applications. In milestone D2.1, the characteristics of performance counters and recent hardware tracing facilities will be studied in the context of real-time applications. Special algorithms, with operating system collaboration, are required to get coherent values when accessing per-cpu hardware counters on a multi-core system where tasks may migrate between CPUs. Then, in milestone D2.2, new algorithms will be developed to efficiently record performance counters and hardware trace data along with the more traditional tracepoint data.
Performance counters are typically used to trigger interrupts when they reach a certain value; the program context is then used as sample. This statistical mode of operation has a very low, adjustable, average overhead. However, in real-time applications, such periodical interrupts may interact with the real-time tasks causing deadline problems or resulting in biased samples. Alternatively, performance counters may be read at specific locations in the program, and then constitute additional payload data at these tracepoints, in which case a special scheme is required to correlate the values with the program execution. Newer processors increasingly include hardware tracing facilities. These facilities typically offer little flexibility in terms of format, buffer size, timestamping and synchronization. The tracing algorithms will then need to be complemented in order to incorporate hardware tracing data while adapting to their specific constraints. These new algorithms will be similarly prototyped, evaluated and refined with real-time avionics applications in milestone D2.3, and in milestone D2.4 published in the scientific literature and transfered to Linux Tracing Tools.