Performance

Overview

CCF pairs strong confidentiality guarantees with very high performance. CCF can sustain high transaction throughput, while also reaching consensus over commits with low latency.

There are several performance metrics in the CI test suite to ensure this, ranging from micro-benchmarks of critical systems to end-to-end tests measuring peak throughput. These are run against every PR and commit to the main branch. You can also run these locally to test the configuration of your machines, and use them as a basis for creating performance tests of your own CCF application.

Micro-benchmarks

The micro-benchmark tests can be run from the CCF build directory:

./tests.sh -VV -L "bench"

These test performance-critical features of CCF such as certificate verification and KV alterations.

The picobench runner does not change CPU affinity. For reproducible comparisons, use the same CPU affinity and machine configuration for both runs. On Linux, use taskset when an explicit CPU selection is needed, allowing enough CPUs for multi-threaded benchmarks such as task_bench.

End-to-end performance tests

The end-to-end service performance tests can also be from the CCF build directory:

./tests.sh -VV -L "perf" -C "perf"

Each of these tests creates a temporary CCF service on the local machine, then sends a high volume of transactions to measure peak and average throughput. The Python test wrappers print throughput and latency summary statistics when the test completes.

For a finer grained view of performance the clients in these tests can also dump the precise times each transaction was sent and its response received, for later analysis. The samples folder contains a plot_tx_times Python script which produces plots from this data.

Profiling

End-to-end performance tests can be run with the Linux utility perf attached to the nodes to produce profile data. Set the CCF_PERF environment variable before running a test to enable this:

CCF_PERF=1 ./tests.sh -VV -R '^basic_blocking_locust$' -C perf

By default, nodes run under perf record -m 16 -e task-clock:u -F 99 -g --call-graph dwarf --quiet. These options keep profiling reliable and useful in development containers:

  • -m 16 limits the mmap data buffer to 16 pages, avoiding failures caused by the low perf_event_mlock_kb limit in constrained environments such as Codespaces.

  • -e task-clock:u selects the software task-clock event and records only user-space execution. Unlike hardware counters, this event remains available when the host does not expose a hardware PMU; excluding the kernel also avoids inaccessible kernel symbols.

  • -F 99 samples at 99 Hz, bounding collection overhead while retaining enough detail for whole-test profiles.

  • -g --call-graph dwarf records call chains using DWARF stack unwinding, which preserves the C++ stacks needed by perf report and flame graphs.

  • --quiet suppresses non-fatal recording warnings that would otherwise be mixed into each node’s error log.

Set CCF_PERF_ARGS to replace these recording options. The harness always appends -o perf.data -- so that each node writes to a predictable path in its workspace directory and the remaining arguments invoke the node.

The profiling process requires permission to use perf_event_open. The CCF development containers install perf and grant the PERFMON capability. Hardware performance counters may still be unavailable when the host does not expose a hardware PMU; the default software task-clock event remains available in that case.

Inspect the recorded profile directly with:

perf report --stdio --no-children -i workspace/basic_blocking_locust_0/perf.data

To install the Inferno tools and render a flame graph, run:

cargo install inferno

perf script -i workspace/basic_blocking_locust_0/perf.data \
  | inferno-collapse-perf \
  | inferno-flamegraph > basic_blocking_locust_flamegraph.svg