Data Persistence

Durability

Persistence to disk is handled by host-side code, outside the enclave, by code that is not attested and may be controlled by an attacker in the CCF threat model.

As a result, CCF cannot make a formal guarantee about data persistence. Durability relies on the operator maintaining enough healthy nodes, and making regular backups of the ledger. To minimise the risk of data loss on node failure, the CCF host component issues an fflush() call on every transaction as soon as it becomes committable (i.e. followed by a signature).

The operator can further minimize the risk of data loss by running their CCF-based service on a larger number of nodes.

Directories

When a new node has joined the network or when a failed node needs to be recovered, the latest committed snapshot file can be copied to the node before it is started. The node will then automatically resume from the latest snapshot file (see Join or Recover From Snapshot).

The new/recovered node may also need to have access to all .committed ledger files in some cases, for example, if the node needs to serve historical queries. It is therefore safe to back up all the .committed ledger and snapshot files. It is recommended to keep a read-write directory where all local ledger and snapshot files reside and separate backup storage containing only committed files.

        graph TD;
subgraph ccf network;
A["Node 0 (Primary)<br> [Read-Write directory]"];
B["Node 1<br> [Read-Write directory]"];
C["Node 2<br> [Read-Write directory]"];
D["Shared Mount<br>[Read-Only ledger directory]"]
style D fill:#bbf,stroke:#f66,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
A-->D
A-. copy committed ledger files .-> D
B-->D
C-->D
end;
    

The read-only ledger directory can be a shared mounted directory accessible to all nodes and specified via ledger.read_only_directories. The legacy snapshots.read_only_directory option is deprecated; joining nodes fetch a recent snapshot by default.

It is recommended to keep the most up-to-date copies of .committed ledger and snapshot files in backup storage (see Best Practices). Operators must take care to avoid any race conditions in the copy process.

Best Practices

It is recommended for operators to backup the ledger and snapshot files as soon as they become committed (i.e. .committed included in file name). Up-to-date nodes have identical committed ledger chunks, while the current primary should have the newest chunks. Snapshot files are only generated by the current primary node. As such, monitoring the directories specified by ledger.directory and snapshots.directory for the current primary node allows operators to retrieve the latest ledger and snapshot files.

Note

It is the responsibility of the operator to move/copy these files safely to avoid “ledger holes”, i.e. historical ledger files not being available to a new node that started from a recent snapshot.

A low value for ledger.chunk_size means that smaller ledger files are generated and can thus be backed up by operators more regularly, at the cost of having to manage a large number of ledger files.

Similarly, a low value for snapshots.tx_count means that snapshots are generated often and that join/recovery time will be short, at the cost of additional workload on the primary node for snapshot generation.

Note

Uncommitted ledger files (which are likely to contain committed transactions) should also be used on recovery, as long as they are copied to the node’s ledger.directory directory.