Skip to content

Cluster Backup

Note

  • The full source code is on GitHub
  • Look at the RsmClusterBackup example in the cluster-rsm/src/main/java/com/aeroncookbook/cluster/rsm/node/RsmClusterBackup.java.

Cluster Backup is part of the Aeron open source offering that allow for a node to take a copy of the log and the snapshots from a running cluster and store them on a separate machine. This can allow for a cold DR like operation or provide a repository of state that can be used to provision a new node in an existing cluster.

Rebuilding a Cluster Member

In some cases it is necessary to rebuild an existing cluster member from scratch. Cluster Backup provides the ability to handle this with a few manual steps. The recommended approach for rebuilding a cluster member from an empty system is:

  1. Trigger a snapshot on the running cluster so that there is a fairly recent snapshot to recover from.
  2. Provision a Cluster Backup instance, look at the reference to the above example for further description. The key configuration options are: a. ClusterBackup.Context.clusterConsensusEndpoints(...), which is the list of endpoints that the live cluster members are using for consensus. This is the 2nd entry of each line of the ConsensusModule.clusterMembers configuration. b. ClusterBackup.Context.catchupEndpoint, which is the host/port combination that the ClusterBackup will subscribe to to receive log data and snapshots from the other nodes when it performs a backup of the data. c. ClusterBackup.Context.consensusChannel, which is the channel that the ClusterBackup will receive notifications from the backup query (i.e. snapshot metadata queries) responses.
  3. Start the backup and let it run to the point where it is roughly inline with the rest of the cluster. Use AeronStat to look at the counters to ensure that the position of the log is roughly inline with the live nodes. Due to the way that the backup will gate on the commit position of the cluster it is always likely to trail the live nodes. Within a single log buffer term is a reasonable delta. We are wanting to ensure that it has successfully replicated a snapshot and started replaying the live log.
  4. Stop the ClusterBackup.
  5. Set up a new node to join the cluster. Ensure that the name or IP address matches the appropriate entry within the clusterMembers configuration. I.e. it's endpoints should match the values in the existing configuration. We don't have a means to live update the cluster members, so without stopping the whole cluster it must reuse that entry of the node it is replacing.
  6. Copy the following data from the backup to the new node. It is also possible to use the backup's data in place. ClusterBackup uses the same data layout for the cluster and archive directories. The new cluster member can point to those directories used by the backup: a. All the data from within the backup's archive directory (and subdirectories), excluding the mark file into the new node's archive directory. b. Copy the recording.log file from the ClusterBackup's cluster directory to the new node's cluster directory.
  7. Start the new node.