Log Buffers & Images¶
In Aeron, a Log Buffer is a memory mapped file broken into four sections:
- three equal sized sections - each a Term with a unique Term ID - which hold header and message data;
- a metadata section, which resides at the end of the file.
Each term can be in one of three logical states:
cleanterms have yet to have data written to them,
activeterm is where data is being written to,
dirtyterms hold data that is no longer active, but is temporarily available for retransmission.
During typical operation, the terms cycle through clean to active and then dirty. Term IDs increment as each term completes a cycle. When a message is offered to a publication, an Aeron header is created, and the header plus message data get written into a term of a log buffer. When offering data to a subscription, sometimes you will receive a response indicating
admin action. Admin action occurs when Aeron is busy rotating the terms.
Every publication will receive its own log buffer. An Image is a subscription's replicated publication stream. You rarely need to interact with Image objects directly - the most common use is via the Aeron Cluster snapshot loading process.
Reconstructing a stream¶
UDP messages can and often do arrive out of order. Aeron guarantees that the order of messages within a stream is retained. Log buffers help with this correct reconstruction of the ordered stream by using the header data within each message to properly place the data within the log buffer.
As messages are received, they are placed at the correct position in the log buffer. The high watermark (
rcv-hwm) keeps track of the furtherest point in the log buffer at which data has been received, and the completed position (
rcv-pos) marks the highest point at which the contiguous sequence has been received. Both of these values are visible in the AeronStat tool - see Aeron Tooling.
It is only once the completed position moves forward that the subscription will receive new fragments. Using the above animation as an example, after receipt of the top message the subscription receives notification that a fragement is available. The next time fragments are received by the subscription is only once the bottom message is written to the log buffer.
Term Buffer Lengths¶
Term Buffer Lengths are set with the system property
aeron.ipc.term.buffer.length. These can also be set on the Media Driver context using
- the smallest term buffer length is 65536 bytes;
- the largest term buffer length is 1,073,741,824 bytes;
- the size must be a power of 2;
- maximum message length is the smallest value of 16 megabytes or term buffer length / 8;
- if needed, the term buffer length for an individual publication can be set. See How to send messages over 8kb
- if the application will be hosting the log buffer on the
/dev/shmfolder, you will need to make sure make sure you have sufficient space.
Aeron performs unblocking in scenarios where a Log Buffer has been blocked through a client or publication failure. Unblocking will happen in the background when either a publication is closed with a claim still in place, or when a client unexpectedly disconnects. To allow the log buffer to continue, Aeron will reset the header and unblock the log buffer after the unblock timeout has been reached (this defaults to 15 seconds).
You can observe the number of publication and control command unblocking events using the Aeron Stat tool. See Aeron Tooling, in particular
Unblocked Publications and
Unblocked Control Commands.