Cassandra Tombstones Performance Impact & Tips to Avoid

May 22, 2023

In this post, we will explore what tombstones are, why they can negatively affect Cassandra performance, and how to avoid them.

Apache Cassandra is a highly scalable and distributed NoSQL database that uses a unique data model designed to handle large amounts of data across many commodity servers, with high availability and fault tolerance. However, like any database system, Cassandra has its limitations and challenges, and one of them is the impact of tombstones on performance.

Table of Contents

What are Tombstones?

In Cassandra, a tombstone is a special type of record that represents the deletion of a data row. When a user deletes a row from a table, Cassandra does not immediately remove the row from disk. Instead, it creates a new record, called a tombstone, to indicate that the row has been deleted.

The tombstone contains the timestamp of the deletion and is written to all replicas of the data. This is necessary because, in Cassandra’s distributed architecture, it is not always possible to immediately propagate the deletion to all nodes. Therefore, each node must keep track of the deleted row until it can be sure that all other replicas have received the deletion message.

Once all replicas have received the deletion message, the tombstone can be safely deleted, and the disk space can be reclaimed.

How Tombstones can affect Cassandra’s Performance?

Tombstones can have a significant impact on Cassandra’s performance, especially when they accumulate in large numbers. This is because Cassandra must perform additional work to reconcile the deleted rows across all replicas during read operations, which can slow down queries and cause timeouts.

When a read request is received by Cassandra, it must first check if any tombstones exist for the requested data. If tombstones are found, Cassandra must examine all replicas and compare the timestamp of the tombstone to determine if the requested data was deleted before or after the tombstone was created.

This process can become very expensive when tombstones accumulate, especially in cases where there are many updates or deletions occurring in a short period of time. In some cases, the number of tombstones can grow to a point where Cassandra cannot handle the load, causing performance issues or even crashes.

Cassandra Tombstone Cleanup – How To

The key to tombstone cleanup is Cassandra’s built-in mechanism called ‘compaction’ (See next section). Compaction consolidates SSTables – the immutable data files, and during this process, it discards the tombstones once their grace period has expired. By default, this grace period is ten days, allowing ample time for deletion markers to propagate throughout the cluster.

However, this process isn’t always perfect. If your database operations create an excessive number of tombstones, you may need to tune your compaction strategy. Different compaction strategies are better suited for varying use cases, and choosing the right one can greatly improve the cleanup process.

How to Avoid Tombstones?

To avoid tombstones and improve Cassandra’s performance, there are several best practices that you can follow:

1. Use Time-To-Live (TTL)

TTL is a feature in Cassandra that allows you to set an expiration time for a row. When the expiration time is reached, the row is automatically deleted, and a tombstone is created. By using TTL, you can avoid creating unnecessary tombstones and reduce the amount of work that Cassandra must do to reconcile data.

2. Use Batch Operations

Cassandra supports batch operations that allow you to perform multiple updates or deletions in a single request. By using batch operations, you can reduce the number of tombstones that are created and improve the performance of your application.

3. Limit the Use of DELETE

The DELETE operation in Cassandra creates tombstones, which can negatively impact performance. Therefore, you should limit the use of DELETE and consider other options, such as setting a TTL or updating the row with null values.

4. Monitor Tombstone Count

You should regularly monitor the tombstone count in your Cassandra cluster and take action when the count reaches a certain threshold. For example, you can set a threshold that triggers an alert when the tombstone count exceeds a certain value or percentage of the total data.

5. Use Compaction

Compaction is a process in Cassandra that merges multiple SSTables (sorted string tables) into a single SSTable. During compaction, tombstones are removed, and disk space is reclaimed. Therefore, you should regularly perform compaction to remove tombstones and improve performance.

Cassandra Tombstone Limit

Firstly, rethink your data modeling. If your application often deletes data, it’s more efficient to model your data so that it expires naturally rather than manually deleting it. For time-series data, for example, you can use Time to Live (TTL) fields to have data expire automatically. This approach reduces the creation of tombstones.

Secondly, tune your compaction strategy wisely. Different compaction strategies handle tombstones differently. For instance, the SizeTieredCompactionStrategy (STCS) is not ideal for high delete operations since it doesn’t compact often, leading to a buildup of tombstones. On the other hand, the LeveledCompactionStrategy (LCS) compacts more frequently, thus efficiently managing tombstones.

Thirdly, be careful with large partitions. Large partitions tend to hold onto tombstones longer because Cassandra needs to ensure all data in the partition has been covered before removing any tombstones. Try to avoid large partitions to limit tombstone retention.

Lastly, consider using the gc_grace_seconds setting wisely. This setting determines the time Cassandra waits before removing tombstones. If you have a stable cluster and fast repairs, consider lowering the gc_grace_seconds.

Final Thoughts

Tombstones can have a significant impact on the performance of Apache Cassandra, especially when they accumulate in large numbers. They can slow down queries, cause timeouts, and even lead to crashes in some cases. To avoid tombstones and improve Cassandra’s performance, you should follow best practices such as using Time-To-Live (TTL), batch operations, limiting the use of DELETE, monitoring the tombstone count, and regularly performing compaction.

By following these best practices, you can ensure that your Cassandra cluster operates efficiently, and your applications perform optimally. Additionally, you can avoid the negative impacts of tombstones and keep your data consistent across all replicas.