High-Density Compression in Hydrolix

Compression is a cornerstone of Hydrolix’s architecture, with log data compression rates typically over 90% and up to 99%. This level of compression doesn’t just help users save money on storage. It also makes it possible for Hydrolix to use S3-compatible storage for both real-time and historical analytics. No other platform offers the combination of cost-effectiveness, scalability, and performance that Hydrolix offers, and it all starts with compression.

Hydrolix achieves these compression rates through a combination of well-known compression techniques and novel approaches. The well-known techniques include algorithms such as run-length encoding, dictionary encoding, and bit-packing. This isn’t an exhaustive list, and the particulars of how these techniques are combined and used in novel ways are mostly beyond the scope of this post.

This post will provide a high-level overview of compression in Hydrolix, including:

• How Hydrolix compresses data at ingest time
• A few common techniques for compressing columnar storage
• How Hydrolix’s merge service optimizes and compresses data further over time

Check out this post on the benefits of compression to better understand how it enables cost-effective real-time analytics. 

Data Transformation and Compression at Ingest Time

Let’s start by taking a look at how Hydrolix handles compression at ingest time. Hydrolix streams data and performs the following steps before storage. Some of these steps are entwined—for instance, as data is written to a partition, it’s both indexed and compressed.

• Transformation: Hydrolix transforms and standardizes data based on user-defined configurations. Data standardization has the beneficial side effect of more efficient compression.
• Partitioning: Hydrolix writes data to partitions that consists of three files: a manifest, an index, and the raw data file.
• Indexing: Hydrolix indexes all columns by default. With most databases, indexing increases the size of the overall data. That is not the case with Hydrolix because data is compressed while it’s being indexed and lists are stored in simple list structures.
• Compression: Hydrolix compresses all files in a partition, including the manifest and index. Raw data compression rates are typically 90-99%.

The target “time to glass” (from ingest to actionable insights) is under ten seconds, even for events generating millions of log lines per second.

Let’s take a look at how standardizing data improves compression.

Standardizing Data Improves Compression

During the data transformation process, Hydrolix transforms all incoming data based on a user-specified transform file for each data source. Let’s look at a quick example of how data transformation can greatly improve compression.

Hydrolix has the unique ability to ingest multiple data sources into a single table. A common use case is combining data from multiple CDNs into a single table to make that data easier to analyze and compare.

However, different sources often have different column names to describe the same thing. For example, one CDN source might use status_code while another uses statusCode. With Hydrolix, you can input the data from both sources into a single column with a standardized name.

As a result, you’ll have a single column, and if the column has low cardinality (many repeated values), you’ll achieve much greater columnar compression. The next section covers columnar compression in greater detail.

Even with high-cardinality columns that are less compressible, a smart optimization during the transform process can dramatically reduce storage space. Read to learn more about how analyzing column cardinality helped a customer optimize a high-cardinality column and reduce costs.

Data Sorting and Specialized Compression

After the transformation process, Hydrolix creates a partition using the data from the local file. Every Hydrolix partition has three files:

• manifest.hdx: Acts as a table of contents to the partition
• index.hdx: Hydrolix indexes all columns in the partition using sparse indexing
• data.hdx: File that holds data

Hydrolix compresses all three files using specialized algorithms.

Many platforms compress data before storage, but they often apply a general compression algorithm to all incoming data. While this is a lot better than nothing, it’s not nearly as effective as using specialized compression algorithms for each data type.

Hydrolix’s compression strategy involves sorting data for more efficient compression, then applying the best compression algorithm for each data type. This tailored approach provides higher compression rates than an all-purpose algorithm designed for all data types.

These specialized algorithms include many optimizations for greater compression. For example, in a set of integers such as 3, 15, 72, 929332342, 12, large outliers like 929332342 will negatively impact compression for the entire column unless they are handled separately. By moving outliers to an exception table, Hydrolix improves compression. The exact implementation for handling outliers is proprietary.

Compressing Values in Columnar Storage

Hydrolix stores data in a columnar format, which is much more compressible than row-based storage. Let’s take a look at some of the reasons why—and why it’s an especially good format for log data.

Unlike rows, columns typically store the same type of data such as integers or strings. As discussed in the previous section, you can use specialized compression algorithms for each data type instead of a general compression algorithm that doesn’t achieve the same rate of compression. This isn’t possible with row-based storage.

A few examples of compression algorithms that work well with low-cardinality tables include run-length encoding and dictionary encoding.

Run-Length Encoding

Columns can often store repeated values, and this holds true for log data. For example, there are a limited number of HTTP status codes with the most common (hopefully) being 200 status codes. Hydrolix uses well-known techniques like run-length encoding to compress repeated values. Here’s a very generalized example of how run-length encoding works. This example shows a column of status codes:

These values can be sorted and compressed using run-length encoding, which would store the binary equivalent of something like this:

200 -> 5

400 -> 1

404 -> 1

The value and the frequency of each value are encoded. For columns that have many rows, techniques like run-length encoding can achieve high rates of compression. And as discussed in the previous section, you can use the transform process to standardize columns across multiple data sources.

For example, if you were ingesting data from five different CDNs into a single table, all reporting the same range of HTTP status codes, you’ll get greater compression than you would if you instead ingested that data into five separate tables.

Read about Hydrolix’s unique ability to ingest multiple data sources into a single table.

Dictionary Encoding

Another well-known technique Hydrolix uses to achieve greater compression on columns with lower cardinality is dictionary encoding. With dictionary encoding, the position of each unique value is stored in a dictionary.

A dictionary representation would look something like this:

Dictionary encoding isn’t as effective for high-cardinality tables with many unique values. That’s because you’ll end up with a huge dictionary that has many unique keys, leading to dictionaries that take up a lot of space. However, since many columns have repeated values in log data, these algorithms can lead to extremely efficient compression rates.

Merge Service Improves Compression

A table in Hydrolix consists of many partitions, each with its own index. Hydrolix includes a merge service that automatically runs in the background, optimizing data partitions over time. Because each part of Hydrolix’s architecture is stateless and decoupled, the merge service doesn’t interfere with ingest, query, or other systems. And there is no need to rebalance or manage the merge service.

At ingest time, data partitions are both small and highly compressed. Small partitions can be written to storage more quickly, reducing the amount of time for users to get actionable insights.

Over time, Hydrolix’s merge service “merges” smaller partitions into larger partitions, leading to greater compression. In the case of Hydrolix, the term merge is a bit misleading because it suggests that Hydrolix simply combines partitions. Rather, the process is immutable. The merge service creates a new partition and then copies data from the original partitions to the new partition. Finally, the service removes the old partitions.

The following simple diagram uses an example of run-length encoding to show how merging partitions can result in greater compression. Let’s say that there are three partitions with similar timestamp ranges that will eventually become a single larger partition. Each of the original partitions includes a column that has many instances of the values A and B.

Originally, data representing the values A and B takes up space in each of the three partitions. After multiple merge processes, though, that information will be in one partition instead of three, leading to greater compression.

While this is a simplistic example, it demonstrates one way that Hydrolix’s merge service continues to optimize and compress data over time.

Decompressing Data at Query Time

Compression algorithms often have tradeoffs between compressibility and performance. In the case of performance, some algorithms take longer to compress and decompress data. Hydrolix partitions are designed to minimize the performance tradeoffs that come with high-density compression.

Partitions can be opened quickly and allow for partial reading to minimize the amount of data that needs to be decompressed at query time. Hydrolix’s query infrastructure can check a partition’s index file to see if the partition incudes data that meets the query conditions. If not, there is no need to decompress any raw data.

If a partition contains data that meets the query conditions, Hydrolix’s query infrastructure retrieves index data and raw data in parallel. Because all columns are indexed, Hydrolix can retrieve just the narrow byte ranges that fit the query conditions, a technique known as predicate pushdown.

Once the compressed raw data is retrieved through HTTP, it’s decompressed locally.

The result—sub-second query latency for time-filtered queries, even on very large datasets—and minimal overhead from decompressing data.

Read more about how Hydrolix achieves sub-second query latency even on 100+ billion row datasets.

Conclusion

Compression provides a lot of benefits to the end user, including a smaller storage footprint and reduced costs. And with Hydrolix, high-density compression makes it possible to have performant and cost-effective real-time analytics while using S3-compatible storage. You can keep more data for much longer and at a lower cost—all with the ability to scale horizontally regardless of the volume of your log data.

Next Steps

• Read this post on the benefits of compression to better understand how it enables cost-effective real-time analytics. 
• If you’re not using Hydrolix yet and would like to learn more, contact Hydrolix about a managed trial or demo.