When you’re analyzing data in real time, every second counts. You need both fast writes (time to glass) and fast reads (real-time queries). But when it comes to storing data, these goals are at odds.
The fastest way to ingest data is to write many small files because small partitions move quickly to storage and become available for queries in seconds. But the fastest way to query data is the opposite: fewer, larger files mean fewer round trips to object storage and better compression ratios.
This creates a challenge for engineering teams. Optimize for ingest speed, and query performance degrades when you have too many partitions. Optimize for query efficiency, and you introduce latency before data ever becomes available, or you add expensive reprocessing pipelines to reorganize data after it’s written to storage.
Hydrolix solves this problem with a merge service that runs continuously in the background, automatically creating larger, more efficient partitions over time, all without interrupting other systems like query and ingest. The merge service is one of the key architectural components that enables Hydrolix to deliver both real-time performance and cost-effective long-term storage at petabyte scale.
This post explains the basics of how it works, why it’s architecturally significant, and what it means for organizations running real-time analytics on massive volumes of data.
Interested in learning more about the merge service from an engineering perspective? Read about the evolution of the merge service and get a deep dive into how it works.
Why the Merge Service Is Essential
When Hydrolix ingests streaming data, it creates partitions that are intentionally small, with a default limit of 5 seconds or 1 million rows, whichever comes first. This design prioritizes making data available for query as fast as possible. In high-volume scenarios where Hydrolix might be ingesting millions of log lines per second, this can lead to thousands of small partitions.
These micro-partitions ensure data reaches storage quickly for real-time analytics. However, smaller partitions aren’t effective for long-term query performance and storage efficiency.
Querying thousands of small partitions means making thousands of HTTP requests to object storage. Each partition has overhead in terms of metadata, indexing, and retrieval costs.
Larger partitions means fewer HTTP requests. It also means better storage because compression efficiency improves significantly with larger datasets. Larger datasets mean more repeated values. And more repeated values leads to better compression using techniques like dictionary encoding and run-length encoding.
How the Merge Service Works
The merge service is an independent, stateless subsystem in Hydrolix’s cloud-native Kubernetes architecture. Like other Hydrolix components, it scales independently and doesn’t interfere with ingest or query operations, even during periods of high load.
The Immutable Merge Process
The merge process is immutable, meaning it never modifies existing partitions. Instead, it reads data from multiple small partitions, combines and optimizes the data, then writes the new, larger partitions to object storage. Once the new partitions are successfully written to object storage, they’re added to the catalog while the old partitions are marked as inactive in an atomic operation. This approach ensures data integrity. If a merge operation fails midway, the original partitions remain untouched and available for querying.
The merge head continuously queries the catalog, which contains basic information about all partitions, to identify candidates for merging. This metadata includes information like the minimum and maximum timestamp of each partition, its size, row count, and storage location.
Three-Tier Merge Pool Architecture
Hydrolix uses a three-pool merge architecture to handle partitions at different stages of their lifecycle. The small, medium, and large merge pools each handle partitions based on criteria like age, size, and time window.
This tiered approach ensures optimal partition sizing and distributes merge workloads across both old and new data. Recent data might be merged from 5-second partitions into 1-minute partitions, then later into 5-minute partitions. As data ages further and is queried less frequently, it can be merged into even larger partitions—up to 4GB or 1 hour in size.
Read more about tiered merge pools.
This gradual compaction strategy balances competing needs: keeping recent data in smaller partitions for better parallelism during frequent queries, while consolidating older data into larger partitions for maximum compression and minimal storage overhead.
Solving the Late-Arriving Data Problem
Data can arrive late for numerous reasons: network latency, service outages, dead batteries in IoT sensors, preprocessing delays, or simply the distributed nature of modern systems. This is a major challenge for many real-time analytics platforms, which aren’t designed to handle late-arriving data.
Hydrolix is designed to efficiently handle late-arriving data, and the merge service is a significant reason why. Hydrolix accepts late-arriving data and initially writes it to small partitions just like fresh data. However, these partitions often contain out-of-order data, with events sorted by ingest time rather than event time.
The merge service automatically sorts data by primary timestamp regardless of whether it’s ingested in real time or hours, days, or even months late. Partitions that contain out-of-order or late-arriving data are eventually sorted into larger, more efficient partitions regardless of how late they arrive.
For many platforms, handling late-arriving data requires complex ELT pipelines, reprocessing entire tables, or relying on expensive transactional databases. Some platforms simply discard late data, leading to inaccurate datasets. Hydrolix’s merge service eliminates these problems entirely.
Benefits of Continuous Merging
There are numerous benefits to continuous merging, including:
- Enhanced compression: As partitions grow larger through merging, compression ratios improve significantly. Columnar compression techniques like run-length encoding become more effective with more data. Fields like HTTP status codes, request types, or data sources that have many repeated values compress extremely well in larger partitions.
- Improved query performance: Fewer, larger partitions mean fewer HTTP requests to object storage during queries. When combined with Hydrolix’s partition pruning (using the catalog to pinpoint which partitions can be eliminated from query consideration) and predicate pushdown (retrieving only specific byte ranges), query performance improves dramatically as data ages and gets merged.
- Automatic data optimization: Late-arriving data gets sorted, out-of-order events are put in order, and suboptimal partitions created during ingestion spikes are reorganized into efficient partitions.
- Cost reduction: Better compression means lower storage costs. More efficient partitions mean less compute needed for queries.
Next Steps
The Hydrolix merge service is essential infrastructure that manages the tradeoff between fast writes at ingest time and fast analytical queries. And with merge continuing to optimize partitions over time and increase compression, it also helps make long-term retention more cost-efficient. With Hydrolix, you can keep full-fidelity data for 15 months or more, all while keeping that data hot for queries.
Interested in learning more about Hydrolix for petabyte-scale data? Request a demo.
