The Hydrolix Merge Service: A Deep Dive

The fastest way to ingest data produces the worst layout for querying it.

To scale write throughput, we write files concurrently across many processes. To make that data available for query as soon as possible, we flush early and often. The result is many small files. But queries perform best against fewer, larger files, where the engine spends its time processing data rather than hunting for it. This is the small files problem, and at Hydrolix scale it needs to be resolved continuously, without interfering with ingest or query. That’s the responsibility of the merge service.

Merge has one job: manage the number of partitions (the fundamental storage units in Hydrolix) per queryable time range. One hour is the window that we’ve found works best. Within that window, the balancing act looks like this at a basic level:

• Enough large partitions to utilize the available query parallelism in a deployment.
• Not so many small partitions that visiting them is mostly overhead.

This post will cover the evolution of our merge service and how it works. If you’re less interested in the history and want to know how it works now, jump directly to how it works.

The Evolution of Merge

Merge started as a simple operation: combining groups of partitions, with little resemblance to traditional compaction. What began as a byproduct of ingest evolved through four architectural phases into a system with similar goals to traditional compaction but fundamentally different constraints.

Before Merge

Before a dedicated merge service existed, the ingest service handled partition shaping on its own. Each ingest instance enforced the same constraints that merge does today (size limits, time boundaries, etc.) and at early scale, the best effort of each individual instance produced reasonable partition shapes and counts.

This worked when replica counts were low. But as capabilities expanded and throughput requirements grew, so too did the number of ingest replicas. Each additional replica independently produced partitions for the same time ranges with no coordination between them. Scaling ingest from 3 replicas to 30 didn’t produce the same partitions faster. Instead, it

produced ten times as many. Without a separate service to consolidate them, partition counts grew unbounded with every scaling event.

The Lambda Era

The first merge system shipped just over five years ago. At that time, Hydrolix was AWS-only and pre-Kubernetes. The system consisted of two services, merge-head and merge-peer, using AWS Lambda for execution and SQS for communication. Ingest services still performed their own best-effort partition creation, but were modified to emit events to an SQS

queue whenever partitions were created. merge-head consumed these events and produced merge candidates—a list of partitions that met all the criteria to be merged—to another SQS queue. merge-peer workers consumed these candidates and performed the actual merge operations.

The Recursive Era

The Lambda-era system worked, but it exposed a fundamental limitation: it only handled live data well. When all incoming data is temporally close, grouping partitions into merge candidates is straightforward. But Hydrolix’s ingest capabilities were expanding: batch processing, Kafka consumption, historical backfills. These methods produce partitions spanning arbitrary time ranges. An event-driven merge-head had no efficient way to track events across a wide time range in hopes of eventually assembling a well-shaped candidate.

The fix was to change how merge-head found its work. Instead of reacting to partition-creation events, it began periodically inspecting the catalog (our partition metadata store) to find partitions from any time range that could benefit from being merged.

This shift had an unexpected benefit beyond solving the late-arriving data problem. In the event-driven model, merge was one-shot: each event covered a single new partition, and that was the only opportunity to merge that partition.

With periodic catalog scanning, the output of a previous merge is itself visible the next time the catalog is inspected. Partitions could be merged incrementally, from small into medium, and medium into large, with each cycle reducing partition count further. This recursive merging became a core property of the system.

The Scaling Era

The scaling era was the longest period without substantial architectural changes to the merge system, and it saw the largest increase in scale, with average cluster ingest volume growing 10-50x. The infrastructure underneath changed significantly: Hydrolix adopted Kubernetes, and merge transitioned from AWS Lambda to Kubernetes deployments communicating via RabbitMQ. This made the system cloud-agnostic, running on AWS, Azure, GCP, and Linode.

But larger scale exposed another deficiency: lack of coordination. With merge-head and merge-peers communicating through a queue, merge-head had no visibility into what work peers were currently performing. It operated pessimistically. Each cycle, it produced candidates assuming they had not been suggested before, because no information was available to suggest otherwise.

The final step of adding a merged partition to the catalog is atomic. If two peers independently merge the same candidate, the first to write to the catalog succeeds and the others are rejected, maintaining data consistency but wasting compute. At small scales, this timing conflict is rare. At the increasingly common large scales, it became frequent and expensive: peers spending significant resources on merge operations only to have the result discarded.

The Controller Era

The current merge service is not an incremental improvement. It is a ground-up rewrite with a complete re-imagining of the architecture. The queue is gone. merge-controller communicates with every merge-peer over direct, persistent gRPC connections, giving the controller full visibility into the state of every active merge operation and solving the coordination problem that defined the scaling era.

How It Works

Persistent Peer Connections

The queue-based architecture gave the controller no visibility beyond submitting work. The current architecture replaces the queue with direct, persistent connections between merge-controller and every merge-peer. The controller sends candidates to peers and receives continuous feedback: current operation state, progress, and results.

Whether a merge succeeds or fails, the controller knows immediately.

This gives the controller a consistent, real-time view of the entire merge system. It knows which candidates are in flight, how much capacity is available across the fleet, and whether operations are succeeding or failing. That view is what enables the controller to eliminate duplicate work, size candidates to available resources, and provide a clear signal for scaling. The elimination of duplicate work alone reduces resource requirements by 10-60% for the same volume of merge operations.

Online Bin Packing

The heart of the controller is a custom online bin-packing algorithm optimized for time to glass (the elapsed time from ingestion to when data is queryable), not packing density. Standard bin-packing algorithms focus on fitting items into containers as tightly as possible. The Hydrolix controller operates under a different constraint. You cannot wait indefinitely for the perfect combination of partitions to reach a 4 GB target while small files degrade query performance.

Instead, the algorithm uses timeout-based rules:

• Idle timeout: If a group of partitions sits for 15 seconds without new data arriving, merge it.
• Open timeout: If a group has been open too long, merge it regardless.

Both rules prioritize query speed over compression density. Because partitions undergo multiple merge cycles as they age, they naturally achieve better compression over time as data within each hour becomes more contiguous. The system also handles late-arriving data regardless of age, whether it arrives hours, days or months after its primary timestamp.

Tiered Merge Pools

Merge uses three tiers that progressively shift their optimization target. Merge-i prioritizes time to glass for newly ingested data. Merge-iii prioritizes long-term query efficiency with larger, better-compressed partitions.

Separate pools exist for two reasons. First, resource efficiency: larger merges require more memory and compute, so sizing all pods uniformly wastes resources on smaller operations. Second, independent scaling: the throughput demands of merge-i (driven by current ingest volume) are temporally independent of merge-iii (driven by accumulated partition history). Each tier scales based on its own workload, with no contention between tiers or with ingest and query.

This also enables deployment flexibility. A cost-conscious deployment, or one that rarely queries the head of incoming data, uses fewer, larger-memory peers that merge small partitions directly into large ones. A latency-sensitive deployment, which is the default, uses all three tiers for incremental, throughput-optimized merging over the head of incoming data.

The Merge Process

The controller reads partition metadata from the catalog: minimum and maximum timestamps, and mem_size, which is the estimated memory required to open each partition. This is the decompressed memory footprint, not the compressed file size on disk, and the controller uses it to plan work within hardware limits. It groups candidates within the same hour until their total mem_size reaches the tier’s target (1 GB for merge-i, 4 GB for merge-iii by default). The peer then opens the source partitions, combines them, and writes a single new partition to object storage.

Data in Hydrolix is append-only and immutable. Merge does not modify partitions in place. It writes an entirely new partition and atomically updates the catalog to replace the sources.

What’s Next

The ground-up rewrite was not only a response to the architectural limitations of previous iterations. It was designed to incorporate operational learnings and position the system for future improvement. The persistent connection between merge-controller and every peer already provides more than coordination: the controller receives detailed results from every merge operation, including operation timing and resource utilization.

This data opens the door to dynamic tuning. Today, peer shapes and scheduling parameters are configured statically. With real-time operational feedback, future iterations can adjust peer sizing and operation scheduling dynamically, allocating resources where they have the most impact as throughput demands change.

Conclusion

Ingest performance and query performance are fundamentally at odds. The merge service turns this from a constraint into an independently scalable concern that is stateless, decoupled from ingest and query, and continuously running. It is not a background task. It is a core part of what makes real-time analytics at petabyte scale work in Hydrolix.

Next Steps

Interested in learning more about Hydrolix for petabyte-scale data? Request a demo.