Imagine you need to ingest billions of log entries per day, and you also need sub-second query response times on that data, not eventually, but right now, while the data is still arriving.
You can’t buy a server big enough to handle all that data. Even if you could, you wouldn’t want to. A single machine is a single point of failure, a maintenance nightmare, and a blunt instrument. Instead, what you really need is surgical control over dozens of interdependent services. Hydrolix is built on Kubernetes because it is the best architecture to meet these requirements reliably at scale.
This post will cover the basics of Kubernetes, how we use it, and how the decisions we’ve made allow us to run hundreds of customer clusters.
Kubernetes Is an OS for Distributed Systems
If you think of Linux or Windows as an operating system for a single machine, Kubernetes is the same idea applied to a collection of machines. This collection is also known as a cluster. Kubernetes provides a management layer and a set of abstractions over all the nodes in a cluster so you don’t have to think about which physical server a process is running on, or what version of Linux that particular machine has. Kubernetes handles all of that for you.
Here are some of the benefits that Kubernetes provides, and Hydrolix maximizes all of these benefits:
- Automatic recovery: Nodes will fail sometimes. Kubernetes detects failures and automatically redistributes the work.
- Load distribution: Incoming work gets spread across available resources automatically.
- Independent scaling: You can scale one service without impacting other services, avoiding resource contention.
- Rolling updates: You can update a service incrementally, one pod at a time, without taking anything offline.
- Consistent behavior across clouds: Whether you’re running in AWS, Google Cloud, Microsoft Azure, or Akamai Connected Cloud, Kubernetes gives you a consistent operational model. (Hydrolix runs on all major clouds.)
Kubernetes makes Hydrolix services scalable and resilient, so that even enterprises ingesting petabyte-scale data can be confident that they’ll have the data they need for use cases like observability and cybersecurity.
The Core Abstractions of Kubernetes
Before getting into Hydrolix-specific architecture, it helps to have a clear picture of the Kubernetes building blocks we work with most.
Pods are the smallest unit of compute that Kubernetes manages. Each Hydrolix service runs in one or more pods. A pod runs containers, which isolate CPU, memory, and storage from everything else on the same node. Pods are intentionally ephemeral. If one crashes, the Kubernetes control plane starts a replacement.
Nodes are the actual machines doing the work. Typically, these are virtual machines in a cloud environment where pods run. A node might run for six months without interruption. When Kubernetes needs to place a pod, it solves what’s essentially a bin-packing problem: given the resource requirements of this pod and the available capacity across all nodes, where does it fit most efficiently? It can also evict a pod from one node and move it to another if that makes better use of available resources.
Deployments manage groups of identical pods as a single unit. If a customer needs 15 query peers for fast queries, a deployment ensures that all 15 are running the same container image and configured exactly the same way. You can trust that any pod in a deployment is interchangeable with any other, except during a rolling update, when you’ll briefly have two versions in flight.
Hydrolix Services Using Kubernetes
Hydrolix isn’t a single application. It consists of more than thirty interconnected services, each with its own resource profile and scaling behavior. This post won’t go in detail about all of them, but here are a few major services.
- Intake heads handle streaming ingestion. During an ingest spike, ingest needs to scale up quickly.
- Batch heads and batch peers process files dropped into an object storage bucket. Batch heads coordinate while batch peers do the work.
- Query heads and query peers handle query execution. Query peers are memory-intensive, often requiring 64GB or more, so they run on larger nodes.
- Merge controllers and merge peers optimize how data is stored on disk after ingest.
- The alter service handles data modifications such as PII deletion requests.
These services don’t peak at the same time or for the same reasons. At 2 AM, you might have a massive ingest job running while query load is nearly zero. You want to scale intake heads up and let query infrastructure sit quiet. With Kubernetes, you can do exactly that. Without it, you’re provisioning for the worst-case combination of every workload at once and paying for that capacity around the clock. With Kubernetes, you can prevent overprovisioning and wasted resources.
The Hydrolix Operator: One File to Rule the Cluster
Kubernetes provides the infrastructure. But what tells Kubernetes what it needs to run, how many instances to maintain, and when to scale? That’s the job of the operator.
Hydrolix has a custom Kubernetes operator that provides a continuous reconciliation loop that compares the desired state of the cluster with the actual state, taking corrective action to close the gaps. You can add a pod to the desired count of a deployment and the operator provisions it. If a pod crashes unexpectedly, the operator replaces it. If ingest load spikes, the operator scales up intake servers.
All of this is driven by a single YAML configuration file: the Hydrolix cluster YAML. This includes important information about each customer cluster, including admin configuration, database bucket locations, per-service resource allocations, scaling ranges, and log verbosity settings. When an engineer is troubleshooting any issues that come up, the YAML is the starting point.
YAML changes go through a GitOps pipeline: edit the file, commit it to the repo, and the automated pipeline applies the change to the cluster. Every modification is tracked, auditable, and reversible. When you’re managing hundreds of clusters and jumping between customer environments, having an audit trail is absolutely essential.
Kubernetes at Super Bowl Scale
Consider what it takes to support a live streaming event at Super Bowl scale. The Super Bowl has millions of concurrent viewers, each generating a continuous stream of performance telemetry that includes buffering events, bitrate changes, errors, and CDN switches. All of it needs to be ingested in real time and queryable within seconds.
Before a major event like the Super Bowl, we don’t just assume the infrastructure will be there. We test the scale. If we think we’ll need a thousand nodes in a specific Google Cloud region, for example, we validate that the account has the quota and that the physical capacity actually exists in that region. Cloud infrastructure isn’t infinite. We’ve had detailed conversations with cloud providers about node availability before major events.
On event day, Kubernetes handles the orchestration. Intake heads scale up as ingest volume climbs, merge peers expand to keep data storage optimized as partitions are created to persist data , and the operator keeps everything in the desired state without anyone manually intervening. When the event ends, the cluster scales back down. For the 2025 Super Bowl, Hydrolix ingested over 200 TB of data for FOX Sports during the course of the event, with 55,000 queries made and peak ingestion rates of 17.4GB/second.
The ability to scale up, serve the event, scale down, and pay only for what you used is only possible with automated orchestration. There’s no manual equivalent.
What Kubernetes Enables for Hydrolix Customers
Our Kubernetes foundation helps support many features that are genuinely difficult for other platforms, and it’s core to our stateless, decoupled architecture. Kubernetes supports many Hydrolix features, including:
- Multi-year hot data retention: Storage and query infrastructure scale independently, so you’re not forced to trade off retention against query performance. You can keep everything and query any of it quickly.
- Sub-second query performance at massive scale: Query load is distributed automatically across query peers. You can add more peers when you need them and release them when you don’t.
- Cost efficiency: You pay only for the infrastructure you’re actually using, not a static allocation sized for peak load. At petabyte scale, the cost difference is significant.
- Multi-cloud deployment: The Kubernetes operational model is consistent regardless of which cloud you run on.
Behind the scenes, Kubernetes also makes it easier for our teams to manage hundreds of clusters, ensuring that you can scale up and rely on Hydrolix even when you’re managing petabytes of data.
Next Steps
Want to see how Hydrolix handles data at scale? Request a demo.
