During this year’s Super Bowl, Hydrolix ingested nearly 200 terabytes of data in the span of a few hours for FOX Sports—with a 10 second time to glass for queries. How does a platform like Hydrolix keep such a stream flowing smoothly without constantly fighting fires?
Enter the Hydrolix Operator–the hero orchestrating all the moving parts of the Hydrolix platform behind the scenes. In plain terms, this operator is a specialized piece of software running in a Hydrolix Kubernetes cluster that automates the deployment and management of the entire Hydrolix data platform.
In this blog post, we’re pulling back the curtain on how the Hydrolix Operator works. We’ll demystify what it is and why it matters in friendly, user-oriented language. We’ll explore how the infrastructure of Hydrolix is built on Kubernetes, and why Hydrolix chose this approach for our platform. We’ll also peek under the hood of the Hydrolix Operator’s architecture and show how our operator tackles complex operational challenges so that users can scale to ingest petabyte-scale volumes of log data.
If you’re interested in learning more about the operator pattern in Kubernetes, check out our post on The New Stack, which covers the basics of Kubernetes operators, how they’re currently being used in the field, and some of the benefits we get from using the operator pattern.
Kubernetes Architecture in Hydrolix
Hydrolix is a cloud-native, real-time data platform designed to handle truly colossal data volumes. At a high level, the Hydrolix platform consists of many specialized components that work in concert to ingest, store, and query data at scale. These include ingestion services for streaming data, indexing and compression services, query engines, and auxiliary services for data transformation, retention, and more.
The architecture of Hydrolix is Kubernetes-native, meaning it runs on a Kubernetes cluster and takes advantage of Kubernetes features for scheduling, scaling, and resilience. This design allows independent scaling of ingest, query, and storage components, so each part can grow as needed without bottlenecking other components. In practice, that means if your ingestion load spikes (say, during a big game), Hydrolix can scale out just the ingest tier, while query and storage remain optimally sized–leading to cost savings and efficient resource use.
All Hydrolix microservices are containerized and deployed as pods in a Kubernetes cluster. Kubernetes’ job is to keep the pods running (including restarting pods that fail and distributing pods across nodes), while the job of Hydrolix is to deliver the data-crunching functionality. This is where the Hydrolix Operator comes in. It’s the component that knows which pods (and how many) to run for a given Hydrolix deployment and uses Kubernetes as the engine to run them. Coordinating all these components manually would be a headache, which is exactly why we use an operator to automate it.
The Hydrolix Operator: An Architectural Overview
Let’s dive into how the Hydrolix Operator is built and how it interacts with the Hydrolix platform. The operator runs as a controller pod in your Kubernetes cluster. When it starts up, one of the first things it does is register Custom Resource Definitions for Hydrolix-specific objects–most importantly, the HydrolixCluster CRD. This CRD defines the schema for describing a Hydrolix deployment. It has all the knobs and dials for the platform, including:
- Number of replicas for each service
- Resource requests (CPU/memory) for pods
- Cloud storage configurations
- Security settings
See the Hydrolix tunables doc for more on what can be tuned. You can think of the HydrolixCluster spec as the blueprint or master config for your Hydrolix cluster.
Once you apply a HydrolixCluster object, the operator’s reconcile loop kicks in. The operator continuously watches the Kubernetes API for any HydrolixCluster resources (create, update, or delete events). On seeing one, it will compare the desired state (from the spec) to the actual state (existing Kubernetes resources running).
If you just created a new cluster spec, the actual state is empty, so the operator proceeds to create all the needed Kubernetes resources to fulfill the spec. This means creating Deployments for each Hydrolix microservice, Services (and load balancers) for networking, ConfigMaps or Secrets for configuration values, and so on. Each component of Hydrolix (including ingest, query, alter, and merge) is defined within the operator, so it knows which pods to start and how they should be configured to work together.
The operator acts as a translator between the Hydrolix world and the Kubernetes world. For example, if your HydrolixCluster spec says "scale_profile: dev" (a smaller cluster preset), the operator knows that means something like 2 query pods and 1 ingest pod with fewer resources defined. It will scale to create exactly those.
If you later edit the spec to use a larger profile or custom settings (say, increase query-peer replicas from 2 to 5), the operator will update the deployments, adding the extra pods and adjusting configurations, until the cluster matches the new spec. All of this happens via standard Kubernetes APIs. The operator issues the create/update/delete calls and Kubernetes does the heavy lifting of actually running the containers.
Key components of the Hydrolix Operator’s architecture include:
- Custom Resource & CRD: Defines the desired state (
HydrolixCluster). This is the only thing the user manages directly. - Controller Loop: The operator process that watches for changes and reconciles the state. It’s event-driven (triggered by changes) but also does periodic checks.
- Resource Templates: The operator has built-in templates or code for each Hydrolix component’s Kubernetes objects. This ensures each service is configured with logical defaults. For example, you don’t have to write the YAML configuration for a merge-peer deployment because the operator handles it.
In effect, the Hydrolix Operator encapsulates the entire deployment architecture of Hydrolix. It knows how the pieces fit together (for example, that Traefik proxy and load balancer service must be deployed and configured to route incoming queries/ingest, or that Prometheus and Grafana should be set up for monitoring). It spins those up as part of the cluster. If Hydrolix introduces a new microservice in a future version, the operator is updated to deploy it automatically when you upgrade. This tight integration means the operator is always in lockstep with the Hydrolix platform version, handling the complexity so you don’t have to.
Tackling Complex Problems with the Hydrolix Operator
Let’s look at some specific challenges that the Hydrolix Operator is built to solve. These are challenges that would be painful if they had to be handled manually.
Full Lifecycle Management
The operator isn’t just a deployment tool; it manages the entire lifecycle of a Hydrolix cluster. This includes installation, configuration changes, scaling, and teardown. When installing, the operator handles initialization tasks such as init-cluster jobs that run once to set up metadata and then exit gracefully. There’s no need to run extra setup scripts. For configuration changes, as mentioned, the operator detects updates to the HydrolixCluster spec and rolls them out. And if you delete a HydrolixCluster, the operator will clean up all the Kubernetes resources, ensuring no orphaned pieces are left behind. This management means you have a single control point for the life of your deployment.
Rolling Ordered Upgrades
Upgrading a distributed system can be tricky. You might need to bring down services in a certain order or ensure one component is updated before another. The Hydrolix Operator is designed to ensure upgrades happen smoothly. When you want to upgrade Hydrolix to a new version, you typically update the operator, which in turn knows how to deploy the new version’s components. The operator can then orchestrate a rolling upgrade of the cluster. If the new version requires a database migration, the operator can handle running that migration job at the right moment. This kind of coordinated upgrade process is encoded in the operator and reduces human error and downtime.
Diffing Actual Versus Desired State
One of the core jobs of any operator is reconciling differences between what should be and what is. The Hydrolix Operator takes this seriously. On every reconcile loop, it computes what the desired state from the HydrolixCluster spec looks like in terms of Kubernetes resources, then checks the live cluster to see if that matches. If a difference is found, it will take action to fix it. If someone manually edited a ConfigMap that the spec manages, the operator will overwrite it back to the spec-defined value. This way, the integrity of the deployment is maintained.
Handling Dependencies and Complex Scenarios
The Hydrolix Operator also tackles nuanced operational problems. For instance, Hydrolix has a component called Traefik (an open-source ingress controller) that it uses for routing API requests, along with a sidecar called traefik-cfg that updates Traefik’s config. The operator ensures that Traefik is deployed and that traefik-cfg restarts whenever there are configuration changes, so new endpoints are registered. You don’t have to worry about it—it just works.
Data lifecycle tasks are another example. Hydrolix has services like decay (to age out old data) and reaper (to delete retired data). These need to run periodically and reliably. The operator makes sure these pods are always running on the correct schedule and have the right permissions to do their jobs.
Real-world proof of these capabilities came during the 2025 Super Bowl. With Hydrolix ingesting 17.4 GB of log data per second at peak, the operator helped ensure the cluster scaled appropriately and none of the critical ingest or query pods fell behind. When we say the operator tackles complex problems, that includes keeping Hydrolix running even in extreme conditions.
Managing the Kubernetes Lifecycle With the Operator
From day one (initial install) to day “who knows” of running Hydrolix, the operator will continue to manage many aspects such as monitoring, upgrades, and maintenance. Let’s walk through what the operator manages and how you can monitor a Hydrolix cluster’s health.
Quickly Install and Deploy Clusters
Installing Hydrolix on a new Kubernetes cluster is as easy as deploying the Hydrolix Operator and applying a cluster spec. The operator takes care of provisioning everything. It can feel a bit magical–apply one YAML and then watch as dozens of pods spring to life, each one fulfilling a role in the Hydrolix platform. By the time the install is finished, you’ve got a fully functional platform running without having manually started a single service yourself.
Provide Monitoring and Status Updates
Once running, both Kubernetes and the Hydrolix Operator provide insight into the running cluster. You can use standard kubectl commands (such as kubectl get pods) to see that all expected pods are in the “Running” state. Hydrolix bundles Prometheus for metrics and various visualization tools like Grafana, Kibana, and Superset for dashboards, providing out-of-the-box monitoring–including for the operator itself. In Grafana, you can have a dashboard that shows cluster health. This includes visualizations of metrics like ingest rate versus pod CPU and number of query requests over time.
If something goes wrong, info on readiness status and logs from the operator pod will often tell the story. We also provide a CLI tool (hkt) which can show cluster config and status information, so you can quickly pinpoint settings or dump out a config report of what the operator is enforcing.
The operator’s built-in observability ensures that the operator works on your behalf and you have confidence in the cluster’s status. The combination of Kubernetes events, operator logs, and metrics allows you to set up alerts to catch issues early. Examples might be if any Hydrolix pods keep restarting (or if reconciliation fails). All of this reduces the operational burden and lets you rely on automation without losing transparency.
Manage Upgrades and Rollbacks
When it’s time to upgrade Hydrolix to a newer version, the operator makes the upgrade a controlled process. Typically, you would upgrade the operator itself (to the new version’s operator image) and, if necessary, update the HydrolixCluster with the new Tunables. The operator will then orchestrate the upgrade, rolling it out service by service. The operator’s design avoids “big bang” upgrades where everything goes down at once. Instead, it strives for zero-downtime updates by using Kubernetes rolling update mechanisms and its own dependency knowledge.
Automate Ongoing Maintenance
Over time, you might need to adjust configurations, tweak retention periods, or scale certain parts of a Hydrolix cluster up or down. Because of the Hydrolix Operator, these are simple spec changes and the operator will gracefully applies them. The Hydrolix Operator even helps with things like horizontal scaling events. If you run the Hydrolix cluster in a cloud environment where nodes scale up, the operator will automatically schedule new pods on the new nodes as needed. Essentially, it turns the routine maintenance of the platform into a hands-off experience. And if you ever want to truly pause or shut down, you can apply the scale_off: true Tunable. You can even simply delete the HydrolixCluster because the operator will terminate pods in the correct order.
Conclusion
The Hydrolix Operator might not be the flashiest part of our platform, but it is absolutely critical for delivering the smooth, scalable experience our users expect. We’re proud to be using the operator pattern to bring next-level automation to data infrastructure. With features like automated lifecycle management and self-tuning capabilities on the horizon, and the hard-earned lessons of running real-world, high-scale workloads baked in, the Hydrolix Operator will continue to be a core part of our success. We hope this overview has shed some light on how it all works in approachable terms. After all, you shouldn’t need a PhD in Kubernetes to benefit from its power. Whether you’re new to the concept of operators or have used them before, one thing is certain: operators can make Kubernetes clusters much easier to run. The Hydrolix Operator helps Hydrolix users store more, query more, and worry less. Happy data adventures!
