Real-time analytics dashboards include critical metrics like average request time, total requests, and quantiles for request latency. All of these metrics have something in common: they involve aggregating raw data (averages, sums, and quantiles).
When you’re working with large volumes of data, it’s a best practice to create separate summary tables for regularly-queried data such as aggregates. A dashboard that refreshes every ten seconds is making nearly ten thousand requests per day. It’s much more efficient to query a separate table than to query and reaggregate the underlying raw data. However, many platforms have challenges keeping aggregated data accurate for real-time analytics, especially at scale. And some platforms can’t update aggregates with late-arriving data, which leads to yet more issues with accuracy.
Hydrolix is designed for these kinds of challenges. You can use summary tables in Hydrolix to store complex aggregations as well as non-aggregated data that you query frequently. And summary tables in Hydrolix also account for late-arriving data. Summary tables have long been a powerful feature of Hydrolix. Recently, our engineering team updated the underlying architecture of summary tables to make them more effective and accurate.
Now they are even more performant and efficient. Summary tables reaggregate data without querying the underlying raw data. The addition of intermediate states, which hold state for more complex aggregations like averages, total uniques, and quantiles, are at the heart of this update.
With intermediate states, the summary service reads data for summary tables only once from raw data partitions at ingest time. There’s no need for summary tables to retrieve raw data multiple times, even for complex aggregations.
Intermediate states also allow users to create new aggregations from summary tables instead of querying the underlying raw data. For example, you can combine 24 one-hour aggregates into a one-day aggregate even for complex aggregations like uniques and quantiles.
The benefit for users is improved performance and the ability to slice and dice aggregate data in new ways. This post will cover:
- A quick overview of the benefits of Hydrolix summary tables
- Intermediate states and the problems they solve
- How Hydrolix summary tables work in our updated architecture
TL;DR: Why Use Hydrolix Summary Tables?
Here’s a quick overview of summary tables in Hydrolix which includes functionality we’ve added this year.
- Accurate, efficient summary data. Summary tables read from source data only once at ingest time while remaining accurate because aggregate columns hold intermediate state.
- Summarize all the data you need. Hydrolix is a columnar datastore that uses high-density compression, so summary tables can be “wide” (high dimensionality) and high cardinality, allowing you to summarize a lot of data efficiently.
- Easily slice and recalculate your aggregated data. With intermediate states, you can easily calculate new aggregates based on summary table data without querying your raw data.
- Manage summary tables in the UI. Hydrolix now includes the option to easily create and manage summary tables in the UI.
- Summary tables account for late-arriving data. Hydrolix handles late-arriving data by design, including in summary tables, no matter how late the data arrives.
Understanding Intermediate States
Some aggregations are easy to recalculate such as basic sums, minimums, and maximums. Other aggregations such as averages and percentiles require holding more complex state to remain accurate. In the case of aggregations like total uniques, you might need to retain millions of values to ensure accuracy.
Let’s jump right into an example that requires a lot of state: calculating the total number of unique IP addresses over a period of time. This is a very important metric for monitoring CDN performance, detecting DDoS attacks, determining user geos, and more.
An application receiving peak traffic or dealing with a DDoS attack can receive requests from millions of IP addresses in a few minutes. As an example, two separate one-minute aggregations might each have a million unique IPs. But if you were to determine the union of unique IPs in these two aggregations, there would be repeats. The total number of uniques over that two-minute period isn’t the sum of each minute’s total unique IPs.
For that reason, an intermediate state for uniques must hold every single unique value.
Without intermediate states, you’d need to recalculate total uniques by querying the raw data. For a real-time dashboard that updates every minute, that’s nearly 1500 queries a day. That defeats the point of having a summary table for more efficient queries.
If you wanted to see the total number of unique IPs over an hour instead, you wouldn’t be able to gather that information from the existing one-minute aggregates without intermediate state. You either need to query the underlying data or have yet another table, leading to more data duplication and resource usage.
Alternatively, you can do fewer roll-ups and have tables that hold stale aggregations and are less accurate, which is also problematic. For example, if you did five-minute roll ups instead, that data will quickly become stale. And you’d still have to query the underlying data every five minutes.
Intermediate states solve these problems. They store all the state you need to recalculate aggregations. This is useful for several reasons:
- Summary tables don’t need to query the raw data to recalculate aggregations when new data arrives, leading to better overall performance and lower compute usage. Fewer queries to raw data tables is the whole point of summary tables in the first place.
- Summary tables are much more flexible with intermediate states, allowing you to slice and dice your data in different ways. For example, you can query a summary table that consists of one-minute aggregations to accurately return an aggregate of the last hour. Hydrolix supports aggregates of aggregates.
- In our case, we incorporated intermediate states into our system to simplify summary architecture and make it more performant.
Let’s take a closer look at how summary tables work in Hydrolix.
Summary Architecture Designed for Efficiency and Late-Arriving Data
Summary tables in Hydrolix support many kinds of complex aggregations. Because Hydrolix uses high-density compression, columnar storage, and cost-effective object storage, summary tables in Hydrolix can be “wide” (including many columns with both aggregated and non-aggregated data) and have high cardinality.
This gives users a lot of flexibility in terms of how they want to summarize their data. A good example (which Hydrolix excels at) is CDN monitoring, where aggregations like the sum of unique IPs may contain millions of values.
So what exactly happens under the hood when it comes to summary tables? Data in Hydrolix is partitioned by time. This applies to both raw data partitions as well as partitions holding summary data.
At ingest time, intake heads process log data in local storage and create the raw data partitions. This process uses Kubernetes infrastructure, massive parallelism, and streaming ingest to handle upwards of ten million rows per second or more.
Data in partitions is then transformed in real time based on user-defined transform files. Transformations can include standardizing, enriching, and obfuscating data, and each data source has its own transform file. This process all happens very quickly—data is typically available for querying and analytics within seconds.
Once the raw log data has been transformed, it’s time to aggregate! Hydrolix will apply any summary aggregations and schema that you’ve specified for the table.
Because Hydrolix uses intermediate states, this is the only time that the summary service needs to read from source. This is one of the benefits of intermediate states in action. Aggregations can always be recalculated when new data arrives without reading from the raw data again.
Hydrolix’s summary service partitions the summary data by timestamp. These partitions are treated in the same manner as partitions that hold raw data. Partitions are small when they are first created and sent to object storage. This ensures that they are quickly available for querying. Over time, Hydrolix’s merge service compacts and optimizes summary partitions, compressing the data further and making querying more efficient.
Hydrolix handles late-arriving data by design, and this applies to both raw data and summary. Whether log data arrives in real time or arrives hours, days, or even months late (depending on the cutoff you specify), it’s written to raw data partitions while summary data is written to summary partitions. Over time, late-arriving data is merged and sorted.
The end result is highly accurate and efficient summary tables that recalculate aggregates based on incoming data and account for late-arriving data. That makes Hydrolix unique in the industry. Summary tables are yet another reason why Hydrolix is so effective at querying log data at petabyte scale.
Next Steps
- Learn more about setting up summary tables in Hydrolix.
- Learn how Hydrolix handles late-arriving data by design.
Want to see summary tables in action and aren’t using Hydrolix yet? Sign up for a technical demo or a free trial.
