Many IT, security, and ops teams love using Splunk because of its powerful query language (SPL), customizable dashboards, and extensive ecosystem of integrations. It’s consistently one of the most popular tools for use cases like SIEM, security, and observability.
But there’s a catch, and it’s a big one. It can be extremely expensive to index and store large volumes of data in Splunk. This leaves enterprises with a tough choice. They can pay high bills, discard data they’d rather keep, or find another vendor, even though their teams prefer Splunk.
Fortunately, you can combine Splunk with a cost-effective backend like Hydrolix to give teams the best of all worlds.
- Splunk provides the interface, query language, and tooling that teams prefer.
- Hydrolix provides long-term, cost-effective retention in object storage, with the ability to ingest, store, and query petabytes of data while providing real-time and historical analytics.
This post will explore how you can query high-volume logs from Hydrolix in Splunk to keep costs down and eliminate gaps in data. This includes two use cases for high-volume log data:
- SIEM data, which is mission-critical for cybersecurity teams. Storing these logs in Hydrolix reduces costs and helps balance budgets.
- CDN data, which have traditionally been so high-volume and costly that enterprises haven’t retained them at all. Keeping these logs in Hydrolix can provide new insights and eliminate gaps in visibility.
Use Case: Reducing the Cost of SIEM Logs
SIEM (security information and event management) logs have been around since the early 2000s, and their volume keeps going up, leading to high costs on platforms like Splunk. Cybersecurity teams that collect and analyze SIEM logs are often forced to make difficult choices when it comes to reducing costs and choosing which SIEM logs to retain. But SIEM logs are mission-critical for many enterprises, and retaining them is a often a necessity. Storing SIEM logs in Hydrolix is a great way to cut costs while eliminating data gaps.
The next diagram shows the results of a Splunk query to a Hydrolix table that stores SIEM data. Once you’ve set up the Hydrolix Splunk connector, it’s as simple as using the | hdxsearch command and specifying the table you want to query. Then you can pipe commands using SPL to get exactly the information you need.
The query covers the last 15 minutes and specifies the field "ruleMessages", which provides contextual information about why WAF rules were triggered. While this is a simple query, Hydrolix is performing multiple query optimizations behind the scenes to ensure low latency and cost-effective compute.
Hydrolix uses time-partitioned, columnar storage to make this query extremely efficient. Using partition pruning, all partitions in storage that don’t include the specified time range are immediately pruned from consideration, minimizing the amount of data that’s scanned. And with column pruning, only the columns specified in the query are retrieved. These optimizations are especially useful when you’re working with massive datasets that have arbitrarily wide data with many attributes.
Hydrolix retrieves the data and then allows Splunk to work its magic. With SPL and pipes, you can quickly get a count and a pie chart visualization that’s useful at a glance or in a dashboard. Splunk users still get the analytics and visualizations they expect without compromises—all while retrieving data from a backend that provides tremendous cost savings.
Hydrolix uses high-density compression and object storage to keep storage costs down. All data is “hot” for querying by default, even historical data, making it much quicker to perform forensic analysis, correlate malicious activity, and conduct threat hunting. There’s no dark data that’s difficult to access, nor any need to rehydrate data from frozen or archived storage.
The following shows the size of a logs table in the Grafana interface Hydrolix offers (your teams can query Hydrolix data wherever works best for them, whether that’s Splunk or another interface like Kibana).
If you look at the total volume row, the compression ratio is more than 21x—reducing 235GB of raw data to under 11GB in storage. This data is in commodity object storage (an S3 bucket), where storage rates are about $23/TB. At 20x compression, a terabyte of raw data costs just over a dollar a month to retain. You can keep all your data in your own cloud infrastructure, take advantage of any cloud provider discounts you have, and have full ownership and control for use cases like compliance, security, and data sovereignty.
Use Case: Increasing Data Visibility and Insights with CDN Logs
CDNs (content delivery networks) generate vast amounts of log data. However, unlike with SIEM data, enterprises didn’t start retaining these logs until fairly recently. That’s because the scale and cost of retaining and monitoring CDN logs simply wasn’t possible with many traditional observability platforms. However, as enterprises increasingly rely on CDNs for performance and to deliver content to their customers, monitoring has become more important than ever.
Let’s take a look at some queries and visualizations of CDN data from Hydrolix, starting with a count of top statusCode over the last 15 minutes.
Fortunately, most of the status codes are 200 OK. But 403 (forbidden), 404 (not found), and 503 (service unavailable), are all issues that ops and other teams will want to investigate further, especially if there are unusual spikes.
So let’s take a closer look at those 403 errors. Specifically, are there any IPs generating an unusually high number of 403 errors? If so, it could suggest malicious behavior. That’s a quick query with SPL—we just need a count of the top cliIP by statusCode=403. To take advantage of Hydrolix column pruning, the query only retrieves the cliIP and statusCode fields.
Those top three cliIP counts are concerning—a typical user isn’t going to generate nearly one hundred 403 status codes in 15 minutes.
Drilling down, it’s time to look at fields=* for the top cliIP to provide full granularity into individual request logs. The next screenshot shows the detail for an individual log row. A WAF (web application firewall) rule denied access due to a SQL-INJECTION-ANOMALY.
In this case, the WAF is successfully doing its job and denying the requester access to the resource. But now it’s clear that this cliIP is engaging in suspicious behavior. Considering the persistent requests from this cliIP, some extra due diligence might be in order. Are there any logs for the cliIP where the statusCode isn’t 403? If so, that might suggest that an attacker has found a way to bypass the WAF and other security measures.
The next query shows 34 requests from the cliIP over the last 15 minutes that had other status codes—worthy of further investigation and possible mitigation such as updates to WAF rules.
This is just the tip of the iceberg in terms of what CDN monitoring can provide in terms of visibility. The ability to monitor CDN performance, see how your users are engaging with assets in real time, and detect application issues is invaluable for enterprises that rely on CDNs to deliver their content.
Correlating Data in Splunk Dashboards
With Splunk, you can bring data from multiple sources into a single dashboard. So you can imagine the potential value of bringing together data from different Hydrolix tables, each holding their own groupings of high-volume logs. The next dashboard shows data from both CDN and SIEM tables.
Request count by country provides a quick snapshot of current traffic, and can also be helpful to see spikes in demand and identify volumetric attacks like DDoS. Meanwhile, request count by status code (not 200) provides a quick view of status codes that might indicate problematic issues—such as 400 and 500 errors.
The other widgets come from the SIEM table. Top rule messages, rule actions, and denied IPs are just a few of the metrics that security teams often need to capture to better understand threats in real time.
This is a simple dashboard, but it demonstrates how you can combine two high-volume log sources from Hydrolix in Splunk for interesting analyses and correlations. Hydrolix also has the unique ability to stream multiple high-volume data sources into a single table, essentially joining data at ingest time. This is especially useful for logical groupings of data. For example, you might want to group your CDN data in one table and your SIEM data in another table. Doing so allows you to correlate data quickly without having to join data from multiple sources at query time. And because Hydrolix supports very large tables (many billions or even trillions of rows), tables can hold truly massive amounts of data.
If you’ve ever been burned by massive queries that time out—and huge compute costs to go with it—Hydrolix has that covered, too. Hydrolix Search for Splunk comes with sensible default circuit breakers so that users can explore queries without using too much compute. Combined with the excellent needle-in-a-haystack query performance of Hydrolix and features that allow you to retrieve just the narrow byte ranges you need from object storage, Hydrolix provides next-generation efficiency and scale that many of today’s observability platforms can’t offer. Meanwhile, Splunk offers a powerful, easy-to-use query language, beautiful visualizations, and flexible tooling. Together, they give you all the insights you need—with the scale and budget to keep all your log data long-term without compromises.
Next Steps
Hydrolix is designed for high volumes of log data. Our direct offering, which is highly customizable and allows you to keep all data in your own virtual private cloud, is designed for log volumes of at least one terabyte per day. Use our pricing calculator to estimate how much you’ll save by combining Hydrolix with Splunk.
Ready to take the leap and try Hydrolix out? With a proof of concept, you can test out Hydrolix at any volume, with dedicated support from Hydrolix to help you get started. Or if you’re not quite ready for a POC yet, you can get started with a demo and bring any questions you have for us.
