Visualize Your Hydrolix Data With Kibana

Kibana is a powerful tool for visualizing and analyzing data, but it comes with a major limitation: it’s designed to be used only with Elastic. While Elastic has many strengths, it’s very costly at scale. That leaves enterprises using the ELK (Elastic-Logstash-Kibana) stack in a difficult position. They must choose to pay high costs, manage multiple tiers of data with varying degrees of latency, discard data altogether, or transition to another solution.

If you’re ingesting terabytes of log data everyday, complementing or even replacing your Elastic clusters with Hydrolix can be a cost-effective solution. But how do you make the transition with as little friction as possible? And what if your teams want to continue to use Kibana for visualizations and analytics?

Our latest integration makes this decision simple: you can now analyze and visualize data in your Hydrolix clusters with Kibana and OpenSearch.

Hydrolix is partnering with Quesma to offer a new integration that allows you to use Kibana (or Opensearch) and Kibana Query Language (KQL) with Hydrolix clusters. By using Hydrolix instead of more expensive solutions like Elastic, you can dramatically reduce the total cost of ownership (TCO) for use cases that involve log data, including observability. Check our pricing estimator to compare the cost of Hydrolix and Elastic.

Read Quesma’s announcement about the partnership with Hydrolix and how Quesma’s smart technology is designed to unbundle the application layer to allow teams to use the visualization tools they prefer—and make database migrations easier.

The next image shows how Hydrolix and Quesma work together. Hydrolix ingests, transforms, and stores large volumes of log data. You can then make queries and visualize data through Kibana or Elasticsearch using Quesma’s smart database gateway.

With the Quesma integration, you can keep your cake and eat it, too, or rather, keep using Kibana and ingest all your data without concerns about high costs or discarding your data.

Kibana and OpenSearch Dashboards Now Available With Hydrolix

Many teams love using Kibana, but Kibana is specifically designed to be used with Elastic. Elastic is a powerful tool with many strengths such as full-text search. However, it is too expensive for large volumes of log data.

This is where Hydrolix comes in. Hydrolix is a real-time data platform built to process, retain, and query high volumes of log data. Each subsystem in Hydrolix is decoupled, including storage. This decoupled architecture allows Hydrolix to efficiently scale even for very large events (10 million log lines per second or more) while offering cost-effective S3-compatible storage.

We believe that the analytics layer should be decoupled, too, so that teams can use the tools that work best for them. That’s why we’re compatible with many solutions such as Grafana, Superset, and Splunk. Through our partnership with Quesma, we’re adding Kibana to the list of analytics tools you can use with Hydrolix.

The following image shows an example of CDN data stored in Hydrolix and visualized in a Kibana dashboard. Multi-CDN monitoring is an example of the kind of high-volume use case that Hydrolix excels at delivering. This type of use case can generate terabytes of log data every day for major enterprises. That is simply too expensive with Elastic.

This dashboard gives an overview of high-level CDN metrics including request time, request count per country, and total bytes delivered per second.

Check out the following demo to see a more in-depth look at this dashboard, including how it can be used for use cases like detecting denial of service attacks and stream piracy.

Can Hydrolix and Kibana Replace the ELK Stack?

TL;DR: Hydrolix can replace Elastic for use cases that involve log data at terabyte scale. It can also complement your existing ELK stack.

If you’re using the ELK stack, you might be wondering whether Hydrolix can replace Elastic altogether. Hydrolix is specifically designed for high-volume log data so you can use Hydrolix in place of Elastic for your log data. For other use cases (such as data that doesn’t include timestamps), you should continue using Elastic or another solution.

In many cases, the best path forward is to complement your existing ELK stack with Hydrolix. You can ingest high-volume log data into Hydrolix and then visualize and analyze that data alongside data from Elastic. This can be especially useful for use cases like multi-CDN monitoring. Or you can ingest your log data into Hydrolix and use Elastic for other data use cases such as full-text search.

What about Logstash, the L in ELK stack? From an ingest standpoint, Hydrolix can ingest data from both Logstash and Beats. That makes it easy to point your existing agents and pipelines towards Hydrolix.

• In the case of Logstash, you can use the HTTP output plugin, which allows you to send data to Hydrolix’s HTTP Stream API.
• In the case of Beats, you can send events to Apache Kafka and then ingest data into Hydrolix via Kafka.

Hydrolix even has you covered if you’re using Elastic Common Schema (ECS). Hydrolix transforms logs at ingest time, giving you tremendous flexibility in terms of how you map your data.

Each data source has its own transform file where you can map ECS-formatted log data to Hydrolix, which supports JSON format. You can optionally flatten complex nested structures into a single table row. Flattened data structures can be easier to understand and query.

Hydrolix also allows you to ingest multiple log sources into a single table. If you’re using ECS already, you’re already using a standardized format so you can easily compare logs across multiple sources.

Introducing Quesma

Quesma was founded to tackle a specific problem: to unbundle the user interface (application) layer from the data layer so that enterprises and teams can use the applications and databases that work best for them.

One of their first use cases was a translation from Elastic Query Language to SQL. This allows customers to much more easily use cost-effective solutions like Hydrolix in place of Elastic—all while continuing to use Kibana.

We’ve partnered with Quesma to provide Kibana functionality directly in Hydrolix. No migration from Elastic is necessary, and you can use your existing dashboards and queries.

Quesma works as a bridge between the Hydrolix platform and your Kibana and OpenSearch dashboards. It translates Elastic query language queries into SQL. It also returns the results in a format that’s compatible with both the Hydrolix platform and Kibana/OpenSarch.

With the Quesma integration, you can use Kibana with Hydrolix and improve your existing ELK stack with minimal friction.

Next Steps

See the Hydrolix documentation on the Kibana Integration via Quesma to set started. You can also read Quesma’s documentation on Adding Hydrolix tables to an existing Kibana/Elasticsearch ecosystem.

Learn more about Quesma and read the partnership announcement.

If you’re not using Hydrolix yet and would like to learn more, contact Hydrolix about a proof of concept or demo.