Przemysław Hejman is a founding engineer at Quesma. Quesma integrates with Hydrolix to allow you to visualize and query your data using Kibana.
Ever since Kibana was first developed, it has captivated users with its powerful data visualization capabilities. However, there’s a catch—these features are tied to Elasticsearch as the underlying data store.
Over the years, there have been community attempts to extend Kibana’s functionality to other databases, some which have even led to billion-dollar companies like Grafana. Yet none of these efforts have made it into Kibana’s core project.
At Quesma, we appreciate both Kibana’s visualization strengths and the blazing-fast, cost-efficient capabilities of columnar, SQL-compatible databases like Hydrolix. So why not bring the best of both worlds together?
That’s why we developed a database gateway—a solution that sits between applications and databases, enabling users to swap out their database backend without disrupting their applications.
With Quesma’s database gateway, you can connect a vanilla instance of Kibana (or any other application using the Elasticsearch API) to back-end data solutions like Hydrolix.
In this post, we’ll take a closer look at how Quesma works under the hood, including:
- A high-level view of how Quesma’s database gateway works
- An example of query translation in action
How Elasticsearch and Kibana Communicate
Elasticsearch provides an HTTP REST API for ingesting and querying data. Kibana, like any other Elasticsearch API client, uses this API to interact with the database.
Kibana itself is a stateless service, relying on Elasticsearch to store saved objects such as dashboards and visualizations. Additionally, user authentication and authorization are delegated to Elasticsearch.
As the next diagram shows, Kibana and Elasticsearch interact through HTTP requests and responses.
Introducing a Proxy Between Kibana and Elasticsearch
As long as you keep the HTTP protocol and messages intact, you can introduce an intermediate component, or proxy, between the two services. An MITM proxy is just one example. And it’s even possible to use the proxy to do some simple operations on the payload from Elasticsearch as long as the payload remains compatible with Kibana.
The next diagram shows an MITM proxy between Kibana and Elasticsearch.
What would it take to replace Elasticsearch with Hydrolix in this diagram?
First, Elasticsearch and Hydrolix communicate in completely different ways. While the networking layer for both is based on TCP, there are several key differences:
- Unlike Elasticsearch, Hydrolix doesn’t wrap queries inside HTTP requests.
- Hydrolix uses ANSI-compliant SQL, not JSON-based Elasticsearch QueryDSL.
The following diagram shows a basic example of how queries are structured in Hydrolix and Elasticsearch. The query in Elasticsearch makes an HTTP request and uses nested JSON while the Hydrolix query uses a SQL SELECT statement.
Mapping Elasticsearch and Hydrolix Domains
This is where Quesma comes in: it will act as a proxy translating queries and query results so that Hydrolix and Kibana can communicate with each other. Replace MITM from the previous diagram with Quesma, and you’ll see that it’s a proxy that will perform these translations for us.
So we need to map two domains to each other, starting with the Elasticsearch domain because this is the domain we need to provide compatibility with.
Here are a few examples:
- The Elasticsearch index (a collection of documents) becomes a Hydrolix table (a collection of records).
- Elasticsearch fields become Hydrolix columns.
Certain Elasticsearch features, such as its advanced full-text search capabilities, don’t have direct equivalents in Hydrolix or SQL. For instance, Elasticsearch’s relevance scoring and performance in text search exceed what SQL’s LIKE or ILIKE operators can achieve. In these cases, we accept the limitations and focus on the primary goal: enabling data visualization through Kibana.
Saving Kibana Dashboards
Kibana stores its internal data, including saved dashboards, in Elasticsearch. For that reason, we didn’t attempt to completely replace Elasticsearch when designing Quesma. Instead, we took a minimalist approach—maintaining a very small Elasticsearch instance for internal data. This allows Quesma to focus on translating queries from Hydrolix.
The updated diagram below shows how Quesma acts as a proxy between Hydrolix, Kibana, and this minimal Elasticsearch instance.
Let’s say you have a Hydrolix table named user_events and configure Quesma to expose this table as an Elasticsearch index with the same name. When a request targets this index, Quesma processes it appropriately. However, requests for Kibana’s internal data (such as targeting the .kibana index) are routed directly to Elasticsearch.
How Quesma Translates Queries: High Level View
Let’s take a look at the procedure for translating queries. From a high level, it looks like this:
- A user makes an Elasticsearch query in Kibana.
- Quesma parses this Elasticsearch query and creates an execution plan that translates it to SQL.
- Quesma sends the SQL query to Hydrolix, which runs the query and returns the results to Quesma.
- Quesma formats the results into an Elasticsearch-compatible JSON response, which it sends to Kibana.
- Kibana renders a dashboard based on the results.
Translating Elasticsearch Queries to SQL
Let’s say that Quesma receives an HTTP request from Kibana targeting a user_events table.
Depending on the Kibana version, this might come at the endpoint of user_events/_search or user_events/_async_search. Asynchronous query processing is beyond the scope of this post, so we’ll focus on using the user_events/_search endpoint.
The HTTP request contains an Elasticsearch Query DSL. Quesma parses this Elasticsearch Query DSL and creates an execution plan. This plan includes:
• The SQL query or queries needed to fulfill the request.
• The target Hydrolix table.
• Metadata, such as type mappings and user-defined configuration overrides.
Here’s an extremely simple example:
POST /books/_search
{
"query": {
"exists": {
"field": "review"
}
}
}Depending on the complexity of the Query DSL, Quesma may need to transform the query’s AST (Abstract Syntax Tree) before serializing it to SQL. The example request checks for the existence of the field "review", which might translate into either of the following SQL clauses:
• WHERE review IS NOT NULL
• WHERE review.size = 0
The choice between these two will depend on the target field data type.
Translating SQL Results Into Elasticsearch-Compatible JSON
After translation, Quesma sends the SQL queries to Hydrolix via its native interface over a TCP connection. When Hydrolix returns the results, Quesma formats them into an Elasticsearch-compatible JSON response. Some elements, like shard statistics or relevance scores, may need to be simulated or omitted. Nonetheless, Kibana can render the data into a visually rich dashboard.
In order for Kibana to understand the query results, Quesma needs to assemble an Elasticsearch-API-compatible response. Quesma takes the result rows returned by Hydrolix and wraps it into JSON format, ensuring proper syntax is maintained.
Of course, some parts of the response must be “faked” by Quesma by providing arbitrary or empty values. For instance, no relevant shard statistics for executed queries or search relevance score will be calculated. However, as the data records are coming via HTTP response to Kibana, it will render a beautiful dashboard, as the next image shows.
We’ve just scratched the surface regarding how Quesma works. There are other gotchas that Kibana requires such as dedicated endpoint fetching field types and resolving index patterns to the actual indices, which Quesma handles, but they are beyond the scope of this post.
Conclusion
Quesma bridges the gap between Hydrolix and Kibana, enabling Elasticsearch-like functionality without the need for Elasticsearch. Whether you’re building dashboards, running complex queries, or managing metadata, Quesma ensures a smooth experience. As we continue to push the boundaries of database interoperability, stay tuned for more insights and innovations in data infrastructure.
Next Steps
Thinking about making the switch from the ELK stack to Hydrolix? Learn how Hydrolix can help cut costs, extend retention, and simplify operations.
- Pricing: Hydrolix vs Elastic Cloud Hosted for Log Data
- Hydrolix vs Elastic Cloud Hosted: Pricing, Retention, and Operations
- Hydrolix vs Elastic Cloud Hosted: A Technical Deep Dive
Learn more about Quesma and Quesma’s integration with Hydrolix.
