Get More Insights From Log Data With Hydrolix and Microsoft Fabric

Earlier this year, we released the Hydrolix Connector for Apache Spark, combining the strengths of Hydrolix and Spark so enterprises can get more insights from their log data. Now your teams can use the connector to analyze Hydrolix log data in both Microsoft Fabric and Azure Databricks.

For enterprises committed to the Microsoft and Azure ecosystem, you can ingest, store, and analyze all your log data within Azure, then use Fabric or Azure Databricks to get deeper insights from that data. Hydrolix is compatible with all major cloud providers, so you can use Microsoft Fabric to analyze your log data wherever it lives, including AWS and GCP.

Over the last few months, we’ve also made performance and usability updates to the connector, adding support for maps and arrays, compatibility with Databricks Photon and Fabric’s native execution engine, more visibility into errors, and easier setup of summary tables for aggregating data in real time.

This post will cover:

• Using Microsoft Fabric and Azure Databricks with Hydrolix
• How Hydrolix is designed for high-volume data and can complement your existing data storage and analytics solutions
• Other updates to the Hydrolix Connector for Apache Spark

Hydrolix Connector for Apache Spark for Advanced Analytics With  Microsoft Fabric and Azure Databricks

The Hydrolix Connector for Apache Spark is designed to make it easier to work with your log data no matter which data analytics platforms your teams prefer—and to make it possible to expand the value of log data for use cases ranging from AI/ML to business intelligence.

The following screenshot shows an example of high-volume CDN logs being analyzed in Microsoft Fabric.

This dashboard shows how CDN session data can be clustered to better understand potential anomalies and detect issues such as stream piracy, CDN leaching, malicious behavior, and more. We’ll be talking more about applying data science techniques to detect anomalies in CDN logs in a future post.

Too often, log data is siloed in observability platforms—and while observability is a mission-critical use case, it shouldn’t come at the expense of training AI/ML models or getting deeper, long-term insights from your data.

For enterprises that use Azure and the Microsoft ecosystem, you can now use the Hydrolix Connector for Apache Spark with both Microsoft Fabric and Azure Databricks for advanced analytics, AI use cases, and collaboration across teams.

You can host all your logs “on prem” in Azure object storage and use Azure compute—with any Azure discounts you’ve negotiated—all within your own secure Azure Virtual Network (VNet). Hydrolix provides a zero-egress solution for ingesting, storing, and analyzing your logs within the Microsoft ecosystem.

Why Use Hydrolix for High-Volume Log Data?

What if your enterprise is already using other solutions such as OneLake for storage or Delta Lake as a storage layer? Why would you want to include Hydrolix in the mix?

Both OneLake and Delta Lake are general-purpose solutions that can work with many types of data. Meanwhile, Hydrolix is specifically designed for timestamped data, including logs, that can complement the solutions you are already using.

You get the benefits of Hydrolix for high-volume timestamped data, including real-time analytics and cost efficiency, all without compromising or overhauling the systems you already have in place. Meanwhile, your teams can analyze data using Microsoft Fabric and other tools regardless of whether that data lives in Hydrolix or elsewhere.

Every subsystem in Hydrolix is designed for efficient, cost-effective processing, storage, and analysis of high-volume log data. This includes:

• Scalable streaming ingest that can handle petabytes of log data on a daily basis or autoscale down as needed to reduce compute costs;
• Full ETL (extract—transform—load) for standardizing, structuring, normalizing, and transforming log data;
• Time-to-glass of under ten seconds for real-time analytics use cases such as observability and CDN monitoring, even at petabyte scale;
• Summary tables that provide extremely efficient and accurate aggregations—all while allowing you to analyze the underlying raw data when needed:
• Compression rates of 20x-50x, allowing enterprises to store data for much longer more cost-efficiently;
• Separate query pools so that all your teams and workloads, from ops to business intelligence to machine learning training runs, can run queries without resource contention or degraded performance;
• “Always hot” query performance regardless of whether data is a minute or a year old—all while maximizing the efficiency of cost-effective object storage.

Hydrolix provides the performance and functionality that your ops teams need to monitor system performance along with the access and long-term storage that other teams need for use cases like business intelligence. Your ops teams can monitor request latency in a Grafana dashboard while your business intelligence teams can use Microsoft Fabric to weed out fraud. And you can still easily run machine learning workloads to improve fine-tune algorithms that work best for your business’ use case, such as anomaly detection, fraud detection, and capacity planning.

Latest Updates: The Hydrolix Connector for Apache Spark Keeps Getting Better

Too often, vendors release a quickstart or new integration, add a logo to their integrations page, and then forget about it. That’s not the case with the Hydrolix Connector for Apache Spark. We are committed to improving it and it keeps getting better. Since we released the connector just a few months ago, we’ve added performance and UI improvements, including:

• Support for maps and arrays
• Photon compatibility (for Databricks and Azure Databricks users) and compatibility with Fabric’s native execution engine
• More visibility into errors
• Easier set up of summary tables for aggregations

Support for Maps and Arrays to Help Manage Complex Data

Maps and arrays can be extremely valuable when it comes to storing data that is semi-structured and high cardinality—and these data types are also useful for representing complex relationships and hierarchies.

However, many SQL dialects provide minimal support for maps and arrays. They may have features for storing and extracting these data types, but not necessarily for high-performance queries, aggregations, and other use cases that are important for data analysts.

Both Hydrolix and Apache Spark have support for maps and arrays—and with the latest update, the Hydrolix Connector for Apache Spark now has full support for these data types as well.

One typical use case for storing more complex data types in Hydrolix is the “catch-all” field, which stores an array of maps. This catch-all holds semi-structured data that hasn’t been defined in a transform file. Catch-alls can be helpful for storing attributes in wide, high-dimensionality tables where the shape of incoming data is unclear or the business value of some fields remains unknown.

Compatible With Databricks Photon and Fabric’s Native Execution Engine

The Hydrolix Connector for Apache Spark is compatible with both Photon, Databricks’ vectorized query engine, and Fabric’s native execution engine, which uses Apache Gluten and Meta’s Velox execution accelerator to speed up queries.

Teams using the Hydrolix Connector for Apache Spark can achieve high performance even without query acceleration, typically enabling more cost-effective compute. Ultimately, teams can choose the compute scenario that best fits their needs.

More Efficient Pagination and Top-N Queries

The Hydrolix Connector for Apache Spark now supports additional pushdowns, improving performance for common use cases like pagination and finding the top-N results in a dataset.

Pushdowns optimize performance by filtering query results closer to the source of the data. The Hydrolix Connector for Apache Spark supports many pushdowns, and we are continuing to make queries more efficient across a wide range of use cases. With our latest release, we’ve added pushdown support for queries that use OFFSET clauses as well as queries that use ORDER BY combined with LIMIT clauses.

The end result: more queries that combine high performance and efficient compute on large datasets.

Better Visibility Into Query Errors and Bottlenecks

In many distributed systems, it can be very tough to find errors—or to understand whether an issue with performance is due to an error or something else. Is it broken, slow, or a weird edge case that you simply can’t replicate? Issues that are intermittent and hard to pin down are often major sources of frustration for engineering teams.

They can also be a huge issue for analysts querying big datasets. For example, if queries are taking longer than expected, is it because the query is misconfigured or poorly optimized, or does it suggest another issue altogether?

To reduce this pain, we’ve added more visibility into errors when using the Hydrolix Connector for Apache Spark with both Databricks and Azure Databricks, including better error reporting and metrics that can help your teams understand how queries are performing.

The following image shows some of the metrics Hydrolix gathers.

You can see how long it took Spark to create a query plan, read data from each Hydrolix partition, and query each partition. With this information, you can better understand whether there are skewed partitions, data locality issues, or issues with fetching metadata. This makes it much easier to troubleshoot inefficient queries and other issues.

Quickly Set Up Tables for Aggregating Data

Aggregate tables, which are known as summary tables in Hydrolix, are an essential part of working with big data. They can provide a high-level view of your data, giving your teams insights on everything from average request latency to total units sold per region.

They can also dramatically improve query performance and keep compute costs down. For example, the majority of dashboards include visualizations of metrics. These metrics can be generated by querying small, highly efficient summary tables instead of the underlying raw data, which might be stored in tables that have billions or even trillions of rows. A dashboard that refreshes every minute can make 1500 queries per day, so costs can quickly add up with inefficient queries.

We’ve now made it easier than ever to set up and use summary tables with the Hydrolix Connector for Apache Spark. Users no longer need to preregister summary tables. This simplified process is in contrast to Delta Live Tables—Delta Lake’s solution for aggregate tables—which involves many setup steps, including declaring aggregate tables and then configuring Spark clusters to access them. 

Next Steps

• Read the docs to set up the Hydrolix Connector for Apache Spark for Microsoft Fabric. Or, if you’re using Databricks, follow the doc for setting up the connector with Databricks.
• Learn more about how the Hydrolix Connector for Apache Spark works—and how it performs against Delta Lake.
• If you’re not using Hydrolix yet and would like to learn more about the Hydrolix Connector for Apache Spark or using Hydrolix within your cloud infrastructure, contact Hydrolix about a managed trial or demo.