Get ready, data enthusiasts! Hydrolix is offering an update to our Hydrolix Connector for Apache Spark (Databricks): adding crucial support for interoperability with Databricks Unity Catalog. We’ve recognized the growing need for integration across diverse data platforms, as organizations often use multiple platforms to maximize the value of their data, accelerate innovation, and future-proof their data lifecycle management strategies.
Unlocking Deeper Insights: The Power of Hydrolix and Unity Catalog Integration
The ability to interoperate with Unity Catalog is essential to fully taking advantage of the power of the Databricks platform. In today’s complex data landscape, companies often store data in various locations depending on what they are using it for. As a result, it is important to be able to join data from those disparate sources to drive necessary business outcomes. With this release, you can now join your Hydrolix data with Unity Catalog managed data. This unlocks crucial insights by combining full-fidelity event data with contextual business data. It improves quality, reduces time to insights, and unlocks a deeper understanding of customer behavior and product performance.
Databricks’ Unity Catalog leads the industry in enterprise data governance and Databricks remains the premier managed Apache Spark provider, so this integration represents a natural and significant progression. You might be asking, so what does this mean? What does this unlock? Continue reading to discover the possibilities.
Use Case: Joining CRM and Log Data
Let’s take a look at a use case where joining high volume time-series data, such as CDN log data, with business specific data, such as CRM, can provide deep insights.
An e-commerce platform using Hydrolix to store their multi-CDN logs and CRM data in Unity Catalog managed tables can make correlations that help support their users and the overall customer experience. Here’s an example. Let’s say there is an increase in 504 Gateway Timeout errors from CDN edge servers that’s detected over a span of several hours. This could indicate network issues between a CDN and the origin, a DDoS attack, or configuration errors. How should the ops team go about figuring out the cause so they can fix it? Before panicking, they consider all of the data available to them.
CDN logs capture raw web traffic, which includes valuable data such as IP addresses, requested URLs, response times, HTTP status codes, edge server locations, and cache hit/miss data. However, on their own, these logs reveal little about who the end users are or their relationship to the business. This narrow view isn’t enough for companies focused on improving customer experience (CX) and offering personalized service. By correlating CDN data with CRM data, they can:
- Identify affected users: Pinpoint specific customers impacted by slow page loads, errors, or interruptions.
- Segment impact by customer attributes: Analyze user experience based on customer tiers, subscription levels, geographic locations, or device types.
- Personalize troubleshooting and support: Empower customer support teams to diagnose complaints more efficiently.
After joining CDN logs with customer session data, the team discovers that the errors mostly impact visitors from a particular country, and most affected users belong to the standard tier, not premium. However, a small percentage of premium plus users are also affected, many in a single account. To act on this data, the company then prioritizes fixing the issue in that region and escalates support for the impacted premium customers immediately, sending personalized apology emails or credits to those affected.
Let’s explore the data and tools that can help achieve these outcomes. Several data points are essential for gaining precise insights into customer experience:
- CDN logs stored in Hydrolix
- User identifiers: Usually, this is a unique identifier like a session ID.
- IP address and timestamp: This can help you identify users when you don’t have clear user IDs, especially when combined with location, device information, or login records.
- Location: You can further enrich CDN data using Hydrolix geo-lookup with MaxMind database.
- Customer data stored in Databricks with Unity Catalog
- Customer relationship management (CRM) system: This provides more context about users such as loyalty tier and account status.
- Authentication and session management platforms: This provides context on login and activity metadata.
- The Hydrolix Connector for Apache Spark (Databricks) facilitates joining disparate datasets, such as CDN and CRM logs. For example, user IP can serve as a JOIN parameter to connect user experience data (such as 504 errors from CDN logs) with user session data (such as login activity from CRM logs), providing a comprehensive view of user experience.
With the Hydrolix Connector for Apache Spark, businesses can achieve significant improvements in data processing efficiency, query performance, and operational cost savings while realizing benefits such as:
- Targeted churn prevention: Quickly identify and support high-value customers to ensure customer satisfaction and reduce churn.
- Optimize resource allocation: Focus engineering and support efforts on the most impactful issues instead of chasing unclear data.
- Improved personalization: Tailor communications and services with accurate data, leading to higher customer satisfaction.
- Actionable insights: Understand how network issues affect customer outcomes, using this knowledge to improve products and infrastructure.
Next Steps
Check out the Hydrolix Connector for Apache Spark documentation, where you’ll find details on all the latest updates, including support for new Databricks Spark LTS versions and as well as support for AWS EMR Serverless with the Hydrolix Connector for Apache Spark (AWS EMR). For those using Microsoft Fabric, the Hydrolix Connector for Apache Spark ensures you can connect and utilize Hydrolix data for powerful insights, no matter your preferred Apache Spark environment.
- Learn about using Hydrolix for AIOps.
- Learn about using the Hydrolix Connector for Apache Spark (Databricks), including how the connector works and how Hydrolix performs against Delta Lake.
- Read about the Hydrolix Connector for Apache Spark (AWS EMR).
- Read about the Hydrolix Connector for Apache Spark (Microsoft Fabric).
- Check out the Hydrolix Connector for Apache Spark documentation.
- Contact Hydrolix about a proof of concept or demo.
