Build Better Models with AWS EMR and Hydrolix

The tech world has been ablaze with talk of the power of Artificial Intelligence (AI) and Machine Learning (ML). The AI and ML space is certainly exciting, but it can be hard to figure out how best to apply and use it for your day-to-day needs.

AWS offers AI services that help businesses go from raw data to real-world impact faster than ever, and there are endless opportunities to do interesting work with these tools. But it can be hard to narrow in on what might be best for you and your work. And once you find something you want to build, it can be extremely challenging to get enough quality data to meaningfully train your model and implement your idea.

With both Hydrolix for AWS and self-hosted Hydrolix, you can combine the full fidelity of event data in Hydrolix with other business data in S3 to train new ML models or build generative AI applications with high-performing foundational models. And now we’ve expanded the Hydrolix Connector for Apache Spark to support AWS EMR. Hydrolix is here to enable you to make the most of AI/ML. Let’s talk about how that works.

Solving the Cost and Scale Challenge of Big Data for Training

One of the challenges of AI/ML training is having access to both the correct data and the correct volume of data needed to build truly reliable models. Luckily, this is where Hydrolix excels. Storing event data is expensive, but in this day and age, large datasets are critical and companies can no longer afford to waste valuable data. Hydrolix makes it possible and affordable to ingest, query, and retain petabytes of event data, which is exactly what you need to build and train reliable models. With Hydrolix, you can: 

• Handle daily petabyte-scale event data with scalable streaming, and reduce costs with autoscaling.
• Standardize, structure, normalize, and transform event data using full ETL (extract, transform, load) capabilities.
• Generate efficient, accurate summary tables that provide aggregations in real time while retaining the ability to analyze the raw underlying data.
• Achieve 20x-50x compression rates, enabling cost-effective long-term data storage for enterprises.
• Maintain consistent “hot” query performance for both recent and historical data, optimizing the efficiency of object storage.

So not only can you train your AI/ML with Hydrolix data, you won’t blow up the budget doing it.

Combine Hydrolix With AWS for AI/ML Workflows

With the full scale of your event data available, you can use the Hydrolix Connector for Apache Spark to query your Hydrolix data using AWS EMR. If you’re using Hydrolix for AWS or self-managed Hydrolix in your own AWS infrastructure, you can keep all your analytics zero-egress (no egress costs) while using AWS EMR and other AWS tools such as: 

• AWS SageMaker: With AWS SageMaker, you can build, train, and deploy ML models with fully managed infrastructure, tools, and workflows using data queried from Hydrolix. 
• AWS Bedrock: With AWS Bedrock, you can build generative AI applications and then scale these applications using foundational models enhanced with data from Hydrolix.

Let’s look into some examples of how you can use Hydrolix data, the Hydrolix Connector for Apache Spark, and AWS SageMaker and Amazon Bedrock to accelerate your AI/ML workflows. 

Use Case: Building a Custom Bot Detection System

Bot traffic can be a major issue for many enterprises. While some bot traffic (such as traffic from crawlers) is legitimate, bot traffic may also be a sign of content piracy, denial of service (DoS) attacks, and other malicious activity. To combat these issues, one approach is to build a bot detection system to automatically detect and respond to bot traffic based on WAF logs.

You can use WAF and CloudFront logs (collected in Hydrolix for AWS and processed using the Hydrolix Connector for Apache Spark for AWS EMR) with a custom bot detection model, developed and deployed with AWS SageMaker and trained with processed Hydrolix data, to inform AWS Bedrock so it can summarize bot activity.

Step 1: Data collection

• Use Hydrolix for AWS to collect your WAF and CloudFront logs.

Step 2: Data collection and processing

• In your AWS EMR cluster, use the Hydrolix Connector for Apache Spark to query and process WAF and CloudFront data in Hydrolix to surface the fields needed.

Step 3: Bot detection mode

• Use AWS SageMaker to develop and deploy your custom bot classification model and then train it using the processed data that is stored in AWS S3 from step 2.
• Deploy as a SageMaker endpoint for real-time inference.

Step 4: Real-time intelligence and response

• Now that your bot classification model is deployed and available as a SageMaker endpoint, add generative AI into the mix using AWS Bedrock.
• Generative AI can summarize unusual bot activity, generate easy-to-understand incident reports, and provide reasons for flagged requests, which is especially beneficial for security operations reviews.

Now, tracing and analysis that would’ve taken days to do can be done in a matter of minutes.

Use Case: Improving Ad Performance With Custom Models

Ads are a major revenue source for many enterprises, including media companies that specialize in live and on-demand streaming. To determine top-performing ads, personalize subsequent ad insertions based on viewer behavior and content, and automatically modify campaign targeting according to performance and audience trends, you might want to build custom models based on historical event data.

You can combine Hydrolix for AWS with AWS Sagemaker and Bedrock to provide AWS Elemental insights into which ads to serve.

Step 1: Data collection

• Use Hydrolix for AWS to ingest OpenRTB logs from AWS Elemental MediaTailor and MediaLive.

Step 2: Process and enhance event data  

• Use the Hydrolix Connector for Apache Spark to expose event data in Hydrolix to AWS EMR for insights on ad views, clicks, and viewer behavior. 
• Use EMR to enrich logs with geo/IP/device and content metadata, and to aggregate stats like clickthrough rate (CTR), engagement time, and drop-off rate.

Step 3: Create an ad performance and personalization model 

• Develop an AWS SageMaker model to predict ad CTR for different audiences, recommend optimal ad slot placements, and detect performance anomalies like sudden CTR decreases.
• Continuously retrain your model using your Hydrolix data from step 2. 

Step 4: Get real-time intelligence on ad performance

• Use AWS Bedrock to generate natural language summaries of ad performance trends for business stakeholders, offer suggestions for creative improvements based on viewer sentiment, and produce dynamic ad copy or voice-over variations at scale.

Step 5: Optimize ad delivery

• Feed these performance insights back into AWS Elemental services to adjust, in real time, which ads to serve.

With this kind of workflow, you can improve ad revenue and ensure that advertisers are satisfied with your brand and ad delivery.

Next Steps

• Learn about using Hydrolix for AIOps.
• Learn about using the Hydrolix Connector for Apache Spark with Databricks, including how the connector works and how Hydrolix performs against Delta Lake.
• Read about the Hydrolix Connector for Apache Spark with Microsoft Fabric.
• Check out the Hydrolix Connector for Apache Spark documentation.
• You can run Hydrolix entirely within your AWS infrastructure. Learn about the benefits of running log infrastructure in your own virtual private cloud. Or consider Hydrolix for AWS, a managed solution that runs in AWS.
• Contact Hydrolix about a proof of concept or demo.