Hydrolix provides you with multiple ways to integrate with other data processing systems, including Cribl. For example, if you’re using Cribl Cloud as your data processing and routing tool and Hydrolix as your data platform, you can query all your data directly in Cribl with Cribl Search. You can also filter, route, and enrich data in Cribl Stream and then send it to Hydrolix as a source using our Cribl integration.
Whether you want to stream or search, Cribl and Hydrolix are complementary tools that allow you to maximize the value of your data while keeping costs down. Using them together gives you tremendous flexibility when it comes to managing your data. You can easily decide which data requires manipulation in Cribl before storing it in Hydrolix or another source, or you can stream it directly to Hydrolix if you want to keep full-fidelity data in Hydrolix. Like Cribl Stream, Hydrolix includes a built-in streaming pipeline that allows you to transform, normalize, and filter your data.
In this post, you’ll learn how to quickly set up Cribl Search to start querying your data in Hydrolix as a source. With this method, you can use Cribl Search to query Hydrolix using standard SQL, all while getting the sub-second needle in a haystack performance on petabyte-scale data that Hydrolix offers. You can even do federated searches in Cribl Search that combine Hydrolix with other data sources.
Set Up Cribl Search with Hydrolix
Cribl Search allows you to query data across multiple sources, such as S3 buckets, data lakes, and various observability tools, all without centralizing or moving it first. Instead of logging into different systems to hunt for answers during an incident or investigation, you can use it as a single search interface across different sources, including Hydrolix. This federated approach gives you the flexibility to store data in Hydrolix while maintaining the ability to search it alongside other sources.
You can set up the connection in just a few minutes.
1. Configure a dataset provider in Cribl Search
First, you’ll need to set up Hydrolix as a dataset provider in Cribl Search. Start by logging into your Cribl console and navigating to Cribl Search. Next, complete the following steps:
- Select Dataset and then select Dataset Provider from the top menu.
- Select Add Provider and a new form will open up.
- Set your provider ID as ‘Hydrolix’ or an identifier of your choosing. You can also optionally add a description.
- From the Dataset Provider type, use the dropdown to select Generic HTTP API.
- Name your endpoint. (You can use the same name as your Hydrolix transform.)
- You can either leave the data field empty if you want to capture the entire record or specify a specific field key.
- The URL should use the GET method with the following format:
https://<cluster-host>.hydrolix.live/query/?query=${query}. - You can add any additional headers if needed, such as
accept: text/json.
2. Set up authentication
Next, you’ll set up authentication.
- Under Authorization, choose the authentication method Login from the radio button menu.
- In the Login URL field, you’ll use the Hydrolix Auth API. Enter the following URL:
http://<cluster-host>.hydrolix.live/config/v1/login/. - In Hydrolix, you’ll need to create a user with role policy of
select_sql. See the docs on how to define a role in the Hydrolix UI. - Enter your Hydrolix username and password in the Cribl authorization menu.
- In the POST body field, use the default parameters.
- For token attribute, point to the token that will be used in the return token structure response:
auth_token.id_token. - Keep the rest of the fields at their default value and save your new dataset provider setting.
3. Configure your dataset
- From the top menu select Dataset, then Add Dataset.
- Provide an ID that identifies the Hydrolix dataset it’s associated with and an optional description.
- Next, select a dataset provider (you should see the name of the provider you created in step 2).
- Select Processing from the lefthand menu, add a datatype named Hydrolix, and keep the default settings.
4. Running queries
Now that you have the dataset configured, you can run queries from Cribl Search or create Cribl notebooks for saved queries. Here’s an example query that uses Cribl Search to query WAF data stored in Hydrolix: dataset="hdx_demo" query="SELECT http_headers FROM aws.waf LIMIT 10".
Next Steps
Interested in learning about Hydrolix? Book a demo.
