Using Fluent Bit to Send Logs and Metrics to Hydrolix

Understanding how your infrastructure is operating using metrics like CPU utilization, memory pressure, network throughput, and disk I/O is essential for keeping systems healthy and catching problems early. But collecting these metrics alongside system logs, storing them cost-effectively, and keeping them queryable over time is a challenge that grows with scale.

Fluent Bit, part of the CNCF’s Graduated Fluentd ecosystem, solves the collection side: it’s a lightweight agent that captures system metrics and logs from across your infrastructure with minimal overhead. Hydrolix solves the storage and analysis side: it ingests high-volume telemetry from multiple sources into a single table, compresses it efficiently, and keeps it searchable for months or years while limiting costs.

In this post, we’ll walk through combining the two: configuring Fluent Bit to stream CPU, memory, network, and disk metrics along with system logs from an AWS environment into Hydrolix, then visualizing everything in Grafana. By the end, you’ll have real-time and historical visibility into your host-level performance in a setup that scales affordably as your infrastructure grows.

Prerequisites

1. You should have a running Hydrolix cluster.
2. You should have a project named demo and a table named fluentbit.
3. You should have a bearer token for authorization to send data to your cluster’s streaming ingest endpoint.

Deploying and Configuring Fluent Bit

This guide acts as a reference for setting up Fluent Bit on an AWS Linux EC2 machine. You can reference that guide if you run into issues with the steps below. 

Install Fluent Bit on a Linux Machine

Run the following command from the command line on your Linux box:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This runs a script which will by default install the most recent version of Fluent Bit.

Configuration

The configuration below enables the following inputs, filters, and outputs:

Inputs

Together, these inputs give you a complete picture of host-level health. They’ll let you correlate a CPU spike with a surge in network traffic or a disk I/O bottleneck within a single queryable table in Hydrolix.

• CPU Log Based Metrics
• Memory Metrics
• Network I/O Log Based Metrics
• Disk I/O Log Based Metrics
• Systemd Metrics

Filter

Filters reshape and enrich the raw data before it reaches Hydrolix, making it easier to query and correlate later. The AWS Metadata filter decorates system metrics with AWS metadata. The Nest filter allows the data to be nested and modified into a JSON blob. Both of them together format the data for Hydrolix and improve their presentation.

• Nest
• AWS Metadata

Outputs

This output configuration will forward the decorated metric data to a Hydrolix cluster.

• HTTP

Edit the Fluent Bit configuration file located at /etc/fluent-bit/fluent-bit.conf and replace it with the following configuration. Note that this configuration references a Hydrolix transform called fluentbit_transform. You do not need to have this transform created yet – you will create it after setting up the Fluent Bit configuration.

Example Fluent Bit Configuration

[INPUT]
    name cpu
    tag cpu
    interval_sec 1

[INPUT]
    name mem
    tag  mem
    interval_sec 1

[INPUT]
    name netif
    tag netif
    interval_sec  1
    interface enX0

[INPUT]
    name disk
    tag disk
    interval_sec  1

[INPUT]
    name systemd
    tag  host.*

[FILTER]
    Name nest
    Match mem
    Operation nest
    Wildcard Mem.*
    Nest_under memstats
    Remove_prefix Mem.

[FILTER]
    Name nest
    Match mem
    Operation nest
    Wildcard Swap.*
    Nest_under swapstats
    Remove_prefix Swap.

[FILTER]
    Name nest
    Match cpu
    Operation nest
    Wildcard *
    Nest_under cpu

[FILTER]
    Name nest
    Match netif
    Operation nest
    Wildcard *
    Nest_under network

[FILTER]
    Name nest
    Match disk
    Operation nest
    Wildcard *
    Nest_under disk

[FILTER]
    Name aws
    Match *
    imds_version v1
    az true
    ec2_instance_id true
    ec2_instance_type true
    private_ip true
    ami_id true
    account_id true
    hostname true
    vpc_id true

[OUTPUT]
    name http
    match *
    host hostname.hydrolix.live
    port 443
    URI /ingest/event?table=demo.fluentbit&transform=fluentbit_transform
    Format json_lines
    json_date_key timestamp
    json_date_format iso8601
    tls on
    compress gzip
    header Authorization Bearer {bearer-token}

Data Example

{
    "MESSAGE": null,
    "PRIORITY": null,
    "SYSLOG_FACILITY": null,
    "SYSLOG_IDENTIFIER": null,
    "_BOOT_ID": null,
    "_CAP_EFFECTIVE": null,
    "_CMDLINE": null,
    "_COMM": null,
    "_EXE": null,
    "_GID": null,
    "_SYSTEMD_CGROUP": null,
    "_SYSTEMD_UNIT": null,
    "_UID": null,
    "account_id": "209166775408",
    "ami_id": "ami-08e6b682a466887dd",
    "az": "us-east-2c",
    "cpu": {"cpu0.p_cpu":1,"cpu0.p_system":0,"cpu0.p_user":1,"cpu1.p_cpu":7,"cpu1.p_system":1,"cpu1.p_user":6,"cpu2.p_cpu":5,"cpu2.p_system":0,"cpu2.p_user":5,"cpu3.p_cpu":8,"cpu3.p_system":0,"cpu3.p_user":8,"cpu4.p_cpu":10,"cpu4.p_system":2,"cpu4.p_user":8,"cpu5.p_cpu":5,"cpu5.p_system":1,"cpu5.p_user":4,"cpu6.p_cpu":10,"cpu6.p_system":1,"cpu6.p_user":9,"cpu7.p_cpu":5,"cpu7.p_system":1,"cpu7.p_user":4,"cpu_p":6,"system_p":0.75,"user_p":5.25},
    "custom": {},
    "disk": {},
    "ec2_instance_id": "i-0df7c9352e02f8c19",
    "ec2_instance_type": "t4g.2xlarge",
    "hostname": "ip-172-31-34-182.us-east-2.compute.internal",
    "memstats": {},
    "network": {},
    "private_ip": "172.31.34.182",
    "swapstats": {},
    "test": null,
    "timestamp": "2022-09-23 12:29:59.083",
    "vpc_id": "vpc-db549db0"
    },
    {
    "MESSAGE": null,
    "PRIORITY": null,
    "SYSLOG_FACILITY": null,
    "SYSLOG_IDENTIFIER": null,
    "_BOOT_ID": null,
    "_CAP_EFFECTIVE": null,
    "_CMDLINE": null,
    "_COMM": null,
    "_EXE": null,
    "_GID": null,
    "_SYSTEMD_CGROUP": null,
    "_SYSTEMD_UNIT": null,
    "_UID": null,
    "account_id": "209166775408",
    "ami_id": "ami-08e6b682a466887dd",
    "az": "us-east-2c",
    "cpu": {},
    "custom": {},
    "disk": {"read_size":0,"write_size":0},
    "ec2_instance_id": "i-0df7c9352e02f8c19",
    "ec2_instance_type": "t4g.2xlarge",
    "hostname": "ip-172-31-34-182.us-east-2.compute.internal",
    "memstats": {},
    "network": {},
    "private_ip": "172.31.34.182",
    "swapstats": {},
    "test": null,
    "timestamp": "2022-09-23 12:29:59.083",
    "vpc_id": "vpc-db549db0"
    },
    {
    "MESSAGE": null,
    "PRIORITY": null,
    "SYSLOG_FACILITY": null,
    "SYSLOG_IDENTIFIER": null,
    "_BOOT_ID": null,
    "_CAP_EFFECTIVE": null,
    "_CMDLINE": null,
    "_COMM": null,
    "_EXE": null,
    "_GID": null,
    "_SYSTEMD_CGROUP": null,
    "_SYSTEMD_UNIT": null,
    "_UID": null,
    "account_id": "209166775408",
    "ami_id": "ami-08e6b682a466887dd",
    "az": "us-east-2c",
    "cpu": {},
    "custom": {},
    "disk": {},
    "ec2_instance_id": "i-0df7c9352e02f8c19",
    "ec2_instance_type": "t4g.2xlarge",
    "hostname": "ip-172-31-34-182.us-east-2.compute.internal",
    "memstats": {},
    "network": {"ens5.rx.bytes":914588,"ens5.rx.errors":0,"ens5.rx.packets":751,"ens5.tx.bytes":45974,"ens5.tx.errors":0,"ens5.tx.packets":504},
    "private_ip": "172.31.34.182",
    "swapstats": {},
    "test": null,
    "timestamp": "2022-09-23 12:29:59.083",
    "vpc_id": "vpc-db549db0"
    },
    {
    "MESSAGE": "12:29:58.759 [warn] Description: 'Authenticity is not established by certificate path validation'",
    "PRIORITY": 6,
    "SYSLOG_FACILITY": 3,
    "SYSLOG_IDENTIFIER": "mix",
    "_BOOT_ID": "5cd29b77348f490987162e1d02d88bd9",
    "_CAP_EFFECTIVE": "0",
    "_CMDLINE": "/usr/lib/erlang/erts-12.0.3/bin/beam.smp -- -root /usr/lib/erlang -progname erl -- -home /home/ubuntu -- -pa /usr/lib/elixir/bin/../lib/eex/ebin /usr/lib/elixir/bin/../lib/elixir/ebin /usr/lib/elixir/bin/../lib/ex_unit/ebin /usr/lib/elixir/bin/../lib/iex/ebin /usr/lib/elixir/bin/../lib/logger/ebin /usr/lib/elixir/bin/../lib/mix/ebin -noshell -s elixir start_cli -extra /usr/bin/mix run --no-halt",
    "_COMM": "beam.smp",
    "_EXE": "/usr/lib/erlang/erts-12.0.3/bin/beam.smp",
    "_GID": 1000,
    "_SYSTEMD_CGROUP": "/system.slice/ctspull.service",
    "_SYSTEMD_UNIT": "ctspull.service",
    "_UID": 1000,
    "account_id": "209166775408",
    "ami_id": "ami-08e6b682a466887dd",
    "az": "us-east-2c",
    "cpu": {},
    "custom": {},
    "disk": {},
    "ec2_instance_id": "i-0df7c9352e02f8c19",
    "ec2_instance_type": "t4g.2xlarge",
    "hostname": "ip-172-31-34-182.us-east-2.compute.internal",
    "memstats": {},
    "network": {},
    "private_ip": "172.31.34.182",
    "swapstats": {},
    "test": null,
    "timestamp": "2022-09-23 12:29:58.760",
    "vpc_id": "vpc-db549db0"
    }

Hydrolix Transform for Fluent Bit Data

You can index the generated data using the following transform output columns:

{
  "is_default": true,
  "output_columns": [
    {
      "name": "timestamp",
      "datatype": {
        "type": "datetime",
        "index": false,
        "primary": true,
        "format": "2006-01-02T15:04:05.999999Z",
        "resolution": "ms"
      }
    },
    {
      "name": "account_id",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "MESSAGE",
      "datatype": {
        "type": "string",
        "index": true
      }
    },
    {
      "name": "PRIORITY",
      "datatype": {
        "type": "uint8"
      }
    },
    {
      "name": "SYSLOG_FACILITY",
      "datatype": {
        "type": "uint8"
      }
    },
    {
      "name": "SYSLOG_IDENTIFIER",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_BOOT_ID",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_CAP_EFFECTIVE",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_CMDLINE",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_COMM",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_EXE",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_GID",
      "datatype": {
        "type": "uint32"
      }
    },
    {
      "name": "_SYSTEMD_CGROUP",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_SYSTEMD_UNIT",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "_UID",
      "datatype": {
        "type": "uint32"
      }
    },
    {
      "name": "ami_id",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "az",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "ec2_instance_id",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "ec2_instance_type",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "hostname",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "private_ip",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "vpc_id",
      "datatype": {
        "type": "string"
      }
    },
    {
      "name": "memstats",
      "datatype": {
        "type": "map",
        "elements": [
          {
            "type": "string"
          },
          {
            "type": "uint32"
          }
        ]
      }
    },
    {
      "name": "swapstats",
      "datatype": {
        "type": "map",
        "elements": [
          {
            "type": "string"
          },
          {
            "type": "uint32"
          }
        ]
      }
    },
    {
      "name": "cpu",
      "datatype": {
        "type": "map",
        "elements": [
          {
            "type": "string"
          },
          {
            "type": "double"
          }
        ]
      }
    },
    {
      "name": "network",
      "datatype": {
        "type": "map",
        "elements": [
          {
            "type": "string"
          },
          {
            "type": "double"
          }
        ]
      }
    },
    {
      "name": "disk",
      "datatype": {
        "type": "map",
        "elements": [
          {
            "type": "string"
          },
          {
            "type": "double"
          }
        ]
      }
    }
  ],
  "compression": "none",
  "format_details": {
    "flattening": {
      "active": false
    }
  }
}

Hydrolix uses its own compression algorithms. Running a query like the following against your Hydrolix cluster catalog table can yield valuable information such as the compression ratio for the data stored from Fluent Bit.

select formatReadableQuantity(sum(rows)) as total_rows
   , formatReadableSize(sum(mem_size)) as raw_data_size
   , formatReadableSize(sum(data_size)) as hdx_data_size
   , formatReadableSize(sum(index_size + manifest_size)) as hdx_index_size
   , round(sum(mem_size)/sum(index_size + manifest_size + data_size),1) as compression_ratio
   FROM demo."fluentbit#.catalog"
   where mem_size != 0

The following are actual results returned for this query after ingesting some Fluent Bit data into the table.

We can therefore visualize the compression ratio with the following bar chart.

Grafana Visualization

After deploying FluentBit into your infrastructure you can use Grafana for data visualization and alerting.

We will create a dashboard query using the Clickhouse plugin for Grafana. For more information on how to set up Hydrolix with Grafana, please see the Grafana integration documentation for Hydrolix.

Create a new dashboard within Grafana. In the following steps, you will configure the following three variables:

• The EC2 instance ID: This allows you to switch between hosts in a multi-instance environment, so one dashboard serves your entire fleet.
• Network interface data for the EC2 instance: This lets you select which network interface to inspect, since instances often have multiple interfaces (e.g., eth0, ens5) with different traffic patterns.
• CPU usage: This enables filtering by CPU metric type (user, system, idle, etc.) so you can isolate the specific utilization dimension you’re troubleshooting.

List of EC2 Instance IDs

Use the following query to obtain the unique EC2 instance IDs reported within the timeframe specified by the Grafana time picker.

SELECT DISTINCT ON (ec2_instance_id) ec2_instance_id
FROM demo.fluentbit
WHERE ec2_instance_id IS NOT NULL

This SQL query uses a built-in filter to limit the execution of the statement to the time range of the dashboard.

For example, if your dashboard is set to the last 6h and your time column is called timestamp, then the following macro… 

AND $__timeFilter(timestamp)

Will be expanded to:

AND timestamp >= toDateTime(1742528649) AND timestamp <= toDateTime(1742550249)

You can confirm this in Grafana’s Query Inspector where you can view both the raw SQL and expanded versions of the following query:

SELECT DISTINCT ON (ec2_instance_id) ec2_instance_id
FROM demo.fluentbit
WHERE ec2_instance_id IS NOT NULL
AND $__timeFilter(timestamp) 

Network Interface Data for the EC2 Instance

The next filter selects the network interface. The network interface is a map(network_interface, value) so the following query retrieves all network interface keys.

SELECT DISTINCT ON (network, ec2_instance_id) network, ec2_instance_id
from demo.fluentbit
where ec2_instance_id IS NOT NULL
AND mapContains(network, 'enX0.rx.bytes')
limit 1

Which returns:

You can use this network data to query for bytes transmitted:

SELECT timestamp, ec2_instance_id, arrayFirst(x -> x IS NOT NULL, mapValues(mapExtractKeyLike(network, 'enX0.rx.bytes')))
from demo.fluentbit
where ec2_instance_id IS NOT NULL
AND mapContains(network, 'enX0.rx.bytes')

And a visualization of the data:

CPU Usage

The following query retrieves the CPU usage of the host:

SELECT timestamp, ec2_instance_id, arrayFirst(x -> x IS NOT NULL, mapValues(mapExtractKeyLike(cpu, 'cpu_p')))
from demo.fluentbit
where ec2_instance_id IS NOT NULL
AND cpu != '{}'

The query uses cpu_p from Fluent Bit, which is a measure of:

CPU usage of the overall system, this value is the summation of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system.

Next Steps

By combining Hydrolix and Fluent Bit, you get the flexibility of a high performance log processor and forwarder combined with a real-time data platform designed for petabyte scale. If you’re currently forwarding logs to a solution that’s forcing you to make compromises on your data (like high costs or short retention periods), consider forwarding those logs to Hydrolix instead.

• Interested in learning more about Hydrolix? Check out the docs.
• See how Hydrolix can support your use case with a guided demo.