HYDROLIX BLOG

Ponderings, insights and industry updates

Indexing Journald logs with Vector

March 4, 2022

Author: David Sztykman |

Tags: , , ,

Vector is a great way to send logs into Hydrolix!
As Vector supports HTTP output and formatting the data in JSON it’s very easy to setup and index everything in Hydrolix.

In this example we’ll instal vector on an Ubuntu server, get the logs from journald and index using streaming to Hydrolix.

Install vector on Ubuntu

Following the installation guide you can use the apt package available:

curl -1sLf   'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo -E bash
sudo apt-get install vector

Vector will be installed and launch as a service on your server, we’ll modify the configuration to send the data to Hydrolix.

Setting Hydrolix table and transform for journald

Hydrolix requires the table and the schema to index your journald logs, you can use the following example to create the project and table.
To use this example you can refer to this article on how to setup VSCode.

### Global variable to replace with your own needs
@host = $customer
@projectname = sample
@tablename = journald
@transformname = journald_transform
@username = "YYYYYYY"
@password = "XXXXXXXX"

#### DO NOT MODIFY BELOW ####

@base_url = https://{{host}}.hydrolix.live/config/v1/
@post_endpoint = https://{{host}}.hydrolix.live/ingest/event
@sql_endpoint = https://{{host}}.hydrolix.live/query/




### Login authentication get access token  and UUID Org variable
# @name login
POST {{base_url}}login
Content-Type: application/json

{
    "username": {{username}},
    "password": {{password}}
}
### Store, parse the login response body to store the access token and organization id
@access_token = {{login.response.body.auth_token.access_token}}
@org_id = {{login.response.body.orgs[0].uuid}}


### Create a new project using the variable {{projectname}}
# @name new_project
POST  {{base_url}}orgs/{{org_id}}/projects/
Authorization: Bearer {{access_token}}
Content-Type: application/json

{
    "name": "{{projectname}}",
    "org": "{{org_id}}"
}
### Store, parse project ID from response
@projectid = {{new_project.response.body.uuid}}


### Create a new table named {{tablename}} in the {{projectname}}
# @name new_table
POST  {{base_url}}orgs/{{org_id}}/projects/{{projectid}}/tables/
Authorization: Bearer {{access_token}}
Content-Type: application/json

{
    "name": "{{tablename}}",
    "project": "{{projectid}}",
    "description": "Journald Logs from Vector",
    "settings": {
        "merge": {
            "enabled": true
        }
    }
}
### Store, parse table ID from response
@tableid = {{new_table.response.body.uuid}}


#### Creates a a transform for the json format and upload to our table
# @name new_transform
POST {{base_url}}orgs/{{org_id}}/projects/{{projectid}}/tables/{{tableid}}/transforms/
Authorization: Bearer {{access_token}}
Content-Type: application/json

{
    "name": "{{transformname}}",
    "description": "Transform JournalD logs from Vector",
    "table": "{{tableid}}",
    "type": "json",
    "settings": {
        "is_default": true,
        "compression": "gzip",
        "format_details": {
            "flattening": {
                "active": false
            }
        },
        "output_columns": [
            {
                "name": "PRIORITY",
                "datatype": {
                    "type": "uint8",
                    "index": true
                }
            },
            {
                "name": "SYSLOG_FACILITY",
                "datatype": {
                    "type": "uint8",
                    "index": true
                }
            },
            {
                "name": "SYSLOG_IDENTIFIER",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "SYSLOG_TIMESTAMP",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_AUDIT_LOGINUID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_AUDIT_SESSION",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_BOOT_ID",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_CAP_EFFECTIVE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_CMDLINE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_COMM",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_EXE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_GID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_MACHINE_ID",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_PID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }   
            },
            {
                "name": "_SELINUX_CONTEXT",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SOURCE_REALTIME_TIMESTAMP",
                "datatype": {
                    "type": "epoch",
                    "index": true,
                    "format": "us",
                    "resolution": "ms"
                }
            },
            {
                "name": "_SYSTEMD_CGROUP",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_INVOCATION_ID",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_OWNER_UID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_SESSION",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_SLICE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_UNIT",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_USER_SLICE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_TRANSPORT",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_UID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "__MONOTONIC_TIMESTAMP",
                "datatype": {
                    "type": "uint64",
                    "index": true

                }
            },
            {
                "name": "__REALTIME_TIMESTAMP",
                "datatype": {
                    "type": "epoch",
                    "index": true,
                    "format": "us",
                    "resolution": "ms"
                }
            },
            {
                "name": "host",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "message",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "source_type",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "timestamp",
                "datatype": {
                    "type": "datetime",
                    "format": "2006-01-02T15:04:05.999999Z",
                    "resolution": "ms",
                    "primary": true
                }
            }
        ]
    }
}

Configuring Vector

After settings up your Hydrolix configuration and table you can setup Vector to start sending logs !
You can use your favourite text editor to edit /etc/vector/vector.toml
This is what the configuration looks like:

[sources.journal_source]
type = "journald"

[sinks.hydrolix]
type = "http"
inputs = ["journal_source"]
uri = "https://$host.hydrolix.live/ingest/event"
encoding = "json"
compression = "gzip"

[sinks.hydrolix.headers]
  X-HDX-Table = "$project.$table"
  X-HDX-Transform = "$transform"

Obviously you need to replace your hostname, your project, table and transform name.
Those are specify in the example above.

Share Now