HYDROLIX BLOG

Ponderings, insights and industry updates

Streaming Fastly CDN Logs to Hydrolix

February 10, 2021

Author: Tobin Sears | Sales Engineering, Hydrolix

Tags:

In this blog post, we’ll show you how to configure Fastly to stream logs to Hydrolix with just a few clicks and in less than 10 minutes.

This post is part of a series showing how to use Hydrolix and an open source dashboard to maximize your Fastly CDN observability quickly, cheaply, and in your own VPC. Start here to configure Fastly to stream log data to Hydrolix, then configure the Hydrolix streaming intake for those logs, see how to analyze Fastly transaction logs using Hydrolix, and learn how to visualize Fastly log data with an open source dashboard. You can also always refer to our Hydrolix documentation.

Hydrolix supports native integration with Fastly’s real-time log streaming service via an HTTPS Endpoint. With Hydrolix and an open source dashboard, you can maximize your Fastly CDN observability quickly, cheaply, and in your own VPC.

Similar documentation can also be found at the Fastly Integrations website: Fastly Log Streaming Guide for Hydrolix

Setup Fastly Log Streaming: HTTPS

Prerequisites

When sending logs to an HTTPS endpoint, Fastly requires proof that you control the domain name specified in the endpoint URL field. This is accomplished by using an HTTP challenge on a well-known path. Fortunately, Hydrolix has a pre-configured path for quick validation:

https://{{your_instance}}.hydrolix.live/.well-known/fastly/logging/challenge

By default, Hydrolix uses an asterisk (*) to allow any service to post to the HTTP endpoint.

Configure the HTTPS Logging Endpoint

In this example, we will create a new HTTPS logging endpoint that will be used to send real-time streaming logs to Hydrolix.

1. Login to Fastly

Login to manage.fastly.com and choose the appropriate service.

2. Create an HTTPS Logging Endpoint

Under the desired Fastly service configuration, select: * Logging * HTTPS Endpoint

Enter the following configuration options:

(For reference, the 14-field default log format for a Fastly HTTPS Logging Endpoint can be found here.)

Name: Name for the Hydrolix logging endpoint

Log format: In order to better leverage the value provided by the Hydrolix platform from a data compression and query performance perspective, we will be using an extended version of the Fastly streaming log format that increases the logged fields from the default of 14 to 64 in our example below. Below is the complete extended log format that you can cut and paste for our example.

Extended Fastly Log Format (click arrow to expand)

{
"service_id": "%{req.service_id}V", "service_version": "%{fastly_info.version}V", "time_start": "%{begin:%Y-%m-%dT%H:%M:%S}t", "time_end": "%{end:%Y-%m-%dT%H:%M:%S}t", "time_elapsed":%{time.elapsed.usec}V, "client_ip": "%{req.http.Fastly-Client-IP}V", "request": "%{req.request}V", "protocol": "%{req.proto}V", "host": "%{req.http.Fastly-Orig-Host}V", "origin_host": "%{req.http.Host}V", "url": "%{cstr_escape(req.url)}V", "is_ipv6":%{if(req.is_ipv6, "true", "false")}V, "is_tls":%{if(req.is_ssl, "true", "false")}V, "tls_client_protocol": "%{cstr_escape(tls.client.protocol)}V", "tls_client_servername": "%{cstr_escape(tls.client.servername)}V", "tls_client_cipher": "%{cstr_escape(tls.client.cipher)}V", "tls_client_cipher_sha": "%{cstr_escape(tls.client.ciphers_sha )}V", "tls_client_tlsexts_sha": "%{cstr_escape(tls.client.tlsexts_sha)}V", "is_h2":%{if(fastly_info.is_h2, "true", "false")}V, "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V, "h2_stream_id": "%{fastly_info.h2.stream_id}V", "request_referer": "%{cstr_escape(req.http.Referer)}V", "request_user_agent": "%{cstr_escape(req.http.User-Agent)}V", "request_accept_content": "%{cstr_escape(req.http.Accept)}V", "request_accept_language": "%{cstr_escape(req.http.Accept-Language)}V", "request_accept_encoding": "%{cstr_escape(req.http.Accept-Encoding)}V", "request_accept_charset": "%{cstr_escape(req.http.Accept-Charset)}V", "request_connection": "%{cstr_escape(req.http.Connection)}V", "request_dnt": "%{cstr_escape(req.http.DNT)}V", "request_forwarded": "%{cstr_escape(req.http.Forwarded)}V", "request_via": "%{cstr_escape(req.http.Via)}V", "request_cache_control": "%{cstr_escape(req.http.Cache-Control)}V", "request_x_requested_with": "%{cstr_escape(req.http.X-Requested-With)}V", "request_x_forwarded_for": "%{cstr_escape(req.http.X-Forwarded-For)}V", "status": "%{resp.status}V", "content_type": "%{cstr_escape(resp.http.Content-Type)}V", "cache_status": "%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V", "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V, "response_age": "%{cstr_escape(resp.http.Age)}V", "response_cache_control": "%{cstr_escape(resp.http.Cache-Control)}V", "response_expires": "%{cstr_escape(resp.http.Expires)}V", "response_last_modified": "%{cstr_escape(resp.http.Last-Modified)}V", "response_tsv": "%{cstr_escape(resp.http.TSV)}V", "geo_datacenter": "%{server.datacenter}V", "geo_city": "%{geoip.city}V", "geo_country_code": "%{geoip.country_code}V", "geo_continent_code": "%{geoip.continent_code}V", "geo_region": "%{geoip.region}V", "req_header_size":%{req.header_bytes_read}V, "req_body_size":%{req.body_bytes_read}V, "resp_header_size":%{resp.header_bytes_written}V, "resp_body_size":%{resp.body_bytes_written}V, "socket_cwnd":%{client.socket.cwnd}V, "socket_nexthop": "%{client.socket.nexthop}V", "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V, "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V, "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V, "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V, "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V, "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V, "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V, "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V, "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V, "socket_ploss":%{client.socket.ploss}V }

URL: The streaming API endpoint for your Hydrolix instance: https://<hydrolix-instance-name>.hydrolix.live/ingest/event

Maximum logs: 0
Maximum bytes: 0

EXPAND “Advanced options”

Content type: application/json
Custom header name: x-hdx-table
Custom header value: <hydrolix_project_name>.<hydrolix_table_name>
Method: POST
JSON log entry format: Newline delimited
Select a log line format: Blank
Placement: Format Version Default

Using your own certificate authority (CA)?
Leave all values empty or customize as required


Congratulations! Fastly has now been configured to send real-time streaming logs to the Hydrolix endpoint. Read on to see how to configure Hydrolix to receive the Fastly logs.

Share Now

Leave a comment

Your email address will not be published. Required fields are marked *