In this post, we will show you how to setup Grafana to visualize your Fastly CDN data which you have streamed to Hydrolix.
This post is part of a series showing how to use Hydrolix and an open source dashboard to maximize your Fastly CDN observability quickly, cheaply, and in your own VPC. Before following these quick instructions for visualizing Fastly data with Grafana, see how to analyze Fastly transaction logs using Hydrolix, how to configure Fastly to stream logs into Hydrolix with an HTTPS endpoint and how to configure the Hydrolix streaming intake for those logs. You can also always refer to our Hydrolix documentation.
Grafana is a popular open source visualization and analytics application that has gained significant traction with the advent of always-on/connected-devices and time-series oriented datasets. It allows you to query, visualize, alert on, and explore metrics independent of where they are stored.
In this final segment of the Working with Fastly CDN Logs tutorial, we’ll focus on visualizing our CDN log data leveraging a new dashboard we have developed for Grafana. You can find the full dashboard configuration file for use with this walkthrough below:
Download: Fastly Composite Dashboard for Hydrolix
Grafana comes bundled as part of the Hydrolix deployment stack and is available for immediate use upon activation. The integrated version is a standard off-the-shelf installation and requires no special attention. It’s been added for ease of deployment and configuration purposes only. Alternatively, a separate instance can be installed and configured using the standard Grafana installation process.
Grafana Pie Chart Plugin
The Fastly dashboard provides a variety of panels and visualizations for use with the Fastly CDN log data. A few of the panels make use of the Grafana Pie Chart plugin which is not included in the core release.
To install the Pie Chart plugin on a local Grafana instance, use the grafana-cli tool from the command line:
grafana-cli plugins install grafana-piechart-panel
After installing any new plugins (or updating Grafana), it’s always good to check for plugin updates. While plugins are not updated automatically, you will be notified when updates are available in the Configuration/Plugins section of the Grafana UI.
If you are on the CLI, you can update all plugins using the following command:
grafana-cli plugins update-all
Full documentation on how to restart the Grafana service for your specific operating system can be found here. For Ubuntu (and other systems systems that leverage init.d), restart Grafana using the following command from the CLI:
sudo service grafana-server restart
sudo /etc/init.d/grafana-server restart
With the dependencies taken care of, we’re ready to login to the Grafana user interface and begin the configuration process.
To login to the Grafana instance included within the Hydrolix client stack:
https://<hydrolix_hostname>.hydrolix.live/grafana [insert your own Hydrolix Hostname]
Please change your credentials after the intial login.
Add the Clickhouse Data Source
In order to connect to your Hydrolix database instance, you’ll need to use the Clickhouse Datasource for Grafana.
Navigate to Configuration/Data Sources and add the Clickhouse data source:
Enter the Required Configuration Settings for your Hydrolix Database:
Name: Enter a name for your Hydrolix data source
URL: The Query API Endpoint for your Hydrolix instance using the format https://
Explore the Data
Once your Grafana instance is connected to your Hydrolix cluster, navigate to the Explore menu and perform some basic data exploration prior to importing the dashboard configuration file.
Populate Query Builder with the Hydrolix Project, Table, and associated DateTime for the Fastly CDN log data:
Using the Query Builder menu, select the following for the
FROM field in the Query Builder menu:
- The Hydrolix Project that contains the desired Fastly tables. Once the appropriate project is selected, any associated tables will automatically populate into the query builder table pulldown menu. The project that contains our tables is called
- The Table that contains the Fastly CDN log data. Our data is contained within the
- The DateTime field for supporting time-series functions and macros. The Clickhouse data source plugin will automatically attempt to detect the appropriate Column:DateTime or Column:Timestamp field from the selected table. It’s
time_startin this case.
This process essentially binds the Project, Table and Datetime fields to the data source connection for use in subsequent queries and integrated variables.
Basic data exploration and validation
If the Clickhouse data source successfully connects to your Hydrolix instance, you will be presented with a basic time-series graph that will allow you to experiment with your dataset. It’s worth spending some time switching between time-series and table-based formats to get familiar with how the results of a query are returned. Ironically, one of the trickiest parts to creating a compelling panel has more to do with time-bucketing and time-windowing versus returning the right data. Fortunately, Grafana (and the Clickhouse datasource) provide some extremely useful macros and functions that assist in the process.
Don’t forget to display the raw generated SQL as you work through your queries. It’s critical to understanding how something like the
$timeSeries macro is interpreted.
Preparing the Dashboard Configuration File for Import
Prior to importing the Fastly dashboard for Hydrolix, there are a few suggested modifications that need to be made to the configuration file. The dashboard configuration file, also known as the JSON Model for the dashboard, is a data structure that defines the entire dashboard. This includes all of the general and panel-related settings, layout, and supporting queries.
Since it’s obviously in JSON format, a source-code editor like Visual Studio Code can help with the editing process.
- Replace all instances of
"datasource": "Hydrolix"with that of your Clickhouse datasource name. If you chose Hydrolix for your datasource name, you are good to go!
- Modify the
"url":value to reflect that of your Hydrolix instance hostname. You will need to replace the hostname portion of https://
"icon": "external link",
"title": "Hydrolix Version Information",
- The majority of the panels leverage a dashboard query variable (
$servers) that allows for filtering by origin server. The relevant section is highlighted below. You will need to potentially replace
"datasource": "Hydrolix"(if you haven’t done so already) and change
fastly.xlogsto the Hydrolix
project.tablethat houses your Fastly CDN logs. Updating Query Builder with your specific Hydrolix
datetimesettings at the datasource level will not automatically remap the dashboard variable settings.
"definition": "SELECT host FROM fastly.xlogs",
"description": "Origin host list to be used in filtering various panels",
"label": "Origin Host",
"query": "SELECT host FROM fastly.xlogs",
Importing the Grafana Fastly Dashboard for Hydrolix
Once the above modifications have been made, it’s as easy as uploading the JSON Model file.
If the panels are not updating after import, please go into Edit on one of the panels and check to ensure that the Hydrolix
datetime settings are properly set in Query Builder configuration. Remedy if required.
Congratulations! You are now observing your Fastly CDN logs using Grafana and Hydrolix.
At this point, your new Grafana dashboard for visualizing Fastly CDN logs on Hydrolix should be filled with a plethora of colors and metrics. Keep in mind, this is a composite dashboard that has been designed for flexibility and inspiration. Please customize and adapt the layout, panels, and supporting queries to make it your own.
If you have any additional suggestions on how we can make future iterations better, of if there’s another tutroial you would like to see, please don’t hesitate to reach out. We welcome input and feedback from the community.