Prometheus is an open-source systems monitoring and alerting tool. Many different types of data are collected by different tools and are forwarded to Prometheus via various types of exporters. For example, the Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics. We developed an exporter for Skydive that reports metrics of individual captured network flows. The connector translates data from Skydive captured flows into a format that can be consumed by Prometheus. The first implementation of the Skydive-Prometheus connector periodically provides the byte transfer counts for each network connection under observation. The code can be easily tailored to provide additional flow information.
The structure of the interacting components is depicted in the following figure.
You can either download a statically linked version of Skydive or you can build it on your own.
To download a pre-compiled statically linked version of Skydive, run the following command.
To build your own version of Skydive, follow the instructions on Build Documentation to install prerequisites and prepare your machine to build the Skydive code. Enter the Skydive root directory and build the code.
Adjust settings in skydive.yml, if desired.
The skydive binary is created in
Alternatively, run a Skydive analyzer on one host and a Skydive agent on each of your hosts.
On one machine:
On each machine:
Be sure to set the field
analyzers in the
skydive.yml to point to the analyzer.
Create Skydive-Prometheus Connector Configuration File
Start from the
prom_sky_con.yml.default in the
prom_sky_con.yml is used as a parameter when running
prom_sky_con from the command line.
prom_sky_con.yml has the following fields:
The paramaters in the yml file are inherited from the generic skydive-flow-exporter.
The main entries that may need adjustment are those involving the address of the Skydive analyzers and subscriber.
connection_timeout parameters under
store.prom_sky_con may also be adjusted.
store.prom_sky_con.port is the network port at which Prometheus will probe the connector to pull the metrics.
This port number must match the value specified in the Prometheus config file for the Skydive-Prometheus connector.
store.prom_sky_con.connection_timeout is the number of seconds that have elapsed with no traffic on a connection before we assume the connection is inactive.
Fields specific to Skydive-Prometheus connector
|pipeline.store.prom_sky_con.port||int||port number used between Prometheus and connector to scrape the supplied metrics|
|pipeline.store.prom_sky_con.connection_timeout||int||number of seconds with no traffic before considering a connection as dead|
Build and Deploy the Skydive-Prometheus connector
Generate and Capture Flows
Via the Skydive WebUI setup captures and generate traffic which should result in the
prom_sky_con pipeline sending flows to the Prometheus.
To connect to the GUI, open a web browser to the address of the Skydive analyzer at port 8082. If running the browser on the same machine as the analyzer, then connect to
You should see the topology of your network, perhaps something like the following image.
Make sure you are on the
Captures view and press on
Fill in the
Targets fields by placing the cursor in the first
Interface field and then use the mouse to point to your network endpoint that you want to capture.
To capture a particular flow end-to-end, fill in the second
Interface field in a similar manner with the other network endpoint.
Then press the
Start button on the GUI.
Alternatively, you can start the capture of flows from the command line with a command like the following:
The command specifies to capture all flows that match the gremlin expression - in this case, all devices that have name ‘eth0’.
Generate some network traffic on the interfaces you specified to capture. For example, run some iperf traffic between entities and capture the flow via the Skydive GUI.
iperf3 (on ubuntu/debian):
Run iperf server:
Either on the same machine or on another machine, run the iperf client:
It is also possible to generate network traffic using the inject feature of Skydive, either from the command line or through the Skydive GUI.
Generator page and specify the source and destination nodes between which to generate the network traffic, as well as the characterization of the flow you want to generate.
In the Prometheus config file, add a stanza specifying the location and port used by the Skydive-Prometheus connector.
Restart Prometheus to pick up the new stanza in the config file.
Output of Skydive-Prometheus connector
|skydive_network_connection_total_bytes||int||total bytes transferred on connection||initiator_ip, target_ip, initiator_port, target_port, direction, node_tid|
Wait for 60 seconds and then check that we have data arriving in the Prometheus GUI.
Look for a metric beginning with
skydive such as
You can display a graph of all the flows that have been captured that report the
To see a graph of the bandwidth (bytes per second instead of total bytes transferred) used by flows, you can graph on the variable
The data shown in the graph should contain the following fields:
Each such tuple is associated with timestamps and corresponding values that appear in the graph.
The graph in the Prometheus GUI may look something like the following:
The graph shows the data rate of various connections that are tracked by Skydive.