Puppet is one of the first infrastructure automation tools we used at Inuits. Our Puppet git repositories have accumulated ten years of commits.
In parallel, we are constantly bringing new tools to our infra, and over the last years, Prometheus usage has grown both internally and in our customers' infrastructure.
It was time for the two projects to meet each other - and therefore, I have contributed native PuppetDB service discovery support to the Prometheus server.
Puppet and Prometheus
Puppet and Prometheus are two popular Open Source projects, and they have users in common. It does not come as a surprise then that the community already bridged the gap between both tools.
Camptocamp’s prometheus-puppetdb-sd was, to date the go-to approach for Service Discovery between Puppet and Prometheus. That project was popular and integrated with VoxPupuli’s Prometheus Puppet module.
Prometheus service discoveries
One important part of Prometheus service discoveries is that we should implement it in a way that we do not have business logic inside the discovery.
In Puppet, it translates as, you should be able to use the service discovery with PuppetDB, not requiring any specific Puppet module to be installed. Therefore, I could not integrate prometheus-puppetdb-sd as-is. I had to start from scratch and make it very generic.
From scratch is a big word. I took some inspiration from Camptocamp’s approach, and I even contacted them before writing the first line of code. They provided me with excellent insights over PuppetDB APIs that I could use to start the service discovery with the correct endpoints.
The result is a very simplified architecture by connecting Prometheus directly with PuppetDB. Any resource will work, so there is no need for specific puppet modules either.
The PuppetDB Service Discovery is using the Puppet Query Language - PQL. Any resource or node that is in PuppetDB can be discovered by Prometheus
Example: monitoring Node Exporters
Relabeling is a crucial part of Prometheus configuration. It enables you to take the output from service discoveries and turn that into the desired target.
The following simple query will start monitoring all the servers that have the
package node_exporter
defined:
scrape_configs:
- job_name: node
puppetdb_sd_configs:
- url: http://puppetdb.local:8080
query: 'resources { type = "Package" and title = "node_exporter" }'
port: 9100
relabel_configs:
- source_labels: [__meta_puppetdb_certname]
target_label: instance
- source_labels: [__meta_puppetdb_environment]
`target_label: environment
Example: Apache vhosts
Now, let’s do something different.
Instead of monitoring an exporter, let’s use the Blackbox Exporter to automatically verify the availability of all the deployed vhosts.
scrape_configs:
- job_name: apache_vhosts
metrics_path: /probe
params:
module: [http_2xx]
puppetdb_sd_configs:
- url: http://puppetdb.local:8080
query: 'resources { type = "Apache::Vhost" }'
include_parameters: true
relabel_configs:
- source_labels: [__meta_puppetdb_parameter_servername]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115
Note that this example is using include_parameters
. The discovery would then
try to include resource parameters as meta labels. You should only do this if
your Prometheus web UI is protected enough or you do not have secrets in the
results of your query.
Replacing prometheus-puppetdb-sd
If you use the prometheus-puppetdb-sd project and your Scrape::Job resources only have one target, you can easily switch to the PuppetDB native service discovery:
scrape_configs:
- job_name: 'puppetdb-scrape-jobs'
puppetdb_sd_configs:
# This example uses the Prometheus::Scrape_job exported resources.
# https://github.com/camptocamp/prometheus-puppetdb-sd
# This example is compatible with Prometheus-puppetdb-sd,
# if the exported Prometheus::Scrape_job resources have at most one target.
- url: https://puppetdb.example.com
query: 'resources { type = "Prometheus::Scrape_job" and exported = true }'
include_parameters: true
tls_config:
cert_file: prometheus-public.pem
key_file: prometheus-private.pem
ca_file: ca.pem
relabel_configs:
- source_labels: [__meta_puppetdb_certname]
target_label: certname
- source_labels: [__meta_puppetdb_parameter_targets]
regex: '(.+),?.*'
replacement: $1
target_label: __address__
- source_labels: [__meta_puppetdb_parameter_job_name]
target_label: job
- regex: '__meta_puppetdb_parameter_labels_(.+)'
replacement: '$1'
action: labelmap
Conclusion
Connecting Prometheus natively with PuppetDB enables you to discover any resource you have defined in your Puppet environment. One of the advantages is that you do not need to instrument your Puppet modules - it works directly with any resources.