Monitoring server with Prometheus + Grafana

Server monitoring is always the major responsibility of system admin, whether you have 1 server or 10 servers, right tools always make your life easy.

I will share the installation for Prometheus + Node exporter + Grafana in this article. I will using docker for Prometheus and Grafana, but for Node exporter it’s not recommended to use docker because we are actually monitoring on the host, if you using docker, than you may need to enable the permission for the docker to access to the host which actually will make the implementation more complicated.

I will install the tools in my Ubuntu machine, but the tools also support for other Linux version as well as Windows server.

Basically, there are 3 major parts in the monitoring:

1. Prometheus is the main piece of this setup, it pulls metrics together from different services across different machines and collects them all in one place.
2. Node exporter is a small application that queries the operating system for a variety of metrics and exposes them over HTTP for other services to consume. Prometheus will query one or several Node Exporter instances to aggregate the metrics.
3. Grafana takes all the metrics Prometheus has aggregated and displays them as graphs and diagrams organized into dashboards.

I will using Nginx as map the domain name to the Prometheus and Grafana, so that I no need to directly access the site by the server IP and port. This is optional step if you not wish to install the Nginx server.

Step 1: Install Node exporter in the server to be monitor

sudo wget https://golang.org/dl/go1.14.7.linux-amd64.tar.gz && \ 
sudo tar -C /usr/local -xzf go1.14.7.linux-amd64.tar.gz && \ 
export PATH=$PATH:/usr/local/go/bin && \ 
go get github.com/prometheus/node_exporter && \ 
cd ${GOPATH-$HOME/go}/src/github.com/prometheus/node_exporter && \ 
make && \ 
sudo mv ${GOPATH-$HOME/go}/src/github.com/prometheus/node_exporter /usr/local/bin/ && \ 
sudo useradd -rs /bin/false node_exporter && \ 
sudo echo " 
[Unit] 
Description=Node 
Exporter After=network.target
[Service] 
User=node_exporter 
Group=node_exporter 
Type=simple 
ExecStart=/usr/local/bin/node_exporter/node_exporter
[Install] 
WantedBy=multi-user.target" > /etc/systemd/system/node_exporter.service && \ 
sudo systemctl daemon-reload && \ 
sudo systemctl start node_exporter && \ 
sudo systemctl status node_exporter && \ 
sudo systemctl enable node_exporter

Node Exporter is a single binary file that can run from anywhere on the system. This makes installation quite easy. Once it is up and running, point your browser at “<server_address>:9100/metrics” to see the metrics it is exposing. It isn’t really human readable, but that is the data Prometheus will collect.

For more detail on how to use the node_exporter, may refer to https://github.com/prometheus/node_exporter

Step 2: Installing Prometheus

Download and run the prometheus docker using following command

docker run -d -p 9090:9090 --name prometheus prom/prometheus

For more detail on how to use the container, may refer to: https://hub.docker.com/r/prom/prometheus

Login into the docker and edit the configuration file at /etc/prometheus

docker exec -it prometheus sh vi /etc/prometheus/prometheus.yml

Add the following targets into the confirm, you may add more target if you want to monitor more server.

static_configs: - targets: ['localhost:9090', '172.31.5.96:9100']

After that, save the configuration file and restart the docker by issue the command “docker restart prometheus”

This change instructs Prometheus to collect metrics from two services: itself (localhost:9090) and a local Node Exporter instance (172.31.5.96:9100), because i’m using Docker to host my Prometheus container, so in order to get to the host to pull the metric from the node_exporter, I need to specific the IP address of the host, if you are directly run the Prometheus in the server it self, you may just use localhost:9100.

Browse to [serverip]:9090 and will see the new target is added and both are in the up state.

Step 3: Installing Grafana for graphical dashboard

Download the run the grafana docker using following command

docker run -d -p 3000:3000 --name grafana grafana/grafana

After the grafana docker is up and running, you may browse to the page at [serverip]:3000 and the default username and password is admin/admin.

Once login to the grafana, you need to setup the source where it should pull the metrics from. On the left menu bar, select configuration and click on datasource.

Click on Add data source, select prometheus, enter the URL for your prometheus server.

Click on Save & Test.

To import a new dashboard, go to the left menu bar, click on the + sign, than click on import. For my testing, I use the dashboard from https://grafana.com/grafana/dashboards/11074, you may just paste the link into the load text box or copy the dashboard id and paste it in, then click on load.

If all the configuration is correct, you should be able to see your first dashboard on the page.

For more detail on how to use the container, may refer to https://grafana.com/docs/grafana/latest/installation/docker/ and https://hub.docker.com/r/grafana/grafana/

Step 4: (Optional) Configure the Nginx to map the domain to the docker

In order to make the URL more easy to remember, it’s always recommend to point a domain to the server, and only whitelist certain IP from your office or your VPN to access. For my case, I’m using Cloduflare to host my DNS, so I just need to add the A record to the DSN to point to my server.

On the server side, I need to add the domain map to the docker container port in order to the site to be able to browse using browser.

To do so, just open the Nginx configuration file either /etc/nginx/Nginx.conf or /etc/nginx/sites-enable/grafana.conf and add the following server block, than reload the nginx “sudo service nginx reload”.

The following script actually maps the domain https://prometheus.david-cheong.com to the prometheus application and https://grafana.david-cheong.com to grafana application.

server {
  listen 80; 
  listen [::]:80; 
  server_name prometheus.david-cheong.com;
  location / { 
    proxy_pass http://localhost:9090; 
  }
}
server {
  listen 80; 
  listen [::]:80; 
  server_name grafana.david-cheong.com;
  location / { 
    proxy_pass http://localhost:3000; 
  }
}

Now I able to browse my Prometheus using the domain https://prometheus.david-cheong.com and Grafana using the domain https://grafana.david-cheong.com.

Conclusion

If all the step is done correctly, you should see your first dashboard showing up and the data should keep on pull from Prometheus and showing on the Grafana dashboard.

By looking at the dashboard, you can easily monitor your server CPU used %, memory use % and other detail metric which generated by the node_exporter.

This is a very simple sample where you can do using these tools, there is plenty more you can do with these projects, including setting up automated alerts when certain thresholds are reached. But you now have the basic building blocks to get some basic monitoring up and running.

Originally published at https://tech.david-cheong.com on August 11, 2020.