Skip to main content

Troubleshooting

The docs cover troubleshooting of prometheus & grafana issues on server side. i.e if any service is down then how you can check configs and logs on server.

Basics Debugging

  • There are 3 services running on prometheus server. i.e prometheus, grafana, node exporter.

  • Configuration Files

    • Prometheus:
    • /etc/prometheus/prometheus.yml
    • Grafana:
    • /etc/grafana/grafana.ini
    • Node exporter:
    • /etc/systemd/system/node_exporter.service

  • To check the status of services use below commands.

    • Prometheus:
    • sudo systemctl status prometheus.service
    • Grafana:
    • sudo systemctl status grafana-server.service
    • Node exporter:
    • sudo systemctl status node_exporter.service

  • To check the logs of services use below commands.

    • Prometheus:
    • journalctl -u prometheus -n100 -f
    • Grafana:
    • tail -f /var/log/grafana/grafana.log
    • Node exporter:
    • journalctl -u node_exporter -n100 -f

Common Scenario

  • If you see any issues like prometheus or granfana is not working.
  • Then login into the peomtheus server & check logs and service status of prometheus and grafana.
  • Then check disk,mem,cpu utilization as well.
  • If prometheus is down due to high disk utilisation, then consult with team/lead first for appropriate action.