06. Prometheus & Grafana

Запуск Prometheus (мониторинг) и Grafana (визуализация) в kuberntes cluster с помощью heml

Добился, чтобы node.js приложение (backend) возвращало метрики.

$ curl backend.minikube.local/metrics
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total{app="prometheus-nodejs-app"} 0.8757879999999999

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total{app="prometheus-nodejs-app"} 0.291205

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total{app="prometheus-nodejs-app"} 1.166993

http://backend.minikube.local/metrics/

В HelmChart добавить http-metrics

name: http-metrics

apiVersion: v1
kind: Service
metadata:
  name: backend
  labels:
    app: backend
spec:
  ports:
    - name: http-metrics
      protocol: 'TCP'
      port: 80
      targetPort: 3000
  selector:
    app: backend

Создать ServiceMonitor

$ cat << 'EOF' | kubectl --namespace monitoring apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    name: backend-service-monitor
spec:
    selector:
        matchLabels:
          app: backend
    namespaceSelector:
      matchNames:
      - default
    endpoints:
    - port: http-metrics
      interval: 15s
    jobLabel: backend
EOF

$ kubectl --namespace monitoring get ServiceMonitor
NAME                                                 AGE
backend-service-monitor                              30s

http://localhost:9090/config

Появился backend-service-monitor

GitOps

И в Targets

GitOps

Конфигурация Prometheus обновляется каждые три минуты.

Применилось ли обновление можно посмотреть командой:

// Не работает
$ kubectl --namespace monitoring logs prometheus-stack-kube-prom-operator-56c4476bdd-5j2tm prometheus-config-reloader

$ kubectl describe endpoints backend
Name:         backend
Namespace:    default
Labels:       app=backend
              app.kubernetes.io/managed-by=skaffold
              skaffold.dev/run-id=52e10050-87b3-4d0d-8cc1-66fc5d657b24
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-02-09T16:42:06Z
Subsets:
  Addresses:          172.17.0.13
  NotReadyAddresses:  <none>
  Ports:
    Name          Port  Protocol
    ----          ----  --------
    http-metrics  3000  TCP

Events:  <none>

Для визуализации предлагают применить dashboard-configmap.yaml

https://gist.githubusercontent.com/vitkhab/02af337e83e66f33903f0320938135f0/raw/73b6c177be97b89e0749bb5ed122b452da094997/reddit-dashboard-configmap.yml

Должен появиться в Grafana dashboard Reddit Monitoring