06. Prometheus & Grafana

Запуск Prometheus (мониторинг) и Grafana (визуализация) в kuberntes cluster с помощью heml


Добился, чтобы node.js приложение (backend) возвращало метрики.


$ curl backend.minikube.local/metrics
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total{app="prometheus-nodejs-app"} 0.8757879999999999

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total{app="prometheus-nodejs-app"} 0.291205

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total{app="prometheus-nodejs-app"} 1.166993


http://backend.minikube.local/metrics/


В HelmChart добавить http-metrics


name: http-metrics
apiVersion: v1
kind: Service
metadata:
    name: backend
    labels:
        app: backend
spec:
    ports:
        - name: http-metrics
          protocol: 'TCP'
          port: 80
          targetPort: 3000
    selector:
        app: backend


Создать ServiceMonitor

$ cat << 'EOF' | kubectl --namespace monitoring apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    name: backend-service-monitor
spec:
    selector:
        matchLabels:
          app: backend
    namespaceSelector:
      matchNames:
      - default
    endpoints:
    - port: http-metrics
      interval: 15s
    jobLabel: backend
EOF


$ kubectl --namespace monitoring get ServiceMonitor
NAME                                                 AGE
backend-service-monitor                              30s


http://localhost:9090/config


Появился backend-service-monitor


GitOps


И в Targets

GitOps


Конфигурация Prometheus обновляется каждые три минуты.

Применилось ли обновление можно посмотреть командой:

// Не работает
$ kubectl --namespace monitoring logs prometheus-stack-kube-prom-operator-56c4476bdd-5j2tm prometheus-config-reloader


$ kubectl describe endpoints backend
Name:         backend
Namespace:    default
Labels:       app=backend
              app.kubernetes.io/managed-by=skaffold
              skaffold.dev/run-id=52e10050-87b3-4d0d-8cc1-66fc5d657b24
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-02-09T16:42:06Z
Subsets:
  Addresses:          172.17.0.13
  NotReadyAddresses:  <none>
  Ports:
    Name          Port  Protocol
    ----          ----  --------
    http-metrics  3000  TCP

Events:  <none>


Для визуализации предлагают применить dashboard-configmap.yaml


https://gist.githubusercontent.com/vitkhab/02af337e83e66f33903f0320938135f0/raw/73b6c177be97b89e0749bb5ed122b452da094997/reddit-dashboard-configmap.yml


Должен появиться в Grafana dashboard Reddit Monitoring