Prometheus in Kubernetes
This is an experimental Prometheus setup for monitoring Kubernetes services that expose prometheus-friendly metrics through address http://service_address:service_port/metrics.
The purpose of the setup is to gather performance-related metrics during load tests and analyze them to find and fix bottlenecks.
- Pick a local directory for promdash. It can be any directory, preferably one which is stable and which you don’t mind keeping around. Then (in our case, we use /mnt/promdash , just run this docker command
docker run -v /mnt/promdash:/mnt/promdash -e DATABASE_URL=sqlite3:/mnt/promdash/file.sqlite3 prom/promdash ./bin/rake db:migrate. In the future, we might use mysql as the promdash database, however, in any case, this 1 time db setup step is required.
Now quickly confirm that /mnt/promdash/file.sqlite3 exists, and has a non-zero size, and make sure its permissions are open so that containers can read from it. For example:
[jay@rhbd kubernetes]$ ls -altrh /mnt/promdash/ total 20K drwxr-xr-x. 6 root root 4.0K May 6 23:12 .. -rwxrwxrwx 1 root root 12K May 6 23:33 file.sqlite3
Looks open enough :).
Now, you can start this pod, like so
kubectl create -f contrib/prometheus/prometheus-all.json. This ReplicationController will maintain both prometheus, the server, as well as promdash, the visualization tool. You can then configure promdash, and next time you restart the pod – you’re configuration will be remain (since the promdash directory was mounted as a local docker volume).
Finally, you can simply access localhost:3000, which will have promdash running. Then, add the prometheus server (localhost:9090)to as a promdash server, and create a dashboard according to the promdash directions.
You can launch prometheus, promdash, and pushgateway easily, by simply running.
kubectl create -f contrib/prometheus/prometheus-all.json
This will launch the pushgateway such that prometheus is collecting metrics from it automatically.
NOTE: If you plan to use the push gateway to push metrics (for example, we use it in e2e tests)
you will want to have a stable IP that you can provide the push gateway from, and you may need to modify the NodePort stuff for your cloud provider.
And then, create corresponding services, like this.. note that the pushgateway service may/may not be functional, but we leave it in for good measure.
kubectl create -f contrib/prometheus/prometheus-service.json ; kubectl create -f contrib/prometheus/pushgateway-service.json
Now, you can access the service
wget 10.0.1.89:9090 , and build graphs.
How it works
This is a v1 api based, containerized prometheus ReplicationController, which scrapes endpoints which are readable on the KUBERNETES service (the internal kubernetes service running in the default namespace, which is visible to all pods).
Use kubectl to handle auth & proxy the kubernetes API locally, emulating the old KUBERNETES_RO service.
The list of services to be monitored is passed as a command line arguments in the yaml file.
The startup scripts assumes that each service T will have 2 environment variables set
Each can be configured manually in yaml file if you want to monitor something that is not a regular Kubernetes service. For example, you can add comma delimited endpoints which can be scraped like so…
- -t - KUBERNETES_RO,MY_OTHER_METRIC_SERVICE
For regular Kubernetes services the env variables are set up automatically and injected at runtime.
By default the metrics are written to a temporary location (that can be changed in the the volumes section of the yaml file). Prometheus’ UI is available at port 9090.
- We should publish this image into the kube/ namespace.
- Possibly use Postgres or mysql as a promdash database.
- stop using kubectl to make a local proxy faking the old RO port and build in real auth capabilities.