With all the convenience Kubernetes operators provide in managing resources, it can still be a challenge to conveniently store, query, manage, and download the database logs from the pods. Since the pod logs are rotated after a certain period or size depending on the configuration and workload, it’s necessary to collect them to analyze for debugging and troubleshooting purposes. And who knows and feels the pain better than a DBA?
To collect pod logs locally, we usually do two things; set up Grafana Loki as a data source in Percona Monitoring and Management (PMM) and LogCLI. While it may sound like just two components to set up and deploy, it’s not easy for someone with limited Kubernetes expertise. The integration of Loki as the data source part is well demonstrated by my colleagues in the blog Store and Manage Logs of Percona Operator Pods with PMM and Grafana Loki; this blog post will focus on LogCLI configuration to query MongoDB pod logs.
CLI Installation instructions can be found here.
Once installation is completed, we’ll now proceed with the remaining steps.
Finding the respective Loki pod that will allow you to query MongoDB pod logs, assuming you follow the initial Grafana Loki deployment
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | $ kubectl get all -n loki-stack NAME READY STATUS RESTARTS AGE pod/loki-stack-promtail-n662l 1/1 Running 0 38m pod/loki-stack-promtail-nh2z9 1/1 Running 0 38m pod/loki-stack-promtail-cfwvj 1/1 Running 0 38m pod/loki-stack-promtail-7p2kb 1/1 Running 0 38m pod/loki-stack-promtail-85w6s 1/1 Running 0 38m pod/loki-stack-promtail-9qfdz 1/1 Running 0 38m pod/loki-stack-promtail-4fscq 1/1 Running 0 38m pod/loki-stack-0 1/1 Running 0 38m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/loki-stack-memberlist ClusterIP None <none> 7946/TCP 38m service/loki-stack-headless ClusterIP None <none> 3100/TCP 38m service/loki-stack ClusterIP 10.43.161.171 <none> 3100/TCP 38m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/loki-stack-promtail 7 7 7 7 7 <none> 38m NAME READY AGE statefulset.apps/loki-stack 1/1 38m |
The Objects that we are interested in are, “loki-stack-0” and service “loki-stack”.
We now need to forward the port default “3100” of “loki-stack-0” to our local machine where we have installed “Logcli”.
1 | $ kubectl port-forward pod/loki-stack-0 -n loki-stack 3100:3100 & |
Next is to export the LOKI_ADDR environment variable which will be used by Logcli to query the logs.
1 | $ export LOKI_ADDR=http://localhost:3100 |
We can now finally query or download the logs from any of the MongoDB deployment pods.
1 2 3 | $ logcli query --timezone=UTC --from="<ISODate>" --to="<ISODate>" --output=raw --limit=N '{ pod="<podName>"}' $ logcli query --output=raw --limit=N '{ pod="<podName>"}' --since 2h |
The above command can be adjusted based on our requirements so I’ve shown pretty basic flags to download the logs for a particular timeframe. More can be found with “logcli –help” or using this cheatsheet.
Similarly, let’s say I want to query a particular MongoDB pod to look for a COLLSCAN log line against the “demo” collection, run the below command,
1 2 | $ logcli query '{pod="<podName>"} |= `foo.demo` |= `COLLSCAN`' -q 2024-03-29T07:09:52Z {} {"t":{"$date":"2024-03-29T07:09:52.236+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn8679","msg":"Slow query","attr":{"type":"command","ns":"foo.demo","appName":"MongoDB Shell","command":{"find":"demo","filter":{"numbers":{"$in":[18,36,54]},"colors":{"$in":["red","blue"]},"a":{"$exists":true}},"lsid":{"id":{"$uuid":"19fb7392-3435-4b06-90c3-ed00fca9e6e5"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1711696183,"i":1}},"signature":{"hash":{"$binary":{"base64":"WG+2G+09CHodJR4xlXyzNSsnsM4=","subType":"0"}},"keyId":7351657218044854278}},"$db":"demo"},"planSummary":"COLLSCAN","keysExamined":0,"docsExamined":8258,"cursorExhausted":true,"numYields":8,"nreturned":7,"queryHash":"E5F27BCC","planCacheKey":"08DB3DED","reslen":1224,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":9}},"Global":{"acquireCount":{"r":9}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"level":"local","provenance":"implicitDefault"},"storage":{},"remote":"127.0.0.1:53486","protocol":"op_msg","durationMillis":10}} |
Hence, based on our usage, we can adjust the commands and get the desired results. This includes querying for different labels like a particular “pod” or a different “container” inside a pod like “pmm-client” or “backup-agent” for PMM and Percona Backup for MongoDB debugging.
I hope you enjoyed this blog. Happy querying!!
Reference: Turbocharging Percona Monitoring and Management With Loki’s Log-shipping Functionality
Get started with Percona Operator for MongoDB
