Recently in my practice, I faced a significant challenge: extracting application logs per day from Loki in the Kubernetes environment. When cold, the application writes about 60 lines per minute, and when someone interacts with the application, it can write 2000-5000 lines of logs per minute – it turns out that was necessary to get more than 300,000 lines of logs. The setup did not include a configured log export, and the primary log viewing tool was Grafana, which imposes a 5000-line limit on log retrieval. Increasing this limit was not feasible as it would significantly strain our resources and was unnecessary for this one-time task. Additionally, accessing logs directly from the Kubernetes pod was not an option due to storage limitations within the pod itself.
So, I need to download logs directly from Loki without changing configurations.
To ensure that the query will use to search for logs is correct, follow these steps:
Grafana
> Explore
label
to filter logs by the service.Use the operation
filter to display lines containing the desired date.
Example query:
{instance="our-service"} |= `2024-07-12`
Install LogCli:
Set Loki address:
export LOKI_ADDR=http://localhost:8000
Port forwarding:
kubectl --namespace loki port-forward svc/loki-stack 8000:3100
Extract logs:
logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log
--limit is set with a high value to ensure all logs are captured.
--since is set to 72 hours to cover a sufficient time range.
This entire process took approximately 10 minutes, resulting in a file with the complete application logs for the specified date. If needed, this process can be further optimized or automated.