Viewing Metrics with SAS Viya Monitoring for Kubernetes | Part 1: Managing Metric Data Retention
In this video, David Stern shows you how to manage the storage and retention period for metric monitoring data in SAS Viya Monitoring for Kubernetes (https://github.com/sassoftware/viya4-.... We'll see how to find the size of the storage for metric data. We will see how to calculate the actual size of metric data in that storage. And we will see how to view and change the retention size and retention time limits for metric data stored by Prometheus.
David Stern has been at SAS for almost 15 years and focuses on SAS administration, especially observability, authorization and migrations.
CLI commands shown in the video:
List PVCs in the monitoring namespace
kubectl -n monitoring get pvc
List files in an NFS Prometheus data directory
ls -al /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-*/prometheus-db
Use du -xs to find size in kb of files in Prometheus data directories
for f in /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-* ; do sudo du -xs $f; done
Use command above with awk and bc to convert size to Mb
for f in /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-* ; do sudo du -xs $f | awk '{print $1}' ; done | awk '{print "scale=1;"$1"/1024"}' | bc | awk '{print $1"M"}'
As previous command, but convert to Gb
for f in /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-* ; do sudo du -xs $f | awk '{print $1}' ; done | awk '{print "scale=1;"$1"/(1024*1024)"}' | bc | awk '{print $1"G"}'
Prometheus command line flag filter string
--storage.tsdb.retention
Patch Prometheus pod spec to set retention time limit to 6 days
kubectl -n v4mmon patch prometheus v4m-prometheus --type merge --patch '{ "spec": { "retention": "6d" }}'
Patch Prometheus pod spec to set retention size limit to 15 GiB
kubectl -n v4mmon patch prometheus v4m-prometheus --type merge --patch '{ "spec": { "retentionSize": "15GiB" }}'
Note: NFS is not ideal for production use. If your OpenSearch PVCs use another storage class, the total file size in the OpenSearch PVCs should be measured in a way appropriate to that storage class.
Content Outline
0:00 – Managing Metric Data Retention
0:28 – Demo | SAS Viya Monitoring for Kubernetes Metric Monitoring Stack
Prometheus
Grafana
5:07 – Additional resources
Related Resources
◉ SAS Demo | Viewing Logs with SAS Viya Monitoring for Kubernetes: Managing Log Retention – • SAS Demo | Viewing Logs with SAS Viya...
◉ Managing monitoring data retention period and size in SAS Viya Monitoring for Kubernetes – https://communities.sas.com/t5/SAS-Co...
SUBSCRIBE TO THE SAS USERS YOUTUBE CHANNEL #SASUsers #LearnSAS
https://www.youtube.com/SASUsers?sub_...
ABOUT SAS
SAS is a trusted analytics powerhouse for organizations seeking immediate value from their data. A deep bench of analytics solutions and broad industry knowledge keep our customers coming back and feeling confident. With SAS®, you can discover insights from your data and make sense of it all. Identify what’s working and fix what isn’t. Make more intelligent decisions. And drive relevant change.
CONNECT WITH SAS
SAS ► https://www.sas.com
SAS Customer Support ► https://support.sas.com
SAS Communities ► https://communities.sas.com
SAS Analytics Explorers ► https://explorers.sas.com
Facebook ► / sassoftware
Twitter ► / sassoftware
LinkedIn ► / sas
Blogs ► https://blogs.sas.com
RSS ► https://www.sas.com/rss
