Viewing Metrics with SAS Viya Monitoring for Kubernetes | Part 1: Managing Metric Data Retention

David Stern has been at SAS for almost 15 years and focuses on SAS administration, especially observability, authorization and migrations.

CLI commands shown in the video:

# List PVCs in the monitoring namespace
kubectl -n monitoring get pvc

# List files in an NFS Prometheus data directory
ls -al /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-*/prometheus-db

# Use du -xs to find size in kb of files in Prometheus data directories
for f in /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-* ; do sudo du -xs $f; done

# Use command above with awk and bc to convert size to Mb
for f in /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-* ; do sudo du -xs $f | awk '{print $1}' ; done | awk '{print "scale=1;"$1"/1024"}' | bc | awk '{print $1"M"}'

# As previous command, but convert to Gb
for f in /srv/nfs/kubedata/*-prometheus-v4m-prometheus-db-prometheus-v4m-prometheus-* ; do sudo du -xs $f | awk '{print $1}' ; done | awk '{print "scale=1;"$1"/(1024*1024)"}' | bc | awk '{print $1"G"}'

# Prometheus command line flag filter string

# Patch Prometheus pod spec to set retention time limit to 6 days
kubectl -n v4mmon patch prometheus v4m-prometheus --type merge --patch '{ "spec": { "retention": "6d" }}'

# Patch Prometheus pod spec to set retention size limit to 15 GiB
kubectl -n v4mmon patch prometheus v4m-prometheus --type merge --patch '{ "spec": { "retentionSize": "15GiB" }}'

Note: NFS is not ideal for production use. If your OpenSearch PVCs use another storage class, the total file size in the OpenSearch PVCs should be measured in a way appropriate to that storage class.

Content Outline
0:00 – Managing Metric Data Retention
0:28 – Demo | SAS Viya Monitoring for Kubernetes Metric Monitoring Stack
* Prometheus
* Grafana
5:07 – Additional resources

Related Resources

SUBSCRIBE TO THE SAS USERS YOUTUBE CHANNEL #SASUsers #LearnSAS

ABOUT SAS
SAS is a trusted analytics powerhouse for organizations seeking immediate value from their data. A deep bench of analytics solutions and broad industry knowledge keep our customers coming back and feeling confident. With SAS®, you can discover insights from your data and make sense of it all. Identify what’s working and fix what isn’t. Make more intelligent decisions. And drive relevant change.

CONNECT WITH SAS