How to locate root cause for sCache usage increasement
Applies to
Element OS
Description
This document describes how to find the reason for sCache usage increasement.
- When Secondary Cache (sCache) usage exceeds 25%, Alert
failedSpaceTooFull
will be generated. - When sCache usage doesn't decrease under a certain level during upgrade or powercycle using sfinstall. Usually sCache usage decrease soon once the node is online.
Example of /var/log/sf-master.info on Management Node :
2020-11-18T04:53:36.648666Z xxxxxx sfinstall[13979]: Waiting for caches to fall below threshold
2020-11-18T04:53:36.648730Z xxxxxx sfinstall[13979]: Cache Size Current: 21900943360 Target: 5000000000 Difference: 16900943360
2020-11-18T04:58:37.371643Z xxxxxx sfinstall[13979]: scacheBytesInUse = [0, 0, 0, 0, 0, 0, 0, 0, 22100951040, 0, 0, 0, 0, 4100177920]
2020-11-18T04:58:37.372485Z xxxxxx sfinstall[13979]: Cache Size Current: 22100951040 Target: 5000000000 Difference: 17100951040
2020-11-18T05:03:38.440873Z xxxxxx sfinstall[13979]: scacheBytesInUse = [0, 0, 0, 0, 0, 0, 0, 1519616, 22800982016, 0, 0, 0, 0, 39452672]
2020-11-18T05:03:38.441049Z xxxxxx sfinstall[13979]: Cache Size Current: 22800982016 Target: 5000000000 Difference: 17800982016
2020-11-18T05:08:39.062177Z xxxxxx sfinstall[13979]: scacheBytesInUse = [0, 0, 0, 0, 0, 0, 0, 0, 23601020928, 0, 0, 0, 0, 0]
2020-11-18T05:08:39.062350Z xxxxxx sfinstall[13979]: Cache Size Current: 23601020928 Target: 5000000000 Difference: 18601020928
2020-11-18T05:13:39.691315Z xxxxxx sfinstall[13979]: scacheBytesInUse = [950272, 0, 0, 0, 0, 0, 2248704, 0, 24701067264, 679936, 0, 0, 0, 181628928]
2020-11-18T05:13:39.691498Z xxxxxx sfinstall[13979]: Cache Size Current: 24701067264 Target: 5000000000 Difference: 19701067264
2020-11-18T05:18:40.310952Z xxxxxx sfinstall[13979]: scacheBytesInUse = [0, 0, 0, 0, 0, 0, 2129920, 0, 25601101824, 0, 0, 0, 0, 200007680]
2020-11-18T05:18:40.311542Z xxxxxx sfinstall[13979]: Cache Size Current: 25601101824 Target: 5000000000 Difference: 20601101824
2020-11-18T05:23:40.925168Z xxxxxx sfinstall[13979]: scacheBytesInUse = [0, 0, 0, 0, 0, 0, 0, 0, 26201124864, 0, 0, 0, 0, 6729728]
2020-11-18T05:23:40.925343Z xxxxxx sfinstall[13979]: Cache Size Current: 26201124864 Target: 5000000000 Difference: 21201124864
2020-11-18T05:28:41.646470Z xxxxxx sfinstall[13979]: scacheBytesInUse = [1454080, 0, 0, 0, 0, 0, 0, 0, 26301128704, 0, 0, 0, 0, 4931584]