Trident provisioned LUN turns offline due to volume full and filesystem goes 'read-only'
Applies to
- Trident
- SAN
Issue
- Linux reports critical space allocation errors:
LINUX kernel: blk_update_request: critical space allocation error, dev xxx
LINUX kernel: blk_update_request: critical space allocation error, dev xxx
- PVC mount goes in Read-only:
/dev/mapper/3600a098038314474443f5339496b724c on /var/lib/kubelet/pods/07fa7645-1294-44b4-bd61-98f5120a8e6b/volumes/kubernetes.io~csi/pvc-18bc95d2-c6e1-4918-922f-f4ca6c0c4cf9/mount type ext4 (ro,relatime,seclabel,stripe=16)
- 'read-only' file system errors generated when trying to write data:
level=warn ts=<time> caller=grpc_logging.go:76 method=/logproto.Pusher/Push duration=234.509µs msg=gRPC err="create new segment file: open <filepath>: read-only file system"
- Data ONTAP sends an alert that a Trident volume (ontap-san backend) is full
[wafl_exempt13: wafl.vol.full:alert]: Insufficient space on volume c1_san_ssd_dc1_pvc_19a8fa1a_b8f1_4cd9_9fb8_xxxx@vserver:0b2a7d0a-0441-11ee-a219-xxxxxxxxxxx to perform operation. 72.0KB was requested but only 8.00KB was available.
- The volume and LUN are provided by Trident automatically according to below rule:
ONTAP SAN configuration options and examples
Total volume size = [(PVC requested size) / (1 - (snapshotReserve percentage) /100)] * 1.1