Unable to create subsystems with NVMe/TCP
Applies to
- Ontap 9
- Trident
Issue
- Unable to create subsystems with NVMe/TCP. Below are the log snippets from Openshift cluster (redhat) and from ONTAP event log -
- Below errors are reported on multiple PODs:
MapVolume.SetUpDevice failed for volume "pvc-b50447d9-c91b-4be1-ac87-b4ee005e5xxx" : rpc error: code = Internal desc = failed to stage volume: failed to connect subsystem nqn.1992-08.com.netapp:sn.a4e4eee357d911efa9ebd039ea5bxxxx:subsystem.s_trident_pvc_b50447d9_c91b_4be1_ac87_b4ee005e59xx to 10.216.2.5: exit status 1; failed to connect subsystem- Below event is reported on Ontap end -
DEBUG nvmf.spdk.err: NVMe/TCP controller limits reached on this node. Queue slots provisioned 69568, additional queue slots requested:128, currently supported queue slots:69632.No more NVMe connects via NVMe/TCP LIFs on this node are allowed.- Encountering scale limits with numerous RWX (Read Write Many, mountable on multiple nodes) volumes.
- Unable to scale to the required number of RWX volumes.
- There is no impact on RWO volumes since per-node subsystems are utilized.
