Services on container-based Storage Node on RHEL or CentOS fail
Applies to
- NetApp StorageGRID
- Container-based Storage Node on RHEL or CentOS
- RHEL or CentOS 7.7
Issue
- Services on multiple container-based Storage Nodes on RHEL or CentOS fail:
- Unable to login Grid Manager if ADC (Administrative Domain Controller) service on multiple nodes are impacted
- May experience alarms of CASA (Data Store Status) Data Store Down
- Unable to view grid services status for
Resource temporarily unavailable:
Command: /usr/local/servermanager/reader.rb
Result:
/usr/local/servermanager/reader.rb:28:in `initialize': can't create Thread: Resource temporarily unavailable (ThreadError)
- Errors about
unable to create new native threadin bycast-err.log:
MMM DD hh:mm:ss dc1-sn1 [ERROR] org.apache.cassandra.service.CassandraDaemon:231 - Exception in thread Thread[ScheduledTasks:1,5,main]
MMM DD hh:mm:ss dc1-sn1 #011java.lang.OutOfMemoryError: unable to create new native thread
- Similar errors in Cassandra system.log:
ERROR [EXPIRING-MAP-REAPER:1] YYYY-MM-DD hh:mm:ss,838 HeapUtils.java (line 66) The heap histogram could not be generated due to the following error:
java.io.IOException: Cannot run program "jcmd": error=11, Resource temporarily unavailable
...
ERROR [EXPIRING-MAP-REAPER:1] YYYY-MM-DD hh:mm:ss,843 JVMStabilityInspector.java (line 85) OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: unable to create new native thread
- Errors in /var/log/storagegrid/nodes/<node-name>.log on base OS:
[YYYY-MM-DDThh:mm:ss.115742] INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable