Services on RHEL/CentOS docker based Storage Nodes failed
Applies to
- StorageGRID
- RHEL/CentOS docker based Storage Node
- Base OS RHEL/CentOS 7.7
Issue
- Many services on all RHEL/CentOS docker based Storage Nodes failed thus unable to login GMI;
- Unable to view Storage Node grid services status for resource temporarily unavailable error:
Command: /usr/local/servermanager/reader.rb
Result:
/usr/local/servermanager/reader.rb:28:in `initialize': can't create Thread: Resource temporarily unavailable (ThreadError)
from /usr/local/servermanager/reader.rb:28:in `new'
from /usr/local/servermanager/reader.rb:28:in `block in current_status'
from /usr/local/servermanager/reader.rb:25:in `each'
from /usr/local/servermanager/reader.rb:25:in `current_status'
from /usr/local/servermanager/reader.rb:97:in `<main>'
- Errors about Cassandra unable to create new native thread found in bycast-err.log:
Oct 27 15:46:11 dc1-sn1 [ERROR] org.apache.cassandra.concurrent.SEPWorker:145 - Failed to execute task, unexpected exception killed worker: unable to create new native thread
Oct 27 15:46:11 dc1-sn1 [ERROR] org.apache.cassandra.transport.Message:734 - Unexpected exception during request; channel = [id: 0x0b8287d4, L:/172.23.60.246:9042 - R:/172.23.60.247:38244]
Oct 27 15:46:11 dc1-sn1 #011java.lang.OutOfMemoryError: unable to create new native thread
- Restart Storage Node / base OS do not help, resource temporarily unavailable errors may be seen in base OS log /var/log/storagegrid/nodes/<node-name>.log during restart:
[2020-10-28T23:57:25.115742] INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable
[2020-10-28T23:57:26.115841] INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable
[2020-10-28T23:57:28.116122] INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable