Skip to main content
NetApp Knowledgebase

Services on RHEL/CentOS docker based Storage Nodes failed

Views:
106
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • StorageGRID
  • RHEL/CentOS docker based Storage Node
  • Base OS RHEL/CentOS 7.7

Issue

  • Many services on all RHEL/CentOS docker based Storage Nodes failed thus unable to login GMI;
  • Unable to view Storage Node grid services status for resource temporarily unavailable error:

Command: /usr/local/servermanager/reader.rb
Result:
/usr/local/servermanager/reader.rb:28:in `initialize': can't create Thread: Resource temporarily unavailable (ThreadError)
    from /usr/local/servermanager/reader.rb:28:in `new'
    from /usr/local/servermanager/reader.rb:28:in `block in current_status'
    from /usr/local/servermanager/reader.rb:25:in `each'
    from /usr/local/servermanager/reader.rb:25:in `current_status'
    from /usr/local/servermanager/reader.rb:97:in `<main>'

  • Errors about Cassandra unable to create new native thread found in bycast-err.log:

Oct 27 15:46:11 dc1-sn1 [ERROR] org.apache.cassandra.concurrent.SEPWorker:145 - Failed to execute task, unexpected exception killed worker: unable to create new native thread
Oct 27 15:46:11 dc1-sn1 [ERROR] org.apache.cassandra.transport.Message:734 - Unexpected exception during request; channel = [id: 0x0b8287d4, L:/172.23.60.246:9042 - R:/172.23.60.247:38244]
Oct 27 15:46:11 dc1-sn1 #011java.lang.OutOfMemoryError: unable to create new native thread

  • Restart Storage Node / base OS do not help, resource temporarily unavailable errors may be seen in base OS log /var/log/storagegrid/nodes/<node-name>.log during restart:

[2020-10-28T23:57:25.115742]    INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable
[2020-10-28T23:57:26.115841]    INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable
[2020-10-28T23:57:28.116122]    INFO -- /usr/bin/initSG.sh: fork: retry: Resource temporarily unavailable

 

CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support