StorageGRID Webscale Upgrade from 10.4.x to 11.0.x hung on the storage node

Applies to

• StorageGRID 10.4
• StorageGRID 10.4.x
• StorageGRID 11.0
• StorageGRID 11.0.x

Issue

GUI upgrade page shows 'Upgrading C* node XXXXX... ' for long time .

From gdu-server.log on admin node :

I, [2018-03-17T03:56:30.672452 0031358] INFO -- gdu-server: Executing command `find -L /var/local/cassandra/data/0/ -path "*snapshots*" -prune -o -name "*-jb-*Data.db" -print 2>/dev/null` on <IP_Address>
I, [2018-03-17T03:56:30.783922 0031358] INFO -- gdu-server: Executing command `find -L /var/local/cassandra/data/0/ -path "*snapshots*" -prune -o -name "*-ka-*Data.db" -print 2>/dev/null` on <IP_Address>
E, [2018-03-17T04:00:45.825342 0031358] ERROR -- gdu-server: Failure occurred during Upgrade Cassandra
E, [2018-03-17T04:00:45.825459 0031358] ERROR -- gdu-server: #<Bycast::Tasks::GduServer::UpgradeCassandra::Error: Cassandra upgrade failed. Error: /usr/local/sbin/upgrade-cassandra:169:in `upgrade_sstables': Unable to upgrade sstables: Aborted upgrading sstables for atleast one column family in keyspace storagegrid, check server logs for more information. (CassandraUpgrade::Error)
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
        at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
        at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)