StorageGRID expansion stuck at Starting Cassandra and streaming data
Applies to
- StorageGRID 11.6
- bare-metal
Issue
- StorageGRID expansion stuck at "step 4" with stage:
"Starting Cassandra and streaming data (0.0% streamed)
Error: Failed to start cassandra
/var/local/log/servermanager.log
shows:2023-01-24 14:48:17 +0000 |cassandra| startingcassandra
2023-01-24 14:48:17 +0000 |cassandra| cassandraended
2023-01-24 14:48:20 +0000 |cassandra| Too manyfailedattempts, entering error state
2023-01-24 14:48:20 +0000 | cassandra| cassandra ended
- Manually starting Cassandra results in error state:
# service cassandra start
starting cassandra ... done
# service cassandra status
cassandra in error state.
- Cassandra
system.log
is empty, andcassandra.errlog
is not being generated in/var/local/log/casssandra
on the node.
- Changing to cassandra user fails:
# sudo su - cassandra
su: warning: cannot change directory to/var/lib/cassandra: Permission denied
su: failed to execute /bin/bash:Permission denied
- Both
/var/lib/cassandra
folder and bash executable show unexpected permissions/folder ownerships: - Cassandra home folder with incorrect owner/permissions:
# ls -al/var/lib/cassandra/
drwxr-x--- 2 root root6 Oct5 07:16 .
drwxr-x--- 1 root root 82 Oct5 07:16 ..
Expected output:
# ls -al /var/lib/cassandra
drwxr-xr-x 1 cassandra cassandra 4096Jan 25 22:57 .
drwxr-xr-x 1 rootroot4096 Feb 102022 ..
- Bash with incorrect permissions:
# ls -al /bin/bash
-rwxr-x--- 1 root root 1099016 May 15 2017 /bin/bash
Expected output:
# ls -al /bin/bash
-rwxr-xr-x 1 root root 1099016 May 15 2017 /bin/bash