StorageGRID MySQL service enters error state during Secondary Admin Node upgrade
Applies to
- StorageGRID
- Secondary Admin Node upgrade from 11.8 to 11.9
- Secondary Admin Node's platform is Red Hat Enterprise Linux (RHEL) Container
Issue
- MySQL service fails during Secondary Admin Node StorageGRID OS upgrade (11.8 -> 11.9).
- Grid Manager displays
SVSTandMISSAlarms:
/usr/local/servermanager/reader.rbcommand reports Database Engine (MySQL) is in Error state:
Collection Time 2025-09-11 11:40:43 UTCHost Name admin-node2IP Address 10.145.254.199Operating System Kernel 4.18.0 VerifiedOperating System Environment Debian 11.8 VerifiedStorageGRID Release 11.8.0.11 VerifiedStorage Subsystem VerifiedDatabase Engine package MySQL ErrorNetwork Monitoring 11.8.0 RunningTime Synchronization 1:4.2.8p15+dfsg Runningams 11.8.0.11 Runningnms 11.8.0.11 Runningssm 11.8.0.11 Runningmi 11.8.0.7 Stoppeddynip 11.8.0.9 Runningmiscd 11.8.0.9 Runningnginx 11.8.0 Runningtomcat 9.0.43 Stoppedgrafana 9.5.1 Runningmgmt api 11.8.0.9 Stoppednginx gw 11.8.0.8 Runningprometheus 11.8.0 Runningacct tunnel 11.8.0.8 Runningpersistence 11.8.0 Runningalertmanager 11.8.0 Runningjaeger agent 11.8.0 RunningattrDownPurge 11.8.0.7 StoppedattrDownSamp1 11.8.0.7 Stoppednode exporter 1.1.2+ds Runningsg snmp agent 11.8.0 Runningjaeger collector 11.8.0 Runninglambda arbitrator 11.8.0 Runningupdate snmp firewall 11.8.0 Runningmysql.errlog indicates that MySQL crashes and enters a restart loop.
2025-09-11 2:30:41 0 [Note] InnoDB: Starting shutdown...
・・・
2025-09-11 2:36:49 0 [ERROR] [FATAL] InnoDB: IO Error: 5 during async read of 16384 bytes, for file 62, returned 0
250911 2:36:49 [ERROR] mysqld got signal 6 ;
・・・- Other related logs display related errors below:
servermanager.log2025-09-11 02:37:58 +0000 | mysql | Exception encountered: #<Timeout::Error: execution expired>. See /var/local/log/mysql-bycastify.log2025-09-11 02:38:02 +0000 | attrDownPurge | attrDownPurge ended2025-09-11 02:38:02 +0000 | mi | mi ended2025-09-11 02:38:02 +0000 | mi | stopping attrDownPurge2025-09-11 02:38:02 +0000 | attrDownSamp1 | attrDownSamp1 ended2025-09-11 02:38:03 +0000 | mysql | mysqld_safe was unexpectedly running. Sending TERM to PID 131532025-09-11 02:38:04 +0000 | mi | stopping attrDownSamp12025-09-11 02:38:04 +0000 | attrDownSamp1 | attrDownSamp1 ended2025-09-11 02:38:05 +0000 | mi | stopping tomcat2025-09-11 02:38:05 +0000 | tomcat | tomcat ended2025-09-11 02:38:08 +0000 | mysql | mysqld_safe was unexpectedly running. Sending KILL to PID 131532025-09-11 02:38:10 +0000 | mgmt-api | starting mgmt-api2025-09-11 02:38:10 +0000 | mysql | mysql ended2025-09-11 02:38:10 +0000 | mgmt-api | Failed to update mysql db schema. See nms.errlog for details.2025-09-11 02:38:11 +0000 | mgmt-api | mgmt-api ended2025-09-11 02:38:11 +0000 | mysql | force-restarting mgmt-api2025-09-11 02:38:19 +0000 | mi | waiting for mysql, waiting 2s to try again2025-09-11 02:38:20 +0000 | mgmt-api | waiting for mysql, waiting 2s to try again2025-09-11 02:38:25 +0000 | sg-snmp-agent | waiting for mgmt-api, waiting 22s to try again2025-09-11 02:38:28 +0000 | mi | waiting for mysql, waiting 4s to try againmysql-bycastify.logI, [2025-09-11T02:37:33.949567 #13119] INFO -- : mariadbd was running with PID 17000unexpected mysqld processes running after mysqladmin shutdownE, [2025-09-11T02:37:33.949824 #13119] ERROR -- : unexpected mysqld processes running after mysqladmin shutdownI, [2025-09-11T02:37:38.952354 #13119] INFO -- : running cmd: pgrep --exact mariadbd 2>&1I, [2025-09-11T02:37:38.962452 #13119] INFO -- : mariadbd was running with PID 17338I, [2025-09-11T02:37:38.962685 #13119] INFO -- : running cmd: mysqladmin --defaults-file=/etc/my.default.cnf shutdown 2>&1error in mysqladmin shutdown, see /var/local/log/mysql-bycastify.logE, [2025-09-11T02:37:38.971827 #13119] ERROR -- : error in mysqladmin shutdown, see /var/local/log/mysql-bycastify.logE, [2025-09-11T02:37:38.971965 #13119] ERROR -- : mysqladmin shutdown details:mysqladmin: connect to server at 'localhost' failederror: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)'Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!