StorageGRID mysql service in error state on admin node
- Views:
- 138
- Visibility:
- Public
- Votes:
- 0
- Category:
- storagegrid
- Specialty:
- sgrid
- Last Updated:
- 6/20/2025, 8:00:34 PM
Applies to
- NetApp StorageGRID
- Admin Node(s)
- StorageGRID software 11.9
- SG110/SG1100 appliance
Issue
- StorageGRID admin node not responsive in Grid Manager
- When checking the admin node via SSH or using the StorageGRID application logs, the following services will continue to report as stopped i
INFO -- [INSG] services still stopped: attrDownPurge attrDownSamp1 mgmt-api mi tomcat
- When checking the admin node via SSH or using the StorageGRID application logs, the mysql and mgmt-api services fail to start. This can be found in the
servermanager.log
located under /var/local/log.
11:00:11 +0000 | mysql | starting mysql
11:00:13 +0000 | mysql | mysql ended
11:00:13 +0000 | mysql | force-restarting mgmt-api
11:00:13 +0000 | mgmt-api | mgmt-api ended
11:00:25 +0000 | mgmt-api | waiting for mysql, waiting 2s to try again
11:00:35 +0000 | mgmt-api | waiting for mysql, waiting 4s to try again
11:00:44 +0000 | mgmt-api | mgmt-api ended
11:00:44 +0000 | mysql | force-restarting mi
11:00:44 +0000 | mi | mi ended
11:00:55 +0000 | mgmt-api | waiting for mysql, waiting 2s to try again
- In mysql.err logs
[ERROR] InnoDB: Corrupted page identifier at 448993352621; set innodb_force_recovery=1 to ignore the record.
[Note] InnoDB: End of log at LSN=448993352621
[ERROR] InnoDB: Plugin initialization aborted with error Generic error
[Note] InnoDB: Starting shutdown...
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
- Other errors in mysql.err could be:
2025-06-12 17:12:34 0 [ERROR] InnoDB: Tablespace 16198 was not found at ./mi/tempwritetree.ibd.
2025-06-12 17:12:34 0 [Note] InnoDB: Set innodb_force_recovery=1 to ignore this and to permanently lose all changes to the tablespace.
2025-06-12 17:12:34 0 [ERROR] InnoDB: Plugin initialization aborted with error Tablespace not found
2025-06-02 22:32:10 24 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: /var/local/mysql_ibdata/mysql_data/ib1
2025-06-02 22:32:10 24 [ERROR] InnoDB: File '/var/local/mysql_ibdata/mysql_data/ib1' is corrupted
2025-06-02 22:32:10 0x7fe0681756c0 InnoDB: Assertion failure in file ./storage/innobase/trx/trx0purge.cc line 262
InnoDB: Failing assertion: undo_page
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
250602 22:32:10 [ERROR] mysqld got signal 6 ;