Skip to main content
NetApp Knowledge Base

OnCommand Unified Manager database corruption caused by Disk 3 (Data Disk) running out of Disk Space

Views:
333
Visibility:
Public
Votes:
0
Category:
oncommand-unified-manager
Specialty:
om
Last Updated:

Applies to

OnCommand Unified Manager for Clustered Ontap

Issue

Unified Manager MYSQL service will not start.  Because MYSQL will not start all other services, also fail to start.
 
Look within the MYSQL error.log and you will see the following logged:
20XX-0X-18T04:11:30.978630Z 17096 [ERROR] Disk is full writing './unified-manager.~rec~' (Errcode: 16727840 - No space left on device). Waiting for someone to free space...
 
If the MYSQL Service can come up after corruption occurs, you may see the following logged within the MYSQL error.log file:
=====
20XX-0X -28T20:20:42.887959Z 17309 [ERROR] MYSQL_BIN_LOG::open failed to sync the index file.
20XX-0X -28T20:20:42.889354Z 17309 [ERROR] /usr/sbin/mysqld: Binary logging not possible. Message: Either disk is full or file system is read only while rotating the binlog. Aborting the server.
=====

If the MYSQL Service is able to start, MYSQL will attempt to recover/repair itself, however, it was unable to recover due to the corruption that was done.
=====
20XX-0X-28T20:20:44.383042Z 0 [Note] InnoDB: Starting crash recovery.
20XX-0X-28T20:20:44.399480Z 0 [Warning] InnoDB: Database page corruption or a failed file read of page [page id: space=23589, page number=2]. Trying to recover it from the doublewrite buffer.
20XX-0X-28T20:20:44.399563Z 0 [Note] InnoDB: Recovered page [page id: space=23589, page number=2] from the doublewrite buffer.
20XX-0X-28T20:20:44.399602Z 0 [Warning] InnoDB: Database page corruption or a failed file read of page [page id: space=23589, page number=3]. Trying to recover it from the doublewrite buffer.
20XX-0X-28T20:20:44.405526Z 0 [Warning] InnoDB: 16384 bytes should have been written. Only 8192 bytes written. Retrying for the remaining bytes.
20XX-0X-28T20:20:44.408596Z 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
 =====


You can also confirm by checking the amount of space that is available on Disk 3 / Data Disk.

=====
Note: As of Unified Manager 7.2 and above, Unified Manager will send alerts to a Configured email address, once the disk gets to approximately:  85% full.  This alerting is intended to provide administrators enough time to extend the Disk, as well as, extend the Disk 3 partition before Corruption occurs.
=====
Note: As of Unified Manager 7.3 and above, only the OVA version of Unified Manager will stop the Acquisition Service when the disk space reaches 95% consumed space.  This action is an effort to reduce the amount of information being written to the database. Understand that stopping the Acquisition service does not stop all writes to the MYSQL Database, as the OCIE Server service, and MYSQL services are still running. 

 

CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support