Skip to main content
NetApp Knowledge Base

OnCommand Unified Manager database corruption caused by Disk 3 (Data Disk) running out of Disk Space

  1. Last updated
  2. Save as PDF
Views:
625
Visibility:
Public
Votes:
0
Category:
oncommand-unified-manager
Specialty:
legacy
Last Updated:

Applies to

OnCommand Unified Manager for Clustered Ontap

Issue

Unified Manager MYSQL service will not start.  Because MYSQL will not start all other services, also fail to start.
 
Look within the MYSQL error.log and you will see the following logged:
20XX-0X-18T04:11:30.978630Z 17096 [ERROR] Disk is full writing './unified-manager.~rec~' (Errcode: 16727840 - No space left on device). Waiting for someone to free space...
 
If the MYSQL Service can come up after corruption occurs, you may see the following logged within the MYSQL error.log file:
=====
20XX-0X -28T20:20:42.887959Z 17309 [ERROR] MYSQL_BIN_LOG::open failed to sync the index file.
20XX-0X -28T20:20:42.889354Z 17309 [ERROR] /usr/sbin/mysqld: Binary logging not possible. Message: Either disk is full or file system is read only while rotating the binlog. Aborting the server.
=====

If the MYSQL Service is able to start, MYSQL will attempt to recover/repair itself, however, it was unable to recover due to the corruption that was done.
=====
20XX-0X-28T20:20:44.383042Z 0 [Note] InnoDB: Starting crash recovery.
20XX-0X-28T20:20:44.399480Z 0 [Warning] InnoDB: Database page corruption or a failed file read of page [page id: space=23589, page number=2]. Trying to recover it from the doublewrite buffer.
20XX-0X-28T20:20:44.399563Z 0 [Note] InnoDB: Recovered page [page id: space=23589, page number=2] from the doublewrite buffer.
20XX-0X-28T20:20:44.399602Z 0 [Warning] InnoDB: Database page corruption or a failed file read of page [page id: space=23589, page number=3]. Trying to recover it from the doublewrite buffer.
20XX-0X-28T20:20:44.405526Z 0 [Warning] InnoDB: 16384 bytes should have been written. Only 8192 bytes written. Retrying for the remaining bytes.
20XX-0X-28T20:20:44.408596Z 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
 =====

You can also confirm by checking the amount of space that is available on Disk 3 / Data Disk.

=====
Note: As of Unified Manager 7.2 and above, Unified Manager will send alerts to a Configured email address, once the disk gets to approximately:  85% full.  This alerting is intended to provide administrators enough time to extend the Disk, as well as, extend the Disk 3 partition before Corruption occurs.
=====
Note: As of Unified Manager 7.3 and above, only the OVA version of Unified Manager will stop the Acquisition Service when the disk space reaches 95% consumed space.  This action is an effort to reduce the amount of information being written to the database. Understand that stopping the Acquisition service does not stop all writes to the MYSQL Database, as the OCIE Server service, and MYSQL services are still running. 

 

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.