How to manually rotate the logs in StorageGRID
Applies to
- StorageGRID 7.5
- StorageGRID 8.1
- StorageGRID 8.5
- StorageGRID 9.0
Description
The common location for the StorageGRID server debug logs is the /var/local/log/directory.
The most recent log messages are placed in the bycast.log file, while the older log messages are periodically rotated to the bycast.log.n file.
Note: This log format covers all the grid software, with the exception of the following:
- Server Manager and Gridstat
- Content Management System (CMS)
- Network Management System - Web server and back end
Example:
In an instance, it was discovered that the /var/local directory ran out of space, causing some of the services to crash.
- The following error was reported in the bycast-err.log:
Jun 14 13:40:53 vch-vgh-mt1 ADE: |20671179 19221 000010 ARNI ETEL 2011-06-14T13:40:53.530316| CRITICAL 0667 ARNI: Assertion Failed [libs/cxd/CXD_DiskQueue.cc:667 UINT32 CXD_DiskQueue_Enqueue(CXD_DiskQueue*, CXD_Memory*)] ASSERT FAILED ("CXD_DiskQueue library unable to write (ERR_SYS_NOSPC)" == false) is false
- The following message was reported in the Kernel log:
Sep 15 20:10:30 servername syslog-ng[2596]: io.c: do_write: write() failed (errno 28), No space left on device
- The
df-hcommand output displayed the following:
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p1 5.0G 2.5G 2.3G 52% /
udev 1014M 128K 1014M 1% /dev
/dev/cciss/c0d0p3 60G 60G 0 100% /var/local
/dev/cciss/c0d1p1 71G 69G 1.9G 98% /var/local/mysql_ibdata
/dev/mapper/fsgvg-fsglv 64G 724K 64G 1% /fsg
/dev/cciss/c0d3p1 139G 29G 110G 21% /var/local/rangedb/0
On checking the output of the du -hx --max-depth=1 command, the logs had stopped rotating correctly. The bycast.log was of 52GB size and the ha.log was of 6.2GB size. Both the files were deleted manually to clear the space.
The ls -ltr directory contained the following:
-rw-r----- 1 root root 108572147 Apr 8 2012 bycast.log.11.gz
-rw-r----- 1 root root 108704513 Apr 29 11:00 bycast.log.10.gz
-rw-r----- 1 root root 108611113 May 20 04:00 bycast.log.9.gz
-rw-r----- 1 root root 108418726 Jun 9 18:00 bycast.log.8.gz
-rw-r----- 1 root root 107182703 Jun 29 16:00 bycast.log.7.gz
-rw-r----- 1 root root 104442374 Jul 20 16:00 bycast.log.6.gz
-rw-r----- 1 root root 105747418 Aug 10 15:00 bycast.log.5.gz
-rw-r----- 1 root root 105832058 Aug 31 18:00 bycast.log.4.gz
-rw-r----- 1 root root 105523073 Sep 21 20:00 bycast.log.3.gz
-rw-r----- 1 root root 105568607 Sep 28 22:00 bycast.log.2.gz
-rw-r----- 1 root root 67548330 Oct 02 12:59 bycast-err.log.1
-rw-r----- 1 root root 231233800 Oct 12 15:33 bycast-err.log
-rw-r----- 1 root root 1075737021 Oct 02 22:00 bycast.log.1
-rw-r----- 1 root root 52335833898 Oct 12 23:13 bycast.log