StorageGRID appliance performance degradation following E-Series upgrade to 11.70.1GA
Applies to
- SG5760
- SG6060 with one or more expansions trays
- SANtricity OS 11.70.1GA
- Storage RAID Mode: DDP16 (16+2 stripe)
- In StorageGRID management interface, check under Nodes >
Node_Name
> Hardware
- In StorageGRID management interface, check under Nodes >
Issue
- StorageGRID appliances that match above criteria may experience significant performance degradation after upgrade to 11.70.1GA.
- The performance degradation may appear in various forms. Below are few signatures that those appliances may experience:
- Increased E-Series drive impending failures due to Stagnant I/O. Following events can be seen in SANtricity System Manager under Support > Events.
Summary of impending drive failure detected by controller
Drive copy triggered by a controller detected impending drive failure
- Increased disk read/write latencies. Check under Support > Diagnostics page in StorageGRID management interface.
- Storage node appliance unplanned reboot highlighted by node down related alerts "
Unable to communicate with node
". The storage node base OSsyslog
indicates the node was rebooted by the slow I/O monitoring script as a recovery mechanism:
Jul XX XX:XX:10 localhost wdogd[1990]: SIMN: '/usr/bin/storagegrid-slowio-monitor report_state emergency BLOCK_DEVICE_RANGEDB_001 BLOCK_DEVICE_RANGEDB_002 BLOCK_DEVICE_RANGEDB_003 BLOCK_DEVICE_RANGEDB_004 BLOCK_DEVICE_RANGEDB_005 BLOCK_DEVICE_RANGEDB_006 BLOCK_DEVICE_RANGEDB_008 BLOCK_DEVICE_RANGEDB_009 BLOCK_DEVICE_RANGEDB_010 BLOCK_DEVICE_RANGEDB_011 BLOCK_DEVICE_RANGEDB_012 BLOCK_DEVICE_RANGEDB_013 BLOCK_DEVICE_RANGEDB_014 BLOCK_DEVICE_RANGEDB_015 warning BLOCK_DEVICE_RANGEDB_000 BLOCK_DEVICE_RANGEDB_007 BLOCK_DEVICE_VAR_LOCAL' executed successfully, WANTS reboot
Jul XX XX:XX:22 localhost systemd[1]: Stopping StorageGRID host service...