How to resolve memory issues with Active IQ Unified Manager
Applies to
- Active IQ Unified Manager (AIQUM)
- OnCommand Unified Manager (OCUM)
Answer
-
Determine the amount of reserved memory
-
Utilizing reserved memory should be full or none, as fractional reserves can cause issues
-
-
Check AIQUM memory recommendations
Software Version | System Requirements |
Active IQ Unified Manager 9.6 and above |
VMware OVA - Virtual infrastructure or hardware system requirements |
Microsoft Windows - Virtual infrastructure or hardware system requirements | |
Red Hat and CentOS Linux - Virtual infrastructure or hardware system requirements | |
OnCommand Unified Manager 9.5 and earlier | Virtual infrastructure or hardware system requirements |
-
Review best practice recommendations for sizing
- See page 10 - "Unified Manager scalability"
-
Make memory adjustments as needed
- VMware Docs - Change the Memory Configuration
- VMware Docs - Allocate Memory Resources to a Virtual Machine in the VMware Host Client
-
Confirm that no ballooning or swapping is occurring on the virtual machine.
- VMware Docs - Memory (Data Counters)
- VMware Docs - Memory (MB)
-
Linux only - Linux may stop Unified Manager processes to free memory
- On Linux platforms, check for OOM "Out of Memory" killer ending processes to free up memory.
- In rare instances Red Hat and CentOS platforms kill Unified Manager services / processes to free up memory.
Dec 8 22:00:18 um-hostname kernel: Out of memory: Kill process 45678 (java) score 228 or sacrifice child
Dec 8 22:00:18 um-hostname kernel: Killed process 45988 (sh) total-vm:122868kB, anon-rss:260kB, file-rss:0kB, shmem-rss:0kB
Dec 8 22:00:19 um-hostname systemd: mysqld.service: main process exited, code=killed, status=9/KILL
Dec 8 22:00:19 um-hostname systemd: Unit mysqld.service entered failed state.
Dec 8 22:00:19 um-hostname systemd: mysqld.service failed.
Dec 8 22:00:20 um-hostname systemd: mysqld.service holdoff time over, scheduling restart.
Dec 8 22:00:20 um-hostname systemd: Stopped MySQL Server.
Dec 8 22:00:20 um-hostname systemd: Starting MySQL Server...
Additional Information
Common memory issues include
- Unified Manager application is running slow:
- Services take a long time to come online.
- WebGUI is slow.
- Backups take a very long time ( and possibly fail ).
- Unified Manager application CPU and/or memory utilization are extremely high; swap / page file is in use.
- Unified Manager application experiences inconsistent behavior:
- Services hanging.
- Services restarting.
- Services not starting at all (especially after upgrade)
- Occasional issues with alert notifications.
- Inventory / performance acquisition polls may not complete within their normal 15 minute / 5 minute cycles.
- Other abnormal (unexplainable) inconsistent behavior when running in a virtual environment.
- VMware ESXi displays one or more of the following:
- Memory Ballooning
- Swapping
- Transparent Page Sharing
- Compression
- Scale Monitoring Messages
- "Unified Manager Server Low On Memory"
- "Unified Manager Server Low On Memory"
- Log indications:
- MySQL "
error.log
" file:2020-04-12T04:02:10.036849Z 205 [Note] Aborted connection XXX to db: 'unconnected' user: 'jboss' host: 'localhost' (Failed on my_net_write())
2020-04-12T08:50:42.808598Z 450 [Note] Got timeout reading communication packets
- Unified Manager "
server_mega.log
" file:2020-04-12 06:11:01,221 ERROR [default task-4] c.n.s.s.a.OriginatorUpdateServlet (OriginatorUpdateServlet.java:82) - Transaction rolled back: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
- Unified Manager "
ocumserver.log
" file:- 2019-06-06 02:49:49,566 INFO [oncommand] [pool-22-thread-4] [c.n.d.e.p.e.b.UnifiedManagerMemoryEventTypeController] Scale Monitor: New Memory Event Created: event id=128123,eventSeverity=low, thresholdValue=5.0, currentUsageValue=6.690000057220459, totalMemorySizeValue=1.365997568E10, jvm=AuJvm
2019-06-06 01:49:49,190 INFO [oncommand] [pool-22-thread-55] [c.n.d.e.p.e.b.UnifiedManagerMemoryEventTypeController] Scale Monitor: New Memory Event Created: event id=128094,eventSeverity=almostOut, thresholdValue=7.0, currentUsageValue=7.195000171661377, totalMemorySizeValue=1.365997568E10, jvm=AuJvm
- MySQL "
NOTE: This is not an exhaustive list of logs indicating memory issues. Please be aware that NetApp Support will find other log lines that can also indicate memory issues.