How to resolve memory issues with Active IQ Unified Manager

Applies to

• Active IQ Unified Manager (AIQUM)
• OnCommand Unified Manager (OCUM)

Answer

1. Determine the amount of memory allocated and reserved

2. Determine the amount of reserved memory

   • ​​​​​​​Utilizing reserved memory should be full or none, as fractional reserves can cause issues

3. Check AIQUM memory recommendations

4. Review best practice recommendations for sizing

   1. See page 10 - "Unified Manager scalability"

5. Make memory adjustments as needed

   1. VMware Docs - Change the Memory Configuration
   2. VMware Docs - Allocate Memory Resources to a Virtual Machine in the VMware Host Client

6. Confirm that no ballooning or swapping is occurring on the virtual machine.

   1. VMware Docs - Memory (Data Counters)
   2. VMware Docs - Memory (MB)

7. Check for memory starved Unified Manager components 

8. Linux only - Linux may stop Unified Manager processes to free memory

   1. On Linux platforms, check for OOM "Out of Memory" killer ending processes to free up memory. 
   2. In rare instances Red Hat and CentOS platforms kill Unified Manager services / processes to free up memory.
      1. How to troubleshoot Out of memory (OOM) killer in Red Hat Enterprise Linux?

Dec 8 22:00:18 um-hostname kernel: Out of memory: Kill process 45678 (java) score 228 or sacrifice child
Dec 8 22:00:18 um-hostname kernel: Killed process 45988 (sh) total-vm:122868kB, anon-rss:260kB, file-rss:0kB, shmem-rss:0kB
Dec 8 22:00:19 um-hostname systemd: mysqld.service: main process exited, code=killed, status=9/KILL
Dec 8 22:00:19 um-hostname systemd: Unit mysqld.service entered failed state.
Dec 8 22:00:19 um-hostname systemd: mysqld.service failed.
Dec 8 22:00:20 um-hostname systemd: mysqld.service holdoff time over, scheduling restart.
Dec 8 22:00:20 um-hostname systemd: Stopped MySQL Server.
Dec 8 22:00:20 um-hostname systemd: Starting MySQL Server...

Additional Information

Common memory issues include

• Unified Manager application is running slow:
  • Services take a long time to come online.
  • WebGUI is slow.
  • Backups take a very long time ( and possibly fail ).
• Unified Manager application CPU and/or memory utilization are extremely high; swap / page file is in use.
• Unified Manager application experiences inconsistent behavior:
  • Services hanging.
  • Services restarting.
  • Services not starting at all (especially after upgrade)
  • Occasional issues with alert notifications.
  • Inventory / performance acquisition polls may not complete within their normal 15 minute / 5 minute cycles.
  • Other abnormal (unexplainable) inconsistent behavior when running in a virtual environment.
• VMware ESXi displays one or more of the following:
  • Memory Ballooning
  • Swapping
  • Transparent Page Sharing
  • Compression
• Scale Monitoring Messages
  • "Unified Manager Server Low On Memory"
    • 
• Log indications:
  • MySQL "error.log" file:
    • 2020-04-12T04:02:10.036849Z 205 [Note] Aborted connection XXX to db: 'unconnected' user: 'jboss' host: 'localhost' (Failed on my_net_write())
    • 2020-04-12T08:50:42.808598Z 450 [Note] Got timeout reading communication packets
  • Unified Manager "server_mega.log" file:
    • 2020-04-12 06:11:01,221 ERROR [default task-4] c.n.s.s.a.OriginatorUpdateServlet (OriginatorUpdateServlet.java:82) - Transaction rolled back: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
  • Unified Manager "ocumserver.log" file: 
    • 2019-06-06 02:49:49,566 INFO [oncommand] [pool-22-thread-4] [c.n.d.e.p.e.b.UnifiedManagerMemoryEventTypeController] Scale Monitor: New Memory Event Created: event id=128123,eventSeverity=low, thresholdValue=5.0, currentUsageValue=6.690000057220459, totalMemorySizeValue=1.365997568E10, jvm=AuJvm
    • 2019-06-06 01:49:49,190 INFO [oncommand] [pool-22-thread-55] [c.n.d.e.p.e.b.UnifiedManagerMemoryEventTypeController] Scale Monitor: New Memory Event Created: event id=128094,eventSeverity=almostOut, thresholdValue=7.0, currentUsageValue=7.195000171661377, totalMemorySizeValue=1.365997568E10, jvm=AuJvm

NOTE: This is not an exhaustive list of logs indicating memory issues. Please be aware that NetApp Support will find other log lines that can also indicate memory issues.