BlueXP Connector is down due to memory shortage

Applies to

Issue

• Email Alert :  BlueXP Connector is down is received
• The canvas is not showing any working environments or working environments are in a failed state.
• Error on CVOs in NetApp Console may read: "No accessKey/secretKey key pair defined" due to Connector failing.
• Update Containers task failures in the Timeline are present

ERROR: for 35ace918a580_cloudmanager_credentials  Cannot start service cloudmanager_credentials: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/memory/docker/26220f4b5e2248f31be1d5aa220c6fcc30621091b182dc6b532e0b438d38a817: cannot allocate memory\"": unknown

• OS log shows  stack trace for out of memory error repeatedly
  • Feb 27 11:56:08 ip-10-200-2-242 kernel: Out of memory: Kill process 20261 (java) score 106 or sacrifice child
Feb 27 11:56:08 ip-10-200-2-242 kernel: Killed process 20261 (java) total-vm:5331016kB, anon-rss:1715480kB, file-rss:0kB, shmem-rss:0kB
Feb 27 11:56:08 ip-10-200-2-242 kernel: service-manager invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Feb 27 11:56:08 ip-10-200-2-242 kernel: service-manager cpuset=/ mems_allowed=0
Feb 27 11:56:08 ip-10-200-2-242 kernel: CPU: 0 PID: 16735 Comm: service-manager Kdump: loaded Tainted: G           O   ------------ T 3.10.0-957.21.3.el7.x86_64 #1
Feb 27 11:56:08 ip-10-200-2-242 kernel: Hardware name: Amazon EC2 t3.xlarge/, BIOS 1.0 10/16/2017
Feb 27 11:56:08 ip-10-200-2-242 kernel: Call Trace:
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f63107>] dump_stack+0x19/0x1b
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f5db2a>] dump_header+0x90/0x229
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4901292>] ? ktime_get_ts64+0x52/0xf0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49584df>] ? delayacct_end+0x8f/0xb0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49ba834>] oom_kill_process+0x254/0x3d0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49ba2dd>] ? oom_unkillable_task+0xcd/0x120
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49ba386>] ? find_lock_task_mm+0x56/0xc0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49bb076>] out_of_memory+0x4b6/0x4f0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f5e62e>] __alloc_pages_slowpath+0x5d6/0x724
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49c1454>] __alloc_pages_nodemask+0x404/0x420
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4a0e308>] alloc_pages_current+0x98/0x110
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49b6697>] __page_cache_alloc+0x97/0xb0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49b92f8>] filemap_fault+0x298/0x490
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffc0284a9e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffc0284c9c>] xfs_filemap_fault+0x2c/0x30 [xfs]
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49e47ea>] __do_fault.isra.59+0x8a/0x100
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49e4d9c>] do_read_fault.isra.61+0x4c/0x1b0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49e9744>] handle_pte_fault+0x2f4/0xd10
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa49ec27d>] handle_mm_fault+0x39d/0x9b0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f70603>] __do_page_fault+0x203/0x4f0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f709d6>] trace_do_page_fault+0x56/0x150
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f6ff62>] do_async_page_fault+0x22/0xf0
Feb 27 11:56:08 ip-10-200-2-242 kernel: [<ffffffffa4f6c798>] async_page_fault+0x28/0x30
Feb 27 11:56:08 ip-10-200-2-242 kernel: Mem-Info:

• Docker OCCM Container restarts constantly, with UI errors:
  • Connection refused to: occm/IP
• Assigned memory is 16GB or less