AltaVault service crashing when connecting to GCP
Applies to
- Altavault
- Google Cloud Platform (GCP)
Issue
- Service is down and restart fails
- CLI error:
(config) # show replication bucket
listing entries from bucket esmav2-20221116
tcmalloc: large alloc 1073741824 bytes == 0x42a72000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 2147483648 bytes == 0x82b54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 4294967296 bytes == 0x102b54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 8589934592 bytes == 0x203354000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 17179869184 bytes == 0x403b54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 34359738368 bytes == 0x804b54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 68719476736 bytes == 0x1006b54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 137438953472 bytes == 0x200ab54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 274877906944 bytes == 0x4012b54000 @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
tcmalloc: large alloc 549755813888 bytes == (nil) @ 0x7effa05547ea 0x7eff9cf7ce99 0x7eff9cf7db0b 0x7eff9cf7dbb0 0x878bfe 0x87b7a5 0x87c243 0x87c3e7 0x87c570 0x87cb56 0x87d197 0x87efa0 0x880e64 0x723f38 0x7239b2 0x575e8e 0x542a77 0x7eff9c61db45 0x573404 (nil)
terminate called after throwing an instance of 'std::bad_alloc'
- System Log Error:
Mar 29 04:23:35 localhost kernel:[41625.205076] Out of memory: Kill process 18070 (rfsd) score 552 or sacrifice child
Mar 29 04:23:35 localhost kernel:[41625.205088] Killed process 18070 (rfsd) total-vm:268800448kB, anon-rss:255060912kB, file-rss:8kB, shmem-rss:65264kBMar 29 04:23:35 localhost kernel:[41625.205173] oom_reaper: reaped process 18070 (rfsd), now anon-rss:255060924kB, file-rss:0kB, shmem-rss:65336kBNov 17 11:11:14 esmav2 check_cloud_connection: Error listing buckets!
Nov 17 11:11:14 esmav2 mgmtd[5426]: [mgmtd.WARNING]: Exit with code 1 from cloudctl
Nov 17 11:11:14 esmav2 mgmtd[5426]: [mgmtd.NOTICE]: Cloud connection check failed: tcmalloc: large alloc 1073741824 bytes
Nov 17 11:11:14 esmav2 mgmtd[5426]: [mgmtd.WARNING]: Error connecting to the cloud: tcmalloc: large alloc 1073741824 bytes
Nov 17 11:11:14 esmav2 mgmtd[5426]: [mgmtd.INFO]: Action '/rbt/cb/main/action/check_cloud_connection' initiated by user admin completed (27/8966)
Nov 17 11:11:14 esmav2 webasd[9193]: [web.INFO]: web: Received return code 1, return message "Error connecting to the cloud: tcmalloc: large alloc 1073741824 bytes