Due to heap memory exhaustion in the SWS agent, agent008 errors occur across all data collectors

Applies to

• Data Infrastructure Insights (DII , Formerly Cloud Insights)
• Storage Workload Security (SWS)

Issue

• All data collectors report Agent008 error.
• cloudsecure-agent-service.log shows an out-of-memory error in heap memory.

 Uncaught error from thread [CloudSecureAgent-akka.actor.default-dispatcher-11]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[CloudSecureAgent]
 java.lang.OutOfMemoryError: Java heap space
 Uncaught error from thread [CloudSecureAgent-akka.actor.default-dispatcher-2]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[CloudSecureAgent]
 java.lang.OutOfMemoryError: Java heap space
 Exception in thread "CloudSecureAgent-shutdown-hook-1" java.lang.InternalError: java.lang.OutOfMemoryError: Java heap space
         at java.base/java.lang.invoke.InnerClassLambdaMetafactory.generateInnerClass(Unknown Source)
         at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(Unknown Source)
         at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(Unknown Source)
         at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(Unknown Source)
         at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(Unknown Source)
         at java.base/java.lang.invoke.CallSite.makeSite(Unknown Source)
         at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(Unknown Source)
         at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(Unknown Source)
         at com.netapp.datafabric.metadatacatalog.agent.app.AppManager.$anonfun$preStart$1(AppManager.scala:101)
         at akka.actor.CoordinatedShutdown.$anonfun$run$3(CoordinatedShutdown.scala:451)
         at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
         at scala.collection.Iterator.foreach(Iterator.scala:943)
         at scala.collection.Iterator.foreach$(Iterator.scala:943)
         at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
         at scala.collection.IterableLike.foreach(IterableLike.scala:74)
         at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
         at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
         at scala.collection.TraversableLike.map(TraversableLike.scala:286)
         at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
         at scala.collection.AbstractTraversable.map(Traversable.scala:108)
         at akka.actor.CoordinatedShutdown.loop$1(CoordinatedShutdown.scala:448)
         at akka.actor.CoordinatedShutdown.run(CoordinatedShutdown.scala:502)
         at akka.actor.CoordinatedShutdown.run(CoordinatedShutdown.scala:400)
         at akka.actor.CoordinatedShutdown$.$anonfun$initJvmHook$1(CoordinatedShutdown.scala:238)
         at akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:575)
 Caused by: java.lang.OutOfMemoryError: Java heap space

• agent.log shows dead error encountered error.

Message [com.netapp.datafabric.metadatacatalog.agent.common.log.LogEvent] from Actor[akka://CloudSecureAgent/user/appManager/agentLogSocketServerActor/$a#-XXXXXXXXX] to Actor[akka://CloudSecureAgent/system/StreamSupervisor-5/flow-0-1-actorRefSource#-XXXXXXXX] was not delivered. [XXXXXXXX] dead letters encountered. If this is not an expected behavior, then [Actor[akka://CloudSecureAgent/system/StreamSupervisor-5/flow-0-1-actorRefSource#-XXXXXXXX]] may have terminated unexpectedly,