Skip to main content
NetApp Knowledge Base

StorageGRID decommission process stalled at Erasure coded data

  1. Last updated
  2. Save as PDF
Views:
224
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

NetApp StorageGRID 11.7 and below
EC (Erasure Coding)

Issue

Storage node decommission is not progressing, showing following alert:

"Unable to communicate with node
One or more services are unresponsive, or the node cannot be reached"


Cassandra system logs show the following error message:

Lumberjack log /Node/Cassandra/system.log

ERROR [MutationStage-3] 2021-05-22 05:58:18,334 StorageProxy.java (line 1473) Failed to apply mutation locally : java.lang.IllegalArgumentException/Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
WARN [MutationStage-5] 2021-05-22 05:58:20,099 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-5,5,main]: Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
WARN [MutationStage-1] 2021-05-22 05:58:21,581 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-1,5,main]: Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
WARN [MutationStage-1] 2021-05-22 05:58:22,878 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-1,5,main]: Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
WARN [MutationStage-3] 2021-05-22 05:58:24,215 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-3,5,main]: Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
WARN [MutationStage-3] 2021-05-22 05:58:25,531 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-3,5,main]: Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
ERROR [MutationStage-2] 2021-05-22 05:58:26,467 StorageProxy.java (line 1473) Failed to apply mutation locally : java.lang.IllegalArgumentException/Mutation of 28.612MiB is too large for the maximum size of 16.000MiB
 
Or
 
WARN [MutationStage-10] 2022-07-02 12:08:21,723 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-10,5,main]: Mutation (keyspace storagegrid, tables [b3f7ac20-fa33-11e9-bdaa-6313fe8d7848], partition key DecoratedKey(4090565182755706843, 333035393332363732)) with size 16.000MiB is too large for the maximum size of 16.000MiB
java.lang.IllegalArgumentException: Mutation (keyspace storagegrid, tables [b3f7ac20-fa33-11e9-bdaa-6313fe8d7848], partition key DecoratedKey(4090565182755706843, 333035393332363732)) with size 16.000MiB is too large for the maximum size of 16.000MiB
   
WARN [MutationStage-3] 2022-07-02 12:08:22,358 AbstractLocalAwareExecutorService.java (line 167) Uncaught exception on thread Thread[MutationStage-3,5,main]: Mutation (keyspace storagegrid, tables [b3f7ac20-fa33-11e9-bdaa-6313fe8d7848], partition key DecoratedKey(4090565182755706843, 333035393332363732)) with size 16.000MiB is too large for the maximum size of 16.000MiB
java.lang.IllegalArgumentException: Mutation (keyspace storagegrid, tables [b3f7ac20-fa33-11e9-bdaa-6313fe8d7848], partition key DecoratedKey(4090565182755706843, 333035393332363732)) with size 16.000MiB is too large for the maximum size of 16.000MiB
    

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.