StorageGRID Cannot delete tenant
Applies to
- StorageGRID 11.6
- StorageGRID 11.7
Issue
- Buckets and/or tenant cannot be deleted even though the bucket is empty (has 0 objects/bytes when listing via an S3 client application).
- Looking at usage numbers in the GUI it shows the bucket as non-empty.
- Unable to update usage counters using the baseline script.
grep UTOT /varlocal/log/bycast.log
shows following warnings:
Jun 19 23:59:08 <node-name> ADE: |<node-id> 1104746074 UTOT CSEN 2023-06-19T23:59:08.414917| NOTICE 0496 8a14e03898a38b1 UTOT: Failed to query for empty buckets Jun 19 23:59:08 <node-name> ADE: |21229880 1104746074 UTOT CSEN 2023-06-19T23:59:08.414967| WARNING 0236 8a14e03898a38b1 UTOT: deletedBuckets Cassandra TimeoutException (Failed to execute cql at consistency EACH_QUORUM: SELECT DISTINCT bucket_id from storagegrid.s3_usage_total_by_bucket - Cassandra Driver Error(Read timeout):'Operation timed out - received only 0 responses.' Detailed Info:[consistency: ONE, responses_received: 0, responses_required: 1, data_present: 0]) Jun 19 23:59:09 <node-name> ADE: |<node-id> 1104746074 UTOT %WKE 2023-06-19T23:59:09.219640| WARNING 0029 8a14e03898a38b1 UTOT: Unhandled message from UTOT:{1104748137@<node-id>}: %WKE Jun 20 00:01:29 <node-name> ADE: |<node-id> 1104746074 UTOT CSEN 2023-06-20T00:01:29.674705| NOTICE 0471 8a14e03898a38b1 UTOT: Failed to query for empty accounts Jun 20 00:01:29 <node-name> ADE: |<node-id> 1104746074 UTOT CSEN 2023-06-20T00:01:29.674777| WARNING 0236 8a14e03898a38b1 UTOT: empty accounts Cassandra TimeoutException (Failed to execute cql at consistency EACH_QUORUM: SELECT DISTINCT account from storagegrid.s3_usage_total_by_account - Cassandra Driver Error(Read timeout):'Operation timed out - received only 0 responses.' Detailed Info:[consistency: ONE, responses_received: 0, responses_required: 1, data_present: 0])