Kafka notifications not working after initial configuration in StorageGRID
Applies to
- StorageGRID
- Kafka
Issue
- When a file is uploaded via S3, the following warning is displayed within the Platform services endpoints screen in StorageGRID Tenant Manager:
- The Connection tab for the affected endpoint displays an
Endpoint failure: UNKNOWN_SERVER_ERROR, even when the Test connection is successful:
- Kafka notification failures can be observed under Nodes > Site > Platform Services:
-
- Alternatively, the failures can be observed under Support > Metrics > Prometheus. Use the following expression:
storagegrid_platform_services_failed_kafka_notifications- Click Execute.
- Select the Graph tab.
- Alternatively, the failures can be observed under Support > Metrics > Prometheus. Use the following expression:
- The
bycast.logshows the following error:
Nov 10 21:23:12 STORAGENODE04 rsm[37388]: [kafka] [INFO] batch in a produce request failed; broker: 5300, topic: TOPICNAME, partition: 0, err: UNKNOWN_SERVER_ERROR: The server experienced an unexpected error when processing the request., err_is_retryable: false, max_retries_reached: false