Can not get the object with the Status Code: 500; Error Code: InternalError
Applies to
NetApp StorageGRID
Issue
- Can not get the object with the
Status Code: 500; Error Code: InternalError
from the Application side.
00:00:06.074 [ERROR] call:1284 - [Attr:H20220127224229][MetaObject ID:H20220127224229/2bD548e53e0077f4ba0a8b9abe7fd5e8950at5wiRgDu] Get MetaDataObject from Netapp failed: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: 1643299204299971)
- The affected object file can be found on the ILM Object Metadata Lookup of NetApp StorageGRID GUI.
- The
bycast.log
shows the logs like below on the affected node.
Jan 28 01:49:19 node14 ADE: |12835068 0876556914 S3RQ %DED 2022-01-28T01:49:19.066558| NOTICE 0137 S3RQ: S3 error response: RequestId=1643334559040622, Resource=/metaobj/H20220127224229/2bD548e53e0077f4ba0a8b9abe7fd5e8950at5wiRgDu, HTTP Status Code=500, ErrorMsg=InternalError, ErrorType=Client, CustomErrorMessage={We encountered an internal error. Please try again.}, Details={unexpected error (GERR). details: ,}
Jan 28 01:49:19 node14 ADE: |12835068 0876556913 HTSM %DED 2022-01-28T01:49:19.066654| NOTICE 0595 HTSM: Connection 1643334559040622/10.185.11.107 GET /metaobj/H20220127224229/2bD548e53e0077f4ba0a8b9abe7fd5e8950at5wiRgDu (child: -/-, --F--p-): Shutting down (from {876556913@12835068})
Jan 28 01:49:19 node14 ADE: |12835068 0876556913 HTSM %DED 2022-01-28T01:49:19.066675| NOTICE 0638 HTSM: Connection 1643334559040622/10.185.11.107 GET /metaobj/H20220127224229/2bD548e53e0077f4ba0a8b9abe7fd5e8950at5wiRgDu (child: -/-, --FS-p-): Delaying shutdown - still processing data from network
Jan 28 01:49:19 node14 ADE: |12835068 0876556913 HTSM ENCR 2022-01-28T01:49:19.067738| NOTICE 0706 HTSM: Connection 1643334559040622/10.185.11.107 GET /metaobj/H20220127224229/2bD548e53e0077f4ba0a8b9abe7fd5e8950at5wiRgDu (child: -/-, --FSC--): Network read error: end of stream
- The os\var\local\log\messages show the
talpa-deny
on the affected node.
Jan 27 14:54:01 node14 kernel: talpa-deny: Access denied while closing /var/local/rangedb/6/p/1B/09/00pxF{dm@qQ_Z0^%uFD2 on behalf of process ldr[263471/264697] owned by 1002(1002)/1000(1000) <0>