Skip to main content
NetApp Knowledge Base

OLST may be reported after EC-rebalance, while object chunks are present on disk

Views:
136
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • StorageGRID 11.5
  • ONTAP
  • FabricPool

Issue

Object lost may be reported after EC rebalance, even though object chunks are present at the physical locations on disk.

Following documentation on Investigating lost objects:

# grep OLST audit.log
2022-02-17T11:00:40.208953 [AUDT:[CBID(UI64):0x8504E653A4550568][UUID(CSTR):"F4A19BD8-8912-4544-B423-26A33E10F7C0"][PATH(CSTR):"ec-bucket/6310eec2-8483-4498-b07a-3bedfce2fff7/0808d4d1_00000000025cfef7_6310eec2-8483-4498-b07a-3bedfce2fff7"][RSLT(FC32):NONE][AVER(UI32):10][ATIM(UI64):1645095640208953][ATYP(FC32):OLST][ANID(UI32):12597448][AMID(FC32):ILMX][ATID(UI64):15044298786699829299]]

# telnet localhost 1402
ade 12448208: / > /proc/OBRP/ObjectByUUID F4A19BD8-8912-4544-B423-26A33E10F7C0

{
    "TYPE": "CTNT",
    "CHND": "1FEC9E09-23CC-4E54-B9E3-099482A8C62A",
    "NAME": "6310eec2-8483-4498-b07a-3bedfce2fff7/0808d4d1_00000000025cfef7_6310eec2-8483-4498-b07a-3bedfce2fff7",
    "CBID": "0x80D54C8D8B62B9D0",
    "PHND": "3058F539-BCE5-11E9-9E53-650300B972EC",
    "PPTH": "ec-bucket",
    "META": {
        "BASE": {
            "ISIA": "10.64.203.173",
            "PHTP": "1",
            "PAWS": "2",
            "ACCT": "12969787153409237262",
            "BKAC": "12969787153409237262",
            "*ctp": "application/x-binary",
        },
        "AWS3": {
            "USDM": "{\"s3b-last-modified\":[\"20190808T092909Z\"],\"sha256\":[\"7f3ceaf54bcff0f63716900f6769b93597aae33e188e88792aa484a93d056dc9\"]}",
            "LOCC": "us-east-1"
        },
        "SEGM": {
            "SHSH": "MD5D 0x075B7CDE47E6140E4F4EC089FC2285A52D3400",
            "CSIZ": "29012864"
        },
        "BYCB": {
            "CTME": "1565603232952248",
            "SHSH": "MD5D 0x1F6225B9C340CDE85ADDBFC5CC818590",
            "CSIZ": "210",
            "BSIZ": "1168",
            "CVER": "196612",
            "CFLG": "256",
            "CTYP": "NONE",
            "CHSH": "SHA1 0x537C146DE8137717EA037AD9DF43779E2E6B63D8",
            "MTME": "1565603232944791",
            "ITME": "1565603232952248"
        },
        "CMSM": {
            "OWNR": "13461290",
            "LATM": "1565603232952248"
        }
    }
}

Note that "CLCO:" (Locations) section is missing.

On ONTAP side this may manifest as ASUP triggered for:
HA Group Notification (WAFL INCONSISTENT USER DATA BLOCK) ALERT

Or error similar to following in ONTAP EMS logs:
Thu Feb 17 12:00:39 +0100 [nodename: wafl_cbin_cloud_obj_create_pr: wafl.raid.incons.set.err.obj:alert]: Data block at VBN <VBN number> tiered to capacity tier (objid <object id>, slot <slot id>, btuuid <>, seqno <sequence number>) and marked with error (120). Block previously had error (120).

Thu Feb 17 12:08:48 +0100 [nodename: wafl_exempt08: wafl.raid.incons.userdata:error]: WAFL inconsistent: inconsistent user data block at VBN <VBN number> (vvbn:<vvbn number> fbn:<fbn> level:0) in public inode (fileid:<file id> snapid:0 file_type:1 disk_flags:0x8002 error:120 raid_set:1) in volume <volumename>@vserver:<vserver id>.

As ONTAP sees blocks gone missing from object-store, RAID marks these blocks as pseudo bad (error:120).

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.