Skip to main content
NetApp Knowledge Base

OLST may be reported after EC-rebalance, while object chunks are present on disk

Views:
73
Visibility:
Public
Votes:
0
Category:
storagegrid
Specialty:
sgrid
Last Updated:

Applies to

  • StorageGRID 11.5
  • ONTAP
  • FabricPool

Issue

Object lost may be reported after EC rebalance, even though object chunks are present at the physical locations on disk.

Following documentation on Investigating lost objects:

# grep OLST audit.log
2022-02-17T11:00:40.208953 [AUDT:[CBID(UI64):0x8504E653A4550568][UUID(CSTR):"F4A19BD8-8912-4544-B423-26A33E10F7C0"][PATH(CSTR):"ec-bucket/6310eec2-8483-4498-b07a-3bedfce2fff7/0808d4d1_00000000025cfef7_6310eec2-8483-4498-b07a-3bedfce2fff7"][RSLT(FC32):NONE][AVER(UI32):10][ATIM(UI64):1645095640208953][ATYP(FC32):OLST][ANID(UI32):12597448][AMID(FC32):ILMX][ATID(UI64):15044298786699829299]]

# telnet localhost 1402
ade 12448208: / > /proc/OBRP/ObjectByUUID F4A19BD8-8912-4544-B423-26A33E10F7C0

{
    "TYPE": "CTNT",
    "CHND": "1FEC9E09-23CC-4E54-B9E3-099482A8C62A",
    "NAME": "6310eec2-8483-4498-b07a-3bedfce2fff7/0808d4d1_00000000025cfef7_6310eec2-8483-4498-b07a-3bedfce2fff7",
    "CBID": "0x80D54C8D8B62B9D0",
    "PHND": "3058F539-BCE5-11E9-9E53-650300B972EC",
    "PPTH": "ec-bucket",
    "META": {
        "BASE": {
            "ISIA": "10.64.203.173",
            "PHTP": "1",
            "PAWS": "2",
            "ACCT": "12969787153409237262",
            "BKAC": "12969787153409237262",
            "*ctp": "application/x-binary",
        },
        "AWS3": {
            "USDM": "{\"s3b-last-modified\":[\"20190808T092909Z\"],\"sha256\":[\"7f3ceaf54bcff0f63716900f6769b93597aae33e188e88792aa484a93d056dc9\"]}",
            "LOCC": "us-east-1"
        },
        "SEGM": {
            "SHSH": "MD5D 0x075B7CDE47E6140E4F4EC089FC2285A52D3400",
            "CSIZ": "29012864"
        },
        "BYCB": {
            "CTME": "1565603232952248",
            "SHSH": "MD5D 0x1F6225B9C340CDE85ADDBFC5CC818590",
            "CSIZ": "210",
            "BSIZ": "1168",
            "CVER": "196612",
            "CFLG": "256",
            "CTYP": "NONE",
            "CHSH": "SHA1 0x537C146DE8137717EA037AD9DF43779E2E6B63D8",
            "MTME": "1565603232944791",
            "ITME": "1565603232952248"
        },
        "CMSM": {
            "OWNR": "13461290",
            "LATM": "1565603232952248"
        }
    }
}

Note that "CLCO:" (Locations) section is missing.

On ONTAP side this may manifest as ASUP triggered for:
HA Group Notification (WAFL INCONSISTENT USER DATA BLOCK) ALERT

Or error similar to following in ONTAP EMS logs:
Thu Feb 17 12:00:39 +0100 [nodename: wafl_cbin_cloud_obj_create_pr: wafl.raid.incons.set.err.obj:alert]: Data block at VBN <VBN number> tiered to capacity tier (objid <object id>, slot <slot id>, btuuid <>, seqno <sequence number>) and marked with error (120). Block previously had error (120).

Thu Feb 17 12:08:48 +0100 [nodename: wafl_exempt08: wafl.raid.incons.userdata:error]: WAFL inconsistent: inconsistent user data block at VBN <VBN number> (vvbn:<vvbn number> fbn:<fbn> level:0) in public inode (fileid:<file id> snapid:0 file_type:1 disk_flags:0x8002 error:120 raid_set:1) in volume <volumename>@vserver:<vserver id>.

As ONTAP sees blocks gone missing from object-store, RAID marks these blocks as pseudo bad (error:120).

Scan to view the article on your device
CUSTOMER EXCLUSIVE CONTENT

Registered NetApp customers get unlimited access to our dynamic Knowledge Base.

New authoritative content is published and updated each day by our team of experts.

Current Customer or Partner?

Sign In for unlimited access

New to NetApp?

Learn more about our award-winning Support