The old system ID is displayed because the controller is replaced during partial giveback
Applies to
- FAS2750
- ONTAP 9
Issue
- The
e0b
on node2 go down, so replacing the network cable, but cannot link up.
Example of EMS snip:
<LR d="13Oct2020 10:29:18" n="node1" t="1602552558" id="1597219663/56726"
p="6" s="Ok" o="kernel" vf="" type="0" seq="732963" >
<netif_linkDown_1
lanType="Ethernet"
ifName="e0b"
status="down, check cable"/>
</LR>
<LR d="13Oct2020 10:29:21" n="node1" t="1602552561" id="0/27191437951018"
p="5" s="Ok" o="vifmgr" vf="" type="1" seq="732964" >
<vifmgr_portdown_1
nodename="node1"
portname="e0b"/>
</LR>
<LR d="13Oct2020 10:29:22" n="node1" t="1602552562" id="0/27191437951019"
p="5" s="Ok" o="vifmgr" vf="" type="1" seq="732965" supp="1" >
<vifmgr_lifmoved_linkdown_1
vifname="vif2"
vsrvid="4294
967293"
vifip="XXX.XXX.XX.XXX"
tonode="node1"
toport="e0a"/>
</LR>
- Therefore, replacing the controller of node2, both e0a/e0b went down.
- The node2 controller is replaced, but the new controller is DOA, so return original controller after checking "
storage failover show
" status.
Example of sysconfig -a
snip:
slot 0: 10 Gigabit Ethernet Controller IX5-SFP+
e0a MAC Address: XX:XX:XX:XX:XX:XX (auto-unknown-fd-down)
e0b MAC Address: XX:XX:XX:XX:XX:XX (auto-unknown-fd-down)
Device Type: X550EM
Firmware Version: 0.86
down)
Device Type: X550EM
Firmware Version: 0.86-7.1
cluster1::> storage failover show
Takeover
Node Partner Possible State Description
-------------- -------------- -------- ------------------------------------
-
node1 node2 true System ID changed on partner (Old:
XXXXX
5509, New: 538110616),
Connected to node2, Partial giveback
node2 node1 - Waiting for cluster applications to
come online on the local node
Offline applications: mgmt, vldb,
vifmgr, bcomd, crs.
2 entries were displayed.
- Replacing the controller on node2 again, the e0a/e0b link are restored, but the remain old system ID(CCCCCCCCC) before the replacement.
- So "
disk reassign
" from the maintenance mode of node2, but couldn't change some of disks.
Example of console log snip after disk reassign:
*> disk reassign -s CCCCCCCCC -d AAAAAAAAA -p BBBBBBBBB
*> disk show -v
Local System ID: AAAAAAAAA
DISK OWNER POOL SERIAL NUMBER HOME DR HOME CHKSUM
Oct 25 09:35:48 [node2:diskown.duplicateSoftEntries:notice]: Found duplicate filer_table entries with same name node2 but different system ids ( CCCCCCCCC, AAAAAAAAA )
------------ ------------- ----- ------------- ------------- ------------- --------
0b.00.21 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.21P1 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.21P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.8 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.8P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.8P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.7 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.7P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.7P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.1P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.1P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.23 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.23P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.23P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.15 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.15P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.15P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.4 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.4P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.4P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.14 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.14P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.14P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.12 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.12P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.12P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.18 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.18P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.18P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.16 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.16P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.16P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.3 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.3P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.3P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.20 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.20P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.20P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.10 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.10P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.10P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.0 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.0P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.0P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.5 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.5P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.5P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.13 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.13P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.13P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.19 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.19P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.19P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.17 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.17P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.17P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.11 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.11P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.11P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.6 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.6P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.6P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.2P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.2P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.22 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.22P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.00.22P2 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXXXXXXXX node2 (AAAAAAAAA) Block
0b.00.9 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.9P1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.00.9P2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.10.11 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.9 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.19 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.23 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.17 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.5 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.22 node2 (AAAAAAAAA) Pool0 XXXXXXXX node2 (AAAAAAAAA) Block
0a.10.7 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.15 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.1 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.3 node1 (BBBBBBBBB) Pool0 XXXXXXXX X node1 (BBBBBBBBB) Block
0a.10.21 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0a.10.13 node1 (BBBBBBBBB) Pool0 XXXXXXXX node1 (BBBBBBBBB) Block
0b.12.5 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.12.10 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXX node2 (AAAAAAAAA) Block
0a.11.10 node2 (AAAAAAAAA) Pool0 XXXXXXXXXXXX node2 (AAAAAAAAA) Block
0a.11.9 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.12.7 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.12.11 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.12.9 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.10.18 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.0 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.4 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.2 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.10 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.12 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.16 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.8 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.14 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.6 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.10.20 node1 (BBBBBBBBB) Pool0 XXXXXXXX node2 (CCCCCCCCC) Block
0a.11.5 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.12.3 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.11.3 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.11.11 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.11.7 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.11.1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0b.12.1 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node1 (BBBBBBBBB) Block
0a.11.8 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0a.11.4 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0a.11.2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.12.2 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.12.6 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.12.4 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.12.8 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0a.11.6 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0a.11.0 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
0b.12.0 node1 (BBBBBBBBB) Pool0 XXXXXXXXXXXX node2 (CCCCCCCCC) Block
- After booting the system, the partial giveback is cleared, but another displayed.
Example:
cluster1::> storage failover show
Takeover
Node Partner Possible State Description
-------------- -------------- -------- -------------------------------------
node1 node2 true Connected to node2. Node owns aggregates belonging to another node
in the cluster.
node2 node1 true Connected to node1
2 entries were displayed.