SP/BMC fails to update firmware due to a faulty SP/BMC port

Applies to

• FAS/AFF
• ONTAP 9
• Service Processor (SP)
• Baseboard Management Controller (BMC)

Issue

• The SP/BMC firmware upgrade fails and the following errors are reported in event logs:

[cluster1-01: servprocd: sp.servprocd.upd.evts:debug]: params: {'reason': 'SP Firmware network update from 10.5 to 10.6 has been triggered.'}
[cluster1-01: servprocd: sp.servprocd.upd.evts:debug]: params: {'reason': 'SP firmware image has been successfully transferred to SP using network interface.'}
[cluster1-01: servprocd: sp.servprocd.upd.unexpt.evts:debug]: params: {'reason': 'Unable to successfully update SP firmware using network interface'}
[cluster1-01: servprocd: sp.servprocd.upd.error:error]: SP update error: SP firmware update failure has been detected.

• The SP-MGMT-MLOG-TXT.GZ from AutoSupport reports that the SP/BMC is not reachable:

[kern_servprocd:info:7198] 0x808707100: 0: NOTICE: Servprocd::FTM: transferred /mroot/etc/RLM_FW/sp_image.tar.gz to sp://cluster1-01:/mnt/sapps/sp_image.tar.gz
[kern_servprocd:info:7198] 0x808707100: 0: NOTICE: Servprocd::SpUpdate: update: SP firmware image has been successfully transferred to SP using network interface.
[kern_servprocd:info:5642] 0x808702b00: 0: ERR: Servprocd::SpUpdate: writeImage: spcs sp update command failed (The Service Processor on node "cluster1-01" is not reachable. Verify that the SP or BMC is online, verify that api-service is enabled on the SP or BMC, verify that the partner node is running, check if pings from SP or BMC to partner node work, check if hw-assist keep-alives are normal, check that network ports are configured correctly and are functional (up). Then, try the command again.)
[kern_servprocd:info:5642] 0x808702b00: 0: ERR: Servprocd::SpUpdate: update: unable to successfully update SP firmware using network interface

• SP-LATEST-RUNTIME from AutoSupport reports MemFree is above 57,000 KB

MemFree:           96968 kB

• SP-LATEST-CONFIGURATION from AutoSupport shows N/A primary and backup firmware versions:

version
=======
Booted primary firmware version 11.6
Primary firmware version N/A
Backup firmware version N/A

• SP-LATEST-SYSLOG from AutoSupport reports errors similar to the following during manual update attempts:

BMC fud_eth[29319]: get_notif_shm: Failed to create the shared memory key
BMC fud_eth[29319]: ipmi_notif_to_ontap: Unable to set progress state as shared memory is not accessible.
BMC fud_eth[29319]: Downloading package...
BMC fud_eth[29319]: system() returned ret1 = 0, ret2 = 0
BMC last message repeated 3 times
BMC fud_eth[29319]:  FW package download from ftp://anonymous@10.10.10.10/software//BMC_FW_308-04071_11.7.tar.gz failed.
BMC fud_eth[29319]: Cleaning up..

• The SP/BMC is online and the service-processor api-service is enabled.
• The SP api-service internal certificates are renewed.
• The network ports are configured correctly and are functional (up).
• Rebooting the SP/BMC and restarting the service-processor daemon does not help.
• An attempt to upgrade SP/BMC from the SP/BMC prompt also fails.
• Performing takeover/giveback of the affected node does not fix the issue.