SP/BMC fails to update firmware due to a faulty SP/BMC port

Applies to

• ONTAP 9
• Service Processor
  • Service Processor (SP)
  • Baseboard Management Controller (BMC)
• Firmware upgrade

Issue

• The SP/BMC firmware upgrade fails and the following errors are reported in event logs:

[cluster1-01: servprocd: sp.servprocd.upd.evts:debug]: params: {'reason': 'SP Firmware network update from 10.5 to 10.6 has been triggered.'}
[cluster1-01: servprocd: sp.servprocd.upd.evts:debug]: params: {'reason': 'SP firmware image has been successfully transferred to SP using network interface.'}
[cluster1-01: servprocd: sp.servprocd.upd.unexpt.evts:debug]: params: {'reason': 'Unable to successfully update SP firmware using network interface'}
[cluster1-01: servprocd: sp.servprocd.upd.error:error]: SP update error: SP firmware update failure has been detected.

• The SP-MGMT-MLOG-TXT.GZ from AutoSupport reports that the SP/BMC is not reachable:

[kern_servprocd:info:7198] 0x808707100: 0: NOTICE: Servprocd::FTM: transferred /mroot/etc/RLM_FW/sp_image.tar.gz to sp://cluster1-01:/mnt/sapps/sp_image.tar.gz
[kern_servprocd:info:7198] 0x808707100: 0: NOTICE: Servprocd::SpUpdate: update: SP firmware image has been successfully transferred to SP using network interface.
[kern_servprocd:info:5642] 0x808702b00: 0: ERR: Servprocd::SpUpdate: writeImage: spcs sp update command failed (The Service Processor on node "cluster1-01" is not reachable. Verify that the SP or BMC is online, verify that api-service is enabled on the SP or BMC, verify that the partner node is running, check if pings from SP or BMC to partner node work, check if hw-assist keep-alives are normal, check that network ports are configured correctly and are functional (up). Then, try the command again.)
[kern_servprocd:info:5642] 0x808702b00: 0: ERR: Servprocd::SpUpdate: update: unable to successfully update SP firmware using network interface

[kern_servprocd:info:6143] 0x80b715000: 0: NOTICE: Servprocd::SpUpdate: ScheduleSpAutoUpdate: Checking whether SP network is available for SP firmware auto-update
[kern_servprocd:info:6143] 0x80b715000: 0: ERR: Servprocd::SpUpdate: doPreUpdateChecks: Ping test to internal BMC IP failed.

[kern_servprocd:info:6143] 0x809426f00: 8003ea000000080d: ERR: Servprocd::CLI: create_imp: The specified package /mroot/etc/software/http://10.16.11.5/web/netapp/BMC_FW.zip could not be read.
[kern_servprocd:info:6143] 0x80946d400: 8503ea0000000445: ERR: Servprocd::CLI: get_bmc_boot_image: Failed to get boot image for BMC

• SP-LATEST-RUNTIME from AutoSupport reports MemFree is above 57,000 KB

MemFree:           96968 kB

• SP-LATEST-CONFIGURATION from AutoSupport shows N/A primary and backup firmware versions:

version
=======
Booted primary firmware version 11.6
Primary firmware version N/A
Backup firmware version N/A

• SP-LATEST-SYSLOG from AutoSupport reports errors similar to the following during manual update attempts:

BMC fud_eth[29319]: get_notif_shm: Failed to create the shared memory key
BMC fud_eth[29319]: ipmi_notif_to_ontap: Unable to set progress state as shared memory is not accessible.
BMC fud_eth[29319]: Downloading package...
BMC fud_eth[29319]: system() returned ret1 = 0, ret2 = 0
BMC last message repeated 3 times
BMC fud_eth[29319]:  FW package download from ftp://anonymous@10.10.10.10/software//BMC_FW_308-04071_11.7.tar.gz failed.
BMC fud_eth[29319]: Cleaning up..

• The SP/BMC is online and the service-processor api-service is enabled.
• The SP api-service internal certificates are renewed.
• The network ports are configured correctly and are functional (up).
• Rebooting the SP/BMC and restarting the service-processor daemon does not help.
• An attempt to upgrade directly from the SP/BMC prompt also fails.
• Performing takeover/giveback of the affected node does not fix the issue.