OnCommand Insight 7.x
OnCommand Insight uses EMC Solutions Enabler to discover EMC Symmetrix arrays. OCI makes a best effort to bubble up to the end user the actual error message EMC Solutions Enabler reports. This article seeks to help you understand what causes commonly seen error messages.
- Failed to execute Sym CLI symcfg, because A requested network service was not found in the service file
Most OCI Solutions Enabler (SE) datasources are intended to operate in client/server mode - meaning, the OCI acquisition unit (AU) has SE installed, and that SE install is talking to another SE server that has connectivity to one or more Symmetrix arrays. Client/server mode requires a netcnfg alias to be defined within the OCI AU's netcnfg file. If your OCI SE datasource is set to work in client/server mode (defined as setting Connection Caching value to REMOTE or REMOTE_CACHED), and the Service Name value is NOT a valid netcnfg alias, EMC SE will fail because the datasource told SE to talk to an alias that does not exist.
- Failed to execute Sym CLI symcfg, because The remote client/server handshake failed. Please consult symapi and storsrvd log files
This is a TLS/SSL negotiation failure of Solutions Enabler operating in client/server mode. All versions of SE since 7.0 default to listening for SECURE connections only, i.e, encrypted. All Solutions Enabler installations generate digital certificates as part of install, or, your environment may replace those with digital certificates signed by an internal Certificate Authority (CA). SE performs mutual certificate authentication, where both the client SE install and server SE install present to each other their certificates and assess their validity.
99% of the time "handshake" failures are seen, are due to bad name resolution configurations - digital certificates have a hostname in them. If the reverse DNS each SE host performs on its peer does NOT resolve to the hostname seen in the peer's cert, handshake failures are likely to occur.
Please realize you need to look at BOTH client and server SE installs to look for these failures - you may have broken name resolution in both directions.
1% of the time these failures occur due to certificate expiration, or bad system clocks - if you install SE on a server set to US EST, and change it to PST, the start of the certificate's validity may be up to 3 hours in the future due to your changing of the system time.
- Failed to execute Sym CLI symcfg, because The remote connection is refused. The client cannot have a newer SYMAPI version than the server
This is another client/server failure. The client SE install needs to be the same, or older version than the SE server - this is an EMC support policy. This means if you have multiple SE datasources on the same OCI AU, you must take care to contemplate your SE upgrade strategies because you cannot install multiple SE versions on an OCI AU.
- Failed to execute Sym CLI symcfg, because The server refused a connection due to security level mismatch
When you define a netcnfg alias as part of your preparation for deploying OCI SE datasources in client/server mode, you can set the security level. As discussed earlier, SE servers default to only wanting to speak across encrypted channels, SECURE. If you define a netcnfg alias to NONSECURE, telling the client install to only attempt NONSECURE communication, it is very likely you will fail to communicate and OCI will report this EMC mismatch error.
- Failed to execute Sym CLI symcfg, because Symmetrix access control denied the request
EMC has multiple, overlapping security technologies that can impact EMC Solutions Enabler usage. "symacl" is a technology that can be enabled on an array by array basis. This error message is one of the ways symacl can block OCI from being able to discover a Symmetrix. It is important to understand that with symacl, you may need to grant permissions to the user account that OCI acquisition is occurring as.
- Failed to execute Sym CLI symcfg, because The gatekeeper device (while using the Base Daemon) has an error (Please see the Log file)
This is the point at which you should realize that having NetApp on the phone, and not EMC, is the wrong approach. One of the possibilities for this message is due to array decommissioning - if you are seeing this error for a Symmetrix serial number that has been decommissioned, you can try going to your Solutions Enabler server, perform a "symcfg discover" to force a rediscovery. That may purge that server's knowledge of the now decommissioned array.
- Failed to execute Sym CLI symcfg, because The remote connection is refused. The client SYMAPI version is not supported by the server
EMC has gotten more aggressive policing what configurations they allow to work. This message is typically seen when the SE server is 8.3 or higher, and the SE client is much older, and EMC does not support your configuration. You need to contemplate how many SE datasources live on this AU, and if you can upgrade SE on the AU to be in a supported configuration for this datasource, while not breaking other SE datasources on this AU.
- Failed to execute Sym CLI symcfg because all connection attempts failed to servers in Balanced or Ordered service entries in the netcnfg file.
This is most likely due to an environmental/configuration issues when setting up SolutionsEnabler. Check server certificates to verify they are pointing to the correct domain.
Datasource 1 and AU A is using SE 22.214.171.124 to talk to a SE 126.96.36.199 server Datasource 2 and AU A is using SE 188.8.131.52 to talk to a SE 8.4 server -> This datasource fails with The client SYMAPI version is not supported by the server.
If you upgrade the AU's SE install from 184.108.40.206 to 8.4, you may fix datasource 2, but will definitely break datasource 1, because you cannot use a newer SE version as a client against an older server
- Failed to execute Sym CLI symcfg, because No devices were found
Your Solutions Enabler datasource is correctly pointed at a path to a Solutions Enabler installation, but when the datasource polls, a symcfg list -v lists no local arrays. Review how this datasource has the "Connection Caching" value set - if it is LOCAL, then the Solutions Enabler installation on the Acquisition Unit hosting this datasource will run the commands locally. This would mean that this AU would need block storage connectivity to one or more Symmetrix arrays to be able to proceed. If your datasource is supposed to be using Solutions/Enabler client/server mode, "Connection Caching" should be set to REMOTE or REMOTE_CACHED. Remember that client/server mode additionally requires proper definition in the datasource, and creation of a valid netcnfg alias on the Solutions Enabler installation on the AU hosting the datasource.
- Failed to execute Sym CLI symcfg, because No more sessions can be started, until other open sessions are closed
This message may be seen when your OCI Solutions Enabler datasource is using client/server mode and the remote SE server is unhealthy, likely due to memoryrelated issues. Restarting the storsrvd daemon/Window service on the remote machine should be attempted. If that does not resolve the issue, contact EMC for assessing why the SE server is unreliable.
The "symcfg list -srp" CLI command may return incorrect data when collected with Solutions Enabler 8.1.0 or earlier from a Solutions Enabler server 8.3 or later.
NetApp has determined that despite what EMC says are supported configurations, certain versions of Solutions Enabler will return inaccurate or unreliable data when used against newer Solutions Enabler versions.
OCI activity prohibits:
- Using SE 7.x against all 8.x versions
- Using SE 8.1.0 or earlier against 8.3.0 or higher
- Using SE 8.2.0 or earlier against 8.4.0 or higher
If you encounter this error message, you need to take a step back and holistically assess your OCI server:
- What Acq Unit hosts this failing datasource?
- Are there other SE datasources on this AU?
- What are all the server SE versions that the SE datasources on this AU are talking to?
If you see this error message, and you only have one SE datasource on the AU, you should upgrade the SE version on the AU to match the Server SE version. If you have more than one SE datasource on this AU, you need to contemplate if you can upgrade SE, to what version, and whether that configuration will work. Unfortunately, in some configurations, you may need to deploy a new RAU with a different version of SE installed to be able to support your environment.
- Failed to execute Sym CLI symsnapvx, because '-linked': Incompatible option
OCI 7.3.5 and higher, as well as OCI 7.3.1 and higher running Data Source Service Pack 5 or higher now have enhanced Vmax3+ Linked Snapshot support. This enhancement causes OCI to run additional symcli commands against Vmax3 or newer arrays, to facilitate reporting Linked Snapshot to parent/source volume relationships. This failure message can be seen in these OCI versions if a Solutions Enabler datasource is utilizing old versions of Solutions Enabler in Vmax3 environments. NetApp first saw this problem on a Solutions Enabler 220.127.116.11 environment with a mix of Vmax1, 2 and 3 arrays - only the Vmax3 array discovery is negatively impacted because OCI now expects that the Solutions Enabler environment ought to support symsnapvx commands. Customers should ensure that their Solutions Enabler environment consists of appropriate versions for the hardware under management.