Snapshot, Volume creation/deletion fail with exception during upgrade or node shutdown
Applies to
- OpenStack Cinder Driver for SolidFire
- Element Software
Issue
Snapshot or volume creation/deletion fail with one of following exceptions which are returned from SolidFire Storage System as API call response.
xNotPrimary
xDBOperationTimeout
xDBConnectionLoss
xNoHandler
xSnapshotFailed
xRecvTimeout
xDBNoSuchPath
xPermissionDenied
Example error message on cinder.log
:
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server [req-75f94287-50a6-4f39-aeca-62b5aa94a6c8 e0bdb7eb798846e794e9a124f30e5542 98f1d3382f234278a90053c0aa081600 - 562bfa8e1b904076931ebdf1dfdcd3fa 562bfa8e1b904076931ebdf1dfdcd3fa] Exception during message handling: SolidFireAPIException: API response: {u'id': None, u'error': {u'message': u'DBClient connection lost during operation at [/accounts/492]', u'code': 500, u'name': u'xDBConnectionLoss'}}
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "<string>", line 2, in delete_snapshot
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server return f(*a, **k)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1163, in delete_snapshot
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server snapshot.save()
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1153, in delete_snapshot
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server self.driver.delete_snapshot(snapshot)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 1557, in delete_snapshot
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server accounts = self._get_sfaccounts_for_tenant(snapshot['project_id'])
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 1081, in _get_sfaccounts_for_tenant
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server 'ListAccounts', {})['result']['accounts']
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 118, in func_retry
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 529, in _issue_api_request
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server raise exception.SolidFireAPIException(msg)
2021-05-28 06:04:00.235 38 ERROR oslo_messaging.rpc.server SolidFireAPIException: API response: {u'id': None, u'error': {u'message': u'DBClient connection lost during operation at [/accounts/492]', u'code': 500, u'name': u'xDBConnectionLoss'}}