OpenStack creates two Element volumes with the same OpenStack name
Issue
This can show up in three possible ways:
- Element Software did not return a create volume status within 30 seconds to OpenStack
- OpenStack cinder was able to create two volumes with the same OpenStack name
- OpenStack Nova was unable to attach to either of the duplicate-OpenStack-named volumes
Example errors from the cinder volume manager log:
Create volume timeout signature:
2021-03-11 18:15:38.067 60 WARNING cinder.volume.manager [req-bed9bef5-bc6b-4528-be6d-893667ca2d3d e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (2270fad5-762e-424a-8af4-82b1d12b27ce) transitioned into state 'FAILURE' from state 'RUNNING'
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager Traceback (most recent call last):
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager result = task.execute(**arguments)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1010, in execute
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager model_update = self._create_raw_volume(volume, **volume_spec)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 977, in _create_raw_volume
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager ret = self.driver.create_volume(volume)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 1398, in create_volume
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager model_update = self._do_volume_create(sf_account, params)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 799, in _do_volume_create
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager 'CreateVolume', params, endpoint=endpoint)['result']['volumeID']
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 118, in func_retry
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager return f(*args, **kwargs)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 515, in _issue_api_request
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager timeout=30)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/requests/api.py", line 112, in post
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager return request('post', url, data=data, json=json, **kwargs)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager return session.request(method=method, url=url, **kwargs)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager resp = self.send(prep, **send_kwargs)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager r = adapter.send(request, **kwargs)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 514, in send
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager raise ReadTimeout(e, request=request)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager ReadTimeout: HTTPSConnectionPool(host='10.45.101.80', port=443): Read timed out. (read timeout=30)
2021-03-11 18:15:38.067 60 ERROR cinder.volume.manager
2021-03-11 18:15:38.072 60 WARNING cinder.volume.manager [req-bed9bef5-bc6b-4528-be6d-893667ca2d3d e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7348fad5-779e-464b-9af4-82b1d12b97ce) transitioned into state 'REVERTED' from state 'REVERTING'
Subsequent failure to attach signature:
2021-03-11 18:17:44.104 60 INFO cinder.volume.manager [req-ca61f2c4-a9de-43e5-a7ef-b8b686610c88 e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Created volume successfully.
2021-03-11 18:17:44.218 60 INFO cinder.volume.flows.manager.create_volume [req-4fb3a2d2-4e04-483e-8408-39ed28943192 e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Volume volume-835ed7d1-7e0b-4673-a863-4d0e28a31f04 (835ed7d1-7e0b-4673-a863-4d0e28a31f04): create
d successfully
2021-03-11 18:17:44.230 60 ERROR cinder.volume.drivers.solidfire [req-ea456ea3-afef-4246-bcab-0f01dda4b4a1 e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Found 2 volumes mapped to id: ba123ceg-c980-4c6c-g20f-5e8038171589.
2021-03-11 18:17:44.231 60 INFO cinder.message.api [req-ea456ea3-afef-4246-bcab-0f01dda4b4a1 e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Creating message record for request_id = req-ea456ea3-afef-4246-bcab-0f01dda4b4a1
2021-03-11 18:17:44.240 60 INFO cinder.volume.manager [req-4fb3a2d2-4e04-483e-8408-39ed28943192 e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Created volume successfully.
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server [req-ea456ea3-afef-4246-bcab-0f01dda4b4a1 e87e52e2605c4db89b545a793e27c391 1dbe350c09fc40e798c76832fb6ac990 - default default] Exception during message handling: DuplicateSfVolumeNames: Detected more than one volume with name ba123ceg-c980-4c6c-g20f-5e8038171589
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4457, in attachment_update
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server fields.VolumeAttachStatus.ERROR_ATTACHING})
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server self.force_reraise()
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4447, in attachment_update
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server connector.get('mountpoint', 'na'))
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 1952, in attach_volume
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server sf_vol = self._get_sf_volume(volume['id'], params)
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/solidfire.py", line 951, in _get_sf_volume
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server raise exception.DuplicateSfVolumeNames(vol_name=uuid)
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server DuplicateSfVolumeNames: Detected more than one volume with name ba123ceg-c980-4c6c-g20f-5e8038171589
2021-03-11 18:17:44.244 60 ERROR oslo_messaging.rpc.server