OpenStack: During controller upgrade, write access to Cinder volumes is lost
Applies to
- OpenStack Yoga
- Canonical packages
- NFSv3 storage pool
Issue
During a OpenStack controller upgrade from Xena to Yoga, user ownership of Cinder volumes on a backing NFSv3 mounted storage pool changes. As a result, this causes instances to lose write access to their underlying storage.
Details of the environment where this issue has been seen:
- Systemd deployment
- Canonical Cinder packages
- NFSv3 Storage Pools
When upgrading the 'cinder-common' package, the following changes are observed:
- Prior to the cinder-common package upgrade, the Cinder volumes on the Compute nodes are owned by a user such as libvirt-qemu:
$ getfacl -e /opt/stack/data/nova/mnt/d25beb55fef91aab4945de91411d8dc5/volume-d6f844c6-5be0-41ba-ae85-898743dfd991
getfacl: Removing leading '/' from absolute path names
# file: opt/stack/data/nova/mnt/d25beb55fef91aab4945de91411d8dc5/volume-d6f844c6-5be0-41ba-ae85-898743dfd991
# owner: libvirt-qemu
# group: kvm
user::rw-
group::rw-
other::---
- The cinder-common package is upgraded
- Ownership of the Cinder volumes on the compute nodes changes to something like cinder:cinder (in this example, there is no cinder user on the compute host therefore a user id such as 64061 is seen):
$ getfacl -e /opt/stack/data/nova/mnt/d25beb55fef91aab4945de91411d8dc5/volume-d6f844c6-5be0-41ba-ae85-898743dfd991
getfacl: Removing leading '/' from absolute path names
# file: opt/stack/data/nova/mnt/d25beb55fef91aab4945de91411d8dc5/volume-d6f844c6-5be0-41ba-ae85-898743dfd991
# owner: 64061
# group: 64061
user::rw-
group::rw-
other::---
- Depending upon the unix permissions on the Cinder volumes, the instances either go into a 'read-only' state or access is lost completely