Description of problem: When we use nfs backend in cinder and attach a cinder volume to an instance, the instance access to the file in nfs share, which is named like volume-<volume id>. When the volume is attached to an instance and we take snapshot with "openstack volume snapshot create <volume> --force", it will create the following 3 files in nfs share. (1) volume-<volume id> base image freezed when taking snapshot (2) volume-<volume id>-<snapshot id> diff image where instance should write into after taking snapshot (3) volume-<volume id>.info json file to manage active snapshot As described above, after taking snapshot, the instance should write into (2) volume-<volume id>-<snapshot id> . It works just after taking snapshot, but if we stop and start the instance, the instance starts to write into (1) volume-<volume id>, which it should not modify. Version-Release number of selected component (if applicable): RHOSP13 Confirmed that the problem is reproduced in z8 How reproducible: Always Steps to Reproduce: 1. Create a volume in cinder nfs backend 2. Create a bfv instance with the volume 3. Take snapshot of the volume 4. Stop and Start the instance Actual results: The instance keeps writing into volume-<volume id>-<snapshot id> Expected results: The instance writes into volume-<volume id> Additional info: Based on the quick investigation the problem is happening because taking snapshot does not update bdm in nova. This caused the instance to use its original path when starting up,.
Pasting the log which shows that I reproduced the problem in my test env. /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf ~~~ ... [my_nfs] backend_host=hostgroup volume_backend_name=tripleo_nfs volume_driver=cinder.volume.drivers.nfs.NfsDriver nas_host=172.17.3.210 nas_share_path=/var/share nfs_snapshot_support=true ~~~ ~~~ [heat-admin@controller-0 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 13.0.8 (Queens) ~~~ 1. Create instance which has boot volume deployed in nfs backend ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack server show nfsinstance +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-0.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000014 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2019-10-02T07:38:02.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | private-network=192.168.1.6 | | config_drive | | | created | 2019-10-02T07:37:53Z | | flavor | m1.tiny (14202d99-0950-4c6d-8e55-41aaacf637a3) | | hostId | 8a640410721dc38e21a3d0f21b746200c3015da0d1f818587491916c | | id | a604ec10-00d6-468f-9603-6e061a149ac6 | | image | | | key_name | None | | name | nfsinstance | | progress | 0 | | project_id | d2d2e530c4a54918af770c11eb2a8413 | | properties | | | security_groups | name='default' | | status | ACTIVE | | updated | 2019-10-02T07:38:02Z | | user_id | c45dd1d768024a0db0d6e06fd8d683aa | | volumes_attached | id='b3ca15c9-888a-4f9f-befb-3c920b97ff65' | +-------------------------------------+----------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack volume show nfsinstancetest +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [{u'server_id': u'a604ec10-00d6-468f-9603-6e061a149ac6', u'attachment_id': u'0e9d21f1-89de-4a6e-9017-cd20027b6a52', u'attached_at': u'2019-10-02T07:37:58.000000', u'host_name': u'compute-0.localdomain', u'volume_id': u'b3ca15c9-888a-4f9f-befb-3c920b97ff65', u'device': u'/dev/vda', u'id': u'b3ca15c9-888a-4f9f-befb-3c920b97ff65'}] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2019-10-02T07:36:21.000000 | | description | None | | encrypted | False | | id | b3ca15c9-888a-4f9f-befb-3c920b97ff65 | | migration_status | None | | multiattach | False | | name | nfsinstancetest | | os-vol-host-attr:host | hostgroup@my_nfs#tripleo_nfs | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | d2d2e530c4a54918af770c11eb2a8413 | | properties | attached_mode='rw' | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | in-use | | type | nfs | | updated_at | 2019-10-02T07:37:58.000000 | | user_id | c45dd1d768024a0db0d6e06fd8d683aa | | volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'cirros-0.3.5-x86_64-disk.img', u'image_id': u'df2181aa-6879-4837-9544-d91a42b84fe1', u'hw_rng_model': u'virtio', u'min_disk': u'0', u'checksum': u'f8ab98ff5e73ebab884d80c9dc9c7290', u'size': u'13267968'} | +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ~~~ 2. See that instance is running with the base image ~~~ [heat-admin@compute-0 ~]$ sudo virsh dumpxml instance-00000014 ... <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65'/> <backingStore/> <target dev='vda' bus='virtio'/> <serial>b3ca15c9-888a-4f9f-befb-3c920b97ff65</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> ... [heat-admin@compute-0 ~]$ sudo ls -lah /var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/ total 18M drwxr-xr-x. 2 root root 123 Oct 2 07:36 . drwxr-xr-x. 3 42436 42436 46 Oct 2 07:37 .. -rw-rw-rw-. 1 root root 1.0G Oct 2 07:33 volume-253785cb-0e2b-4183-9688-b09070aaa043 -rw-rw-rw-. 1 qemu qemu 1.0G Oct 2 07:39 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65 ~~~ 3. Create snapshot for the instance ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot create --volume nfsinstancetest --force testsnap +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | created_at | 2019-10-02T07:41:53.181796 | | description | None | | id | 96b7cc38-6bb0-4de8-8b66-af6797ebdeff | | name | testsnap | | properties | | | size | 1 | | status | creating | | updated_at | None | | volume_id | b3ca15c9-888a-4f9f-befb-3c920b97ff65 | +-------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot list +--------------------------------------+---------------------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------------------------+-------------+-----------+------+ | 96b7cc38-6bb0-4de8-8b66-af6797ebdeff | testsnap | None | available | 1 | | 1151670a-b942-4d0b-b25d-c152e30f186f | snapshot for ebstestimage | None | available | 1 | +--------------------------------------+---------------------------+-------------+-----------+------+ ~~~ 4. And now see that the instance is running with diff image ~~~ [heat-admin@compute-0 ~]$ sudo virsh dumpxml instance-00000014 ... <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.96b7cc38-6bb0-4de8-8b66-af6797ebdeff'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <serial>b3ca15c9-888a-4f9f-befb-3c920b97ff65</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> ... [heat-admin@compute-0 ~]$ sudo ls -lah /var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/ total 18M drwxr-xr-x. 2 root root 267 Oct 2 07:41 . drwxr-xr-x. 3 42436 42436 46 Oct 2 07:37 .. -rw-rw-rw-. 1 root root 1.0G Oct 2 07:33 volume-253785cb-0e2b-4183-9688-b09070aaa043 -rw-rw-rw-. 1 qemu qemu 1.0G Oct 2 07:39 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65 -rw-rw-rw-. 1 qemu qemu 193K Oct 2 07:41 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.96b7cc38-6bb0-4de8-8b66-af6797ebdeff -rw-rw-rw-. 1 root root 223 Oct 2 07:41 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.info ~~~ 5. stop and start the instance ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack server stop a604ec10-00d6-468f-9603-6e061a149ac6 (overcloud) [stack@undercloud-0 ~]$ openstack server start a604ec10-00d6-468f-9603-6e061a149ac6 (overcloud) [stack@undercloud-0 ~]$ ~~~ 6. Now instance is running with the base image ~~~ [heat-admin@compute-0 ~]$ sudo virsh dumpxml instance-00000014 ... <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65'/> <backingStore/> <target dev='vda' bus='virtio'/> <serial>b3ca15c9-888a-4f9f-befb-3c920b97ff65</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> ... [heat-admin@compute-0 ~]$ sudo ls -lah /var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/ total 19M drwxr-xr-x. 2 root root 267 Oct 2 07:41 . drwxr-xr-x. 3 42436 42436 46 Oct 2 07:46 .. -rw-rw-rw-. 1 root root 1.0G Oct 2 07:33 volume-253785cb-0e2b-4183-9688-b09070aaa043 -rw-rw-rw-. 1 qemu qemu 1.0G Oct 2 07:47 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65 -rw-rw-rw-. 1 root root 768K Oct 2 07:46 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.96b7cc38-6bb0-4de8-8b66-af6797ebdeff -rw-rw-rw-. 1 root root 223 Oct 2 07:41 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.info ~~~
This looks like a valid bug to me. I haven't fully grokked volume_snapshot_create, yet, but you don't need to. We don't store any data about the snapshot independent of the libvirt xml, which we rewrite when power_on makes a call to _hard_reboot. This is probably not a simple fix.
Sorry but I noticed that I misplaced actual result/expected result. They should be; Actual results: The instance writes into volume-<volume id> Expected results: The instance keeps writing into volume-<volume id>-<snapshot id>
*** Bug 1854796 has been marked as a duplicate of this bug. ***
*** Bug 1854826 has been marked as a duplicate of this bug. ***
*** Bug 1927136 has been marked as a duplicate of this bug. ***
This is now going to be fixed within openstack-cinder within the remotefs driver by updating the connection_info of the underlying attachments *before* Nova refreshes things by GET'ing them again. *** This bug has been marked as a duplicate of bug 1977845 ***