Bug 1757691
| Summary: | Instance uses base image file when it is rebooted after snapshot creation if cinder nfs backend is used | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Takashi Kajinami <tkajinam> |
| Component: | openstack-nova | Assignee: | Lee Yarwood <lyarwood> |
| Status: | CLOSED DUPLICATE | QA Contact: | OSP DFG:Compute <osp-dfg-compute> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 13.0 (Queens) | CC: | abishop, chhu, dasmith, eglynn, eharney, gcharot, jhakimra, jparker, kchamart, lyarwood, sbauza, senrique, sgordon, takito, vromanso |
| Target Milestone: | Upstream M3 | Keywords: | Patch, Triaged, ZStream |
| Target Release: | 17.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-05 17:20:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1927136 | ||
|
Description
Takashi Kajinami
2019-10-02 08:01:19 UTC
Pasting the log which shows that I reproduced the problem in my test env.
/var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
~~~
...
[my_nfs]
backend_host=hostgroup
volume_backend_name=tripleo_nfs
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nas_host=172.17.3.210
nas_share_path=/var/share
nfs_snapshot_support=true
~~~
~~~
[heat-admin@controller-0 ~]$ cat /etc/rhosp-release
Red Hat OpenStack Platform release 13.0.8 (Queens)
~~~
1. Create instance which has boot volume deployed in nfs backend
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack server show nfsinstance
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000014 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-10-02T07:38:02.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | private-network=192.168.1.6 |
| config_drive | |
| created | 2019-10-02T07:37:53Z |
| flavor | m1.tiny (14202d99-0950-4c6d-8e55-41aaacf637a3) |
| hostId | 8a640410721dc38e21a3d0f21b746200c3015da0d1f818587491916c |
| id | a604ec10-00d6-468f-9603-6e061a149ac6 |
| image | |
| key_name | None |
| name | nfsinstance |
| progress | 0 |
| project_id | d2d2e530c4a54918af770c11eb2a8413 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2019-10-02T07:38:02Z |
| user_id | c45dd1d768024a0db0d6e06fd8d683aa |
| volumes_attached | id='b3ca15c9-888a-4f9f-befb-3c920b97ff65' |
+-------------------------------------+----------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack volume show nfsinstancetest
+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'a604ec10-00d6-468f-9603-6e061a149ac6', u'attachment_id': u'0e9d21f1-89de-4a6e-9017-cd20027b6a52', u'attached_at': u'2019-10-02T07:37:58.000000', u'host_name': u'compute-0.localdomain', u'volume_id': u'b3ca15c9-888a-4f9f-befb-3c920b97ff65', u'device': u'/dev/vda', u'id': u'b3ca15c9-888a-4f9f-befb-3c920b97ff65'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2019-10-02T07:36:21.000000 |
| description | None |
| encrypted | False |
| id | b3ca15c9-888a-4f9f-befb-3c920b97ff65 |
| migration_status | None |
| multiattach | False |
| name | nfsinstancetest |
| os-vol-host-attr:host | hostgroup@my_nfs#tripleo_nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | d2d2e530c4a54918af770c11eb2a8413 |
| properties | attached_mode='rw' |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | nfs |
| updated_at | 2019-10-02T07:37:58.000000 |
| user_id | c45dd1d768024a0db0d6e06fd8d683aa |
| volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'cirros-0.3.5-x86_64-disk.img', u'image_id': u'df2181aa-6879-4837-9544-d91a42b84fe1', u'hw_rng_model': u'virtio', u'min_disk': u'0', u'checksum': u'f8ab98ff5e73ebab884d80c9dc9c7290', u'size': u'13267968'} |
+--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
~~~
2. See that instance is running with the base image
~~~
[heat-admin@compute-0 ~]$ sudo virsh dumpxml instance-00000014
...
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>b3ca15c9-888a-4f9f-befb-3c920b97ff65</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
...
[heat-admin@compute-0 ~]$ sudo ls -lah /var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/
total 18M
drwxr-xr-x. 2 root root 123 Oct 2 07:36 .
drwxr-xr-x. 3 42436 42436 46 Oct 2 07:37 ..
-rw-rw-rw-. 1 root root 1.0G Oct 2 07:33 volume-253785cb-0e2b-4183-9688-b09070aaa043
-rw-rw-rw-. 1 qemu qemu 1.0G Oct 2 07:39 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65
~~~
3. Create snapshot for the instance
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot create --volume nfsinstancetest --force testsnap
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2019-10-02T07:41:53.181796 |
| description | None |
| id | 96b7cc38-6bb0-4de8-8b66-af6797ebdeff |
| name | testsnap |
| properties | |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | b3ca15c9-888a-4f9f-befb-3c920b97ff65 |
+-------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot list
+--------------------------------------+---------------------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+---------------------------+-------------+-----------+------+
| 96b7cc38-6bb0-4de8-8b66-af6797ebdeff | testsnap | None | available | 1 |
| 1151670a-b942-4d0b-b25d-c152e30f186f | snapshot for ebstestimage | None | available | 1 |
+--------------------------------------+---------------------------+-------------+-----------+------+
~~~
4. And now see that the instance is running with diff image
~~~
[heat-admin@compute-0 ~]$ sudo virsh dumpxml instance-00000014
...
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='native'/>
<source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.96b7cc38-6bb0-4de8-8b66-af6797ebdeff'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<serial>b3ca15c9-888a-4f9f-befb-3c920b97ff65</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
...
[heat-admin@compute-0 ~]$ sudo ls -lah /var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/
total 18M
drwxr-xr-x. 2 root root 267 Oct 2 07:41 .
drwxr-xr-x. 3 42436 42436 46 Oct 2 07:37 ..
-rw-rw-rw-. 1 root root 1.0G Oct 2 07:33 volume-253785cb-0e2b-4183-9688-b09070aaa043
-rw-rw-rw-. 1 qemu qemu 1.0G Oct 2 07:39 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65
-rw-rw-rw-. 1 qemu qemu 193K Oct 2 07:41 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.96b7cc38-6bb0-4de8-8b66-af6797ebdeff
-rw-rw-rw-. 1 root root 223 Oct 2 07:41 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.info
~~~
5. stop and start the instance
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack server stop a604ec10-00d6-468f-9603-6e061a149ac6
(overcloud) [stack@undercloud-0 ~]$ openstack server start a604ec10-00d6-468f-9603-6e061a149ac6
(overcloud) [stack@undercloud-0 ~]$
~~~
6. Now instance is running with the base image
~~~
[heat-admin@compute-0 ~]$ sudo virsh dumpxml instance-00000014
...
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>b3ca15c9-888a-4f9f-befb-3c920b97ff65</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
...
[heat-admin@compute-0 ~]$ sudo ls -lah /var/lib/nova/mnt/755caf2d247b0b75a52b1c58b757da63/
total 19M
drwxr-xr-x. 2 root root 267 Oct 2 07:41 .
drwxr-xr-x. 3 42436 42436 46 Oct 2 07:46 ..
-rw-rw-rw-. 1 root root 1.0G Oct 2 07:33 volume-253785cb-0e2b-4183-9688-b09070aaa043
-rw-rw-rw-. 1 qemu qemu 1.0G Oct 2 07:47 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65
-rw-rw-rw-. 1 root root 768K Oct 2 07:46 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.96b7cc38-6bb0-4de8-8b66-af6797ebdeff
-rw-rw-rw-. 1 root root 223 Oct 2 07:41 volume-b3ca15c9-888a-4f9f-befb-3c920b97ff65.info
~~~
This looks like a valid bug to me. I haven't fully grokked volume_snapshot_create, yet, but you don't need to. We don't store any data about the snapshot independent of the libvirt xml, which we rewrite when power_on makes a call to _hard_reboot. This is probably not a simple fix. Sorry but I noticed that I misplaced actual result/expected result. They should be; Actual results: The instance writes into volume-<volume id> Expected results: The instance keeps writing into volume-<volume id>-<snapshot id> *** Bug 1854796 has been marked as a duplicate of this bug. *** *** Bug 1854826 has been marked as a duplicate of this bug. *** *** Bug 1927136 has been marked as a duplicate of this bug. *** This is now going to be fixed within openstack-cinder within the remotefs driver by updating the connection_info of the underlying attachments *before* Nova refreshes things by GET'ing them again. *** This bug has been marked as a duplicate of bug 1977845 *** |