Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1714805

Summary: [OSP13] cinder-backup backup creation fails with NFS driver
Product: Red Hat OpenStack Reporter: Brendan Shephard <bshephar>
Component: openstack-cinderAssignee: Alan Bishop <abishop>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: urgent Docs Contact: Tana <tberry>
Priority: urgent    
Version: 13.0 (Queens)CC: abishop, amcleod, astupnik, dsedgmen, eharney, gfidente, gkadam, pcaruana, ravsingh, tenobreg, tshefi
Target Milestone: z9Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-12.0.8-3.el7ost Doc Type: Bug Fix
Doc Text:
Previously, the cinder NFS driver contained a bug that prevented the driver from creating a snapshot during volume clone operations, and cloned volumes were used when backing up NFS volumes. As a result, you were not able to back up cinder NFS volumes. With this release, the bug has been fixed and you can back up cinder NFS volumes successfully.
Story Points: ---
Clone Of:
: 1753055 (view as bug list) Environment:
Last Closed: 2019-11-07 13:59:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1753055, 1753056, 1766133    
Bug Blocks:    

Description Brendan Shephard 2019-05-28 22:25:41 UTC
Description of problem:
When creating a snapshot of an active volume, the following error is observed:
Exception during message handling: AttributeError: 'Snapshot' object has no attribute '_context'


Version-Release number of selected component (if applicable):
openstack-cinder-12.0.4-3.el7ost.noarch 


How reproducible:
Every time a backup is attempted of an attached volume

Steps to Reproduce:
1. Create NFS backend for Cinder
2. Attach volume to Instance
3. Try to create snapshot of volume

Actual results:
oslo_messaging.rpc.server AttributeError: 'Snapshot' object has no attribute '_context'
Backup fails

Expected results:
Backup would succeed

Additional info:
This appears to be addressed by: https://review.opendev.org/#/c/570885/

Comment 13 Alan Bishop 2019-08-23 04:36:16 UTC
The upstream patch that fixes this issue has merged on master. The next step is to backport it to earlier releases.

Comment 14 Alan Bishop 2019-08-23 04:40:16 UTC
Sorry, I jumped the gun. The upstream patch has a few approvals, but has not merged yet (though it's making good progress).

Comment 16 Alan Bishop 2019-09-11 15:13:20 UTC
The upstream patch has been accepted, and I started the process of backporting the fix to the upstream stable releases. The downstream backports will go quickly once the upstream stable patches are complete.

Comment 24 Tzach Shefi 2019-10-28 11:54:39 UTC
While trying to verify with puddle 13  -p 2019-10-23.1 which includes fixed-in for this bz. 
Keep hitting SQL issues on three separate deployments, 
can't verify as I'm failing to boot instance and do Cinder related commands. 

Opened a blocking bz
https://bugzilla.redhat.com/show_bug.cgi?id=1766133

Comment 25 Tzach Shefi 2019-10-28 13:40:13 UTC
Verified on:
python2-os-brick-2.3.8-1.el7ost.noarch

Created an NFS backed volume:
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volNFS
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2019-10-28T13:25:48.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volNFS                               |
| os-vol-host-attr:host          | controller-0@nfs#nfs                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 2d63596584a84572aa0742301c1afe37     |
| replication_status             | None                                 |
| size                           | 2                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | 2019-10-28T13:25:49.000000           |
| user_id                        | ef7525ff5409412ab964865920b3a4d9     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+


Booted an instance, attached volume to instance

(overcloud) [stack@undercloud-0 ~]$ nova volume-attach af661e74-e3e0-47e3-8c23-c6d2973f0a4a 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 auto
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 |
| serverId | af661e74-e3e0-47e3-8c23-c6d2973f0a4a |
| volumeId | 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 |
+----------+--------------------------------------+

Now lets  mkfs and write some data on vol, inside instance//
..
# mkdir mnt
# mount /dev/vdb mnt/
# vi mnt/tshefi.txt

Drums.. now create a cinder snapshot of said attached vol

(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 --force --name SnapOfAttachedNFSBackedVolume
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2019-10-28T13:33:11.465138           |
| description | None                                 |
| id          | 4c22ca78-e7ea-432e-bcb9-9dfc547e8465 |
| metadata    | {}                                   |
| name        | SnapOfAttachedNFSBackedVolume        |
| size        | 2                                    |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 |
+-------------+--------------------------------------+


Wait for it wait for it.. 

Looking good snashot created without error
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+--------------------------------------+--------------------------------------+-----------+-------------------------------+------+
| ID                                   | Volume ID                            | Status    | Name                          | Size |
+--------------------------------------+--------------------------------------+-----------+-------------------------------+------+
| 4c22ca78-e7ea-432e-bcb9-9dfc547e8465 | 0970fe8e-f985-41e0-aae2-0ea3cfceeec6 | available | SnapOfAttachedNFSBackedVolume | 2    |


Now lets just be extra sure it has our original content:

(overcloud) [stack@undercloud-0 ~]$ cinder create 3 --snapshot-id 4c22ca78-e7ea-432e-bcb9-9dfc547e8465 --name VolFromSnap
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2019-10-28T13:34:57.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 418d8910-d281-48a7-817b-a55c1553c3e8 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | VolFromSnap                          |
| os-vol-host-attr:host          | controller-0@nfs#nfs                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 2d63596584a84572aa0742301c1afe37     |
| replication_status             | None                                 |
| size                           | 3                                    |
| snapshot_id                    | 4c22ca78-e7ea-432e-bcb9-9dfc547e8465 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | 2019-10-28T13:34:57.000000           |
| user_id                        | ef7525ff5409412ab964865920b3a4d9     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+


Attach second/new vol to instance as vdc

(overcloud) [stack@undercloud-0 ~]$ nova volume-attach af661e74-e3e0-47e3-8c23-c6d2973f0a4a 418d8910-d281-48a7-817b-a55c1553c3e8 auto
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | 418d8910-d281-48a7-817b-a55c1553c3e8 |
| serverId | af661e74-e3e0-47e3-8c23-c6d2973f0a4a |
| volumeId | 418d8910-d281-48a7-817b-a55c1553c3e8 |
+----------+--------------------------------------+


Login to instance mount second vol check data

(overcloud) [stack@undercloud-0 ~]$ ssh cirros.0.216
Warning: Permanently added '10.0.0.216' (ECDSA) to the list of known hosts.
cirros.0.216's password: 
$ sudo -i
# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    2G  0 disk /root/mnt
vdc     253:32   0    2G  0 disk 
# mkdir mnt2
# mount /dev/vdc mnt2/
# cat mnt/tshefi.txt 
Hello...
# cat mnt2/tshefi.txt   -> vol from snap has same data OK to verify bz. 
Hello...

Snapshot of an attached NFS backed volume is now working as expected.

Comment 27 errata-xmlrpc 2019-11-07 13:59:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3800