Bug 1624744

Summary: NFS assisted snapshot fails
Product: Red Hat OpenStack Reporter: Keigo Noha <knoha>
Component: openstack-cinderAssignee: Cinder Bugs List <cinder-bugs>
Status: CLOSED DUPLICATE QA Contact: Avi Avraham <aavraham>
Severity: low Docs Contact: Kim Nylander <knylande>
Priority: unspecified    
Version: 13.0 (Queens)CC: abishop, afazekas, apevec, chkumar, eharney, knoha, lhh, mkopec, slinaber, srevivo, tshefi, udesale
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-21 15:04:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Keigo Noha 2018-09-03 07:58:42 UTC
Description of problem:
tempest should run its test with proper user.

In some cases, tempest needs to run its test with admin role.
For example, if cinder uses NFS backend, snpashot used tests require admin user to run assisted volume snapshots.

Because assisted volume snapshots is limited to admin user only by nova's policy.


nova/policies/assisted_volume_snapshots.py
~~~
from oslo_policy import policy

from nova.policies import base


POLICY_ROOT = 'os_compute_api:os-assisted-volume-snapshots:%s'


assisted_volume_snapshots_policies = [
    policy.DocumentedRuleDefault(
        POLICY_ROOT % 'create',
        base.RULE_ADMIN_API,
        "Create an assisted volume snapshot",
        [
            {
                'path': '/os-assisted-volume-snapshots',
                'method': 'POST'
            }
        ]),
    policy.DocumentedRuleDefault(
        POLICY_ROOT % 'delete',
        base.RULE_ADMIN_API,
        "Delete an assisted volume snapshot",
        [
            {
                'path': '/os-assisted-volume-snapshots/{snapshot_id}',
                'method': 'DELETE'
            }
        ]),
]
~~~

By this limitation, 4 test are failed.

~~~
tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_delete_with_volume_in_use
tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_offline_delete_online
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_create_ebs_image_and_check_boot
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
~~~

Version-Release number of selected component (if applicable):
Current openstack-tempest in RHOSP13.

How reproducible:
Every time

Steps to Reproduce:
1.
2.
3.

Actual results:
If cinder uses nfs backend, the tests fails.

Expected results:
Tempest should run the tests with proper user.

Additional info:

Comment 1 Martin Kopec 2018-09-03 08:31:20 UTC
If the tests require an admin role and admin user, then it's needed to be set up in tempest.conf. All configuration (credentials, resources ..) tempest uses are stored in that file. Therefore I don't understand, how this can be a tempest bug.

Comment 5 Attila Fazekas 2018-09-10 23:03:04 UTC
My first guess is the cinder policy determinate is the user able or not able to do snapshot (default they can, and cinder also considered they can in this case).

The assisted snapshot is admin only, because it is supposed to be called by a service user with admin permission (only cinder) instead of delegating the caller user permissions. 

Likely the bug is on cinder (cinder config?) side, unless letting users to call that internal api directly really safe (unlikely).

Comment 6 Martin Kopec 2018-09-10 23:26:07 UTC
It is very strange that a user needs to have an admin credentials in order to create a snapshot. Based on that and on the launchpad bug Attila attached in the comment above, it looks like a cinder issue. The test should be allowed also for non-admin users.
I'm moving the bug to cinder component in order to get cinder point of view on this issue.

Comment 7 Eric Harney 2018-09-11 15:31:32 UTC
The root cause of this failure is noted in the nova-api log from the sosreport here:

2018-07-31 00:37:34.161 19 DEBUG nova.api.openstack.wsgi [req-fe35624b-49b2-489d-9d2b-d8813fef9c95 53d8fb1b391e4d3cb39602799668e722 8387183b6b6b42b3a5b16c8e41b3dd2f - default default] Returning 403 to user: Policy doesn't allow os_compute_api:os-assisted-volume-snapshots:create to be performed. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1064


When deploying the NFS driver, both Cinder and Nova need configuration in place for calls from Cinder to Nova for assisted snapshots to succeed.

Cinder needs to point to the correct Nova endpoint, and Nova policy must allow the operation.  I haven't studied which pieces of this are missing yet.

Comment 8 Keigo Noha 2018-10-10 00:38:34 UTC
Hello Eric,

Do you have any updates on this bugzilla?

Best Regards,
Keigo Noha

Comment 9 Keigo Noha 2018-11-07 08:10:29 UTC
Hello Eric,

A configuration in upstream works fine in the customer's environment.
So, this bugzilla can be closed as NOTABUG.

Best Regards,
Keigo Noha

Comment 14 Alan Bishop 2018-11-21 15:04:44 UTC
The root cause of the NFS assisted snapshot failure is the [nova] section of cinder.conf is missing the required authentication information. That problem has been solved upstream, and backports are under way. The fix for OSP-13 is being tracked in bug #1641825.

*** This bug has been marked as a duplicate of bug 1641825 ***