Bug 1119708

Summary: unable to modify owner for backup volume with nfs_7mode
Product: Red Hat OpenStack Reporter: bkopilov <bkopilov>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED NOTABUG QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 7)CC: bkopilov, eharney, nlevinki, scohen, yeylon
Target Milestone: ---Keywords: AutomationBlocker, Unconfirmed, ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-30 15:01:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs - /var/log and /etc none

Description bkopilov 2014-07-15 10:29:09 UTC
Created attachment 918115 [details]
logs - /var/log  and /etc

Description of problem:
openstack 5.0 , rhel 7 with nfs 7_mode backend.
Automation runs tempest code .
Tempest fails on :
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsTest.test_volume_backup_create_get_detailed_list_restore_delete

From the cinder/backup code we see problem when trying to chown of mounted path we got error .
Looks like the mount path is not valid .

2014-07-15 04:02:41.250 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f of=/dev/null count=1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:142
2014-07-15 04:02:41.303 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:142
2014-07-15 04:02:41.677 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Result was 1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:167
2014-07-15 04:02:41.678 18424 DEBUG cinder.volume.driver [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] volume 71e32945-6d61-45f2-9c8e-05f207f4ee6f: removing export _detach_volume /usr/lib/python2.7/site-packages/cinder/volume/driver.py:456
2014-07-15 04:02:41.724 18424 ERROR oslo.messaging.rpc.dispatcher [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Exception during message handling: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f
Exit code: 1
Stdout: ''
Stderr: '/bin/chown: changing ownership of \xe2\x80\x98/var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f\xe2\x80\x99: Invalid argument\n'
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 280, in create_backup
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     'fail_reason': unicode(err)})
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 273, in create_backup
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     backup_service)
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 498, in backup_volume
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     with utils.temporary_chown(volume_path):
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     return self.gen.next()
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 682, in temporary_chown
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     execute('chown', owner_uid, path, run_as_root=True)
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 136, in execute
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py", line 173, in execute
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher     cmd=' '.join(cmd))
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Exit code: 1
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Stdout: ''
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Stderr: '/bin/chown: changing ownership of \xe2\x80\x98/var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f\xe2\x80\x99: Invalid argument\n'
2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher 
2014-07-15 04:02:41.759 18424 INFO oslo.messaging._drivers.impl_rabbit [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Connected to AMQP server on 192.168.2.22:5672
2014-07-15 04:02:41.759 18424 ERROR oslo.messaging._drivers.common [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Returning exception Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f
Exit code: 1
Stdout: ''


Version-Release number of selected component (if applicable):


How reproducible:
Run tempest tests on nfs_7mode backend
automation test:
http://jenkins.rhev.lab.eng.brq.redhat.com:8080/view/RHOS-STORAGE-QE/view/RHOS-5.0-RHEL7.0/job/rhos-5-rhel-7.0-cinder-netapp_nfs_7mode/75/

Attaching to bug all logs for cinder / config .

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Eric Harney 2014-07-17 14:25:36 UTC
This command looks ok:

sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f

Can you try just running "chown 165 <file>" for some file on that mount point by hand?  I think this may be an NFS configuration error related to id mapping.

Note the second of these three messages that occurs when the chown is performed:
Jul 15 04:02:41 test9283 cinder-backup: 2014-07-15 04:02:41.303 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:142

Jul 15 04:02:41 test9283 kernel: NFS: v4 server netapp.qa.lab.tlv.redhat.com does not accept raw uid/gids. Reenabling the idmapper.

Jul 15 04:02:41 test9283 cinder-backup: 2014-07-15 04:02:41.677 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Result was 1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:167



If this happens when trying by hand we will need to look at the NFS server configuration, it isn't clear that this is an issue in Cinder itself.

Comment 4 bkopilov 2014-07-29 12:29:07 UTC
Hi , 
I installed baremetal machine configured with this backend .
machine ip : 10.35.3.109
user : root   pass: qum5net

I am checking this issue now , you can access to the machine if you like .

Thanks , 
Benny

Comment 5 bkopilov 2014-07-29 12:32:27 UTC
(In reply to bkopilov from comment #4)
> Hi , 
> I installed baremetal machine configured with this backend .
> machine ip : 10.35.3.109
> user : root   pass: qum5net
> 
>  I run the command manually and got this error:

[root@test632 d80058dce1a9f0bd61fe7f15b093af65]# chown 165 volume-4ddc2c23-47c9-402c-a386-1d03baedbd29
chown: changing ownership of ‘volume-4ddc2c23-47c9-402c-a386-1d03baedbd29’: Invalid argument

> 
> Thanks , 
> Benny

Comment 6 bkopilov 2014-07-29 12:52:23 UTC
(In reply to bkopilov from comment #5)
> (In reply to bkopilov from comment #4)
> > Hi , 
> > I installed baremetal machine configured with this backend .
> > machine ip : 10.35.3.109
> > user : root   pass: qum5net
> > 
> >  I run the command manually and got this error:
> 
> [root@test632 d80058dce1a9f0bd61fe7f15b093af65]# chown 165
> volume-4ddc2c23-47c9-402c-a386-1d03baedbd29
> chown: changing ownership of ‘volume-4ddc2c23-47c9-402c-a386-1d03baedbd29’:
> Invalid argument
> 


mount  :
[root@test632 d80058dce1a9f0bd61fe7f15b093af65]# mount | grep cinder
netapp.qa.lab.tlv.redhat.com:/vol/vol_rhos_auto_nfs on /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65 type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.35.3.109,local_lock=none,addr=10.35.64.11)
[r
> > 
> > Thanks , 
> > Benny

Comment 7 Eric Harney 2014-07-29 15:29:30 UTC
This is caused by the NetApp server's configuration.  It appears to have uid squashing enabled for NFSv4.  It squashes ownership of files to "nobody" instead of "cinder" or "root".

If you change cinder.conf to include
nfs_mount_options=nfsvers=3

it will use NFSv3 and work as expected.  You need to configure the NetApp machine correctly for NFSv4.  (I am not sure on the details of how to do this.)

Comment 8 bkopilov 2014-07-30 05:48:37 UTC
Hi , 
I add to cinder conf nfs_mount_options=nfsvers=3 , it works !
Thanks 
Benny

Comment 9 Sean Cohen 2014-07-30 15:01:46 UTC
With NFSv4, you have to configure user mapping or you end up with the nobody/nogroup situation.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-nfs.html

Closing the bug

Sean