Created attachment 918115 [details] logs - /var/log and /etc Description of problem: openstack 5.0 , rhel 7 with nfs 7_mode backend. Automation runs tempest code . Tempest fails on : tempest.api.volume.admin.test_volumes_backup.VolumesBackupsTest.test_volume_backup_create_get_detailed_list_restore_delete From the cinder/backup code we see problem when trying to chown of mounted path we got error . Looks like the mount path is not valid . 2014-07-15 04:02:41.250 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f of=/dev/null count=1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:142 2014-07-15 04:02:41.303 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:142 2014-07-15 04:02:41.677 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Result was 1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:167 2014-07-15 04:02:41.678 18424 DEBUG cinder.volume.driver [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] volume 71e32945-6d61-45f2-9c8e-05f207f4ee6f: removing export _detach_volume /usr/lib/python2.7/site-packages/cinder/volume/driver.py:456 2014-07-15 04:02:41.724 18424 ERROR oslo.messaging.rpc.dispatcher [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Exception during message handling: Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f Exit code: 1 Stdout: '' Stderr: '/bin/chown: changing ownership of \xe2\x80\x98/var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f\xe2\x80\x99: Invalid argument\n' 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 280, in create_backup 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher 'fail_reason': unicode(err)}) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__ 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 273, in create_backup 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher backup_service) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 498, in backup_volume 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher with utils.temporary_chown(volume_path): 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher return self.gen.next() 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 682, in temporary_chown 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher execute('chown', owner_uid, path, run_as_root=True) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 136, in execute 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py", line 173, in execute 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher cmd=' '.join(cmd)) 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command. 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Exit code: 1 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Stdout: '' 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher Stderr: '/bin/chown: changing ownership of \xe2\x80\x98/var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f\xe2\x80\x99: Invalid argument\n' 2014-07-15 04:02:41.724 18424 TRACE oslo.messaging.rpc.dispatcher 2014-07-15 04:02:41.759 18424 INFO oslo.messaging._drivers.impl_rabbit [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Connected to AMQP server on 192.168.2.22:5672 2014-07-15 04:02:41.759 18424 ERROR oslo.messaging._drivers.common [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Returning exception Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f Exit code: 1 Stdout: '' Version-Release number of selected component (if applicable): How reproducible: Run tempest tests on nfs_7mode backend automation test: http://jenkins.rhev.lab.eng.brq.redhat.com:8080/view/RHOS-STORAGE-QE/view/RHOS-5.0-RHEL7.0/job/rhos-5-rhel-7.0-cinder-netapp_nfs_7mode/75/ Attaching to bug all logs for cinder / config . Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This command looks ok: sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f Can you try just running "chown 165 <file>" for some file on that mount point by hand? I think this may be an NFS configuration error related to id mapping. Note the second of these three messages that occurs when the chown is performed: Jul 15 04:02:41 test9283 cinder-backup: 2014-07-15 04:02:41.303 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 165 /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65/volume-71e32945-6d61-45f2-9c8e-05f207f4ee6f execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:142 Jul 15 04:02:41 test9283 kernel: NFS: v4 server netapp.qa.lab.tlv.redhat.com does not accept raw uid/gids. Reenabling the idmapper. Jul 15 04:02:41 test9283 cinder-backup: 2014-07-15 04:02:41.677 18424 DEBUG cinder.openstack.common.processutils [req-b94d9a60-38d8-4667-af62-932a74b35d6a 54919d66f28a47d19e19a45c45e271d1 314c9de5d5a44833995e37ef2fd87c20 - - -] Result was 1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:167 If this happens when trying by hand we will need to look at the NFS server configuration, it isn't clear that this is an issue in Cinder itself.
Hi , I installed baremetal machine configured with this backend . machine ip : 10.35.3.109 user : root pass: qum5net I am checking this issue now , you can access to the machine if you like . Thanks , Benny
(In reply to bkopilov from comment #4) > Hi , > I installed baremetal machine configured with this backend . > machine ip : 10.35.3.109 > user : root pass: qum5net > > I run the command manually and got this error: [root@test632 d80058dce1a9f0bd61fe7f15b093af65]# chown 165 volume-4ddc2c23-47c9-402c-a386-1d03baedbd29 chown: changing ownership of ‘volume-4ddc2c23-47c9-402c-a386-1d03baedbd29’: Invalid argument > > Thanks , > Benny
(In reply to bkopilov from comment #5) > (In reply to bkopilov from comment #4) > > Hi , > > I installed baremetal machine configured with this backend . > > machine ip : 10.35.3.109 > > user : root pass: qum5net > > > > I run the command manually and got this error: > > [root@test632 d80058dce1a9f0bd61fe7f15b093af65]# chown 165 > volume-4ddc2c23-47c9-402c-a386-1d03baedbd29 > chown: changing ownership of ‘volume-4ddc2c23-47c9-402c-a386-1d03baedbd29’: > Invalid argument > mount : [root@test632 d80058dce1a9f0bd61fe7f15b093af65]# mount | grep cinder netapp.qa.lab.tlv.redhat.com:/vol/vol_rhos_auto_nfs on /var/lib/cinder/mnt/d80058dce1a9f0bd61fe7f15b093af65 type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.35.3.109,local_lock=none,addr=10.35.64.11) [r > > > > Thanks , > > Benny
This is caused by the NetApp server's configuration. It appears to have uid squashing enabled for NFSv4. It squashes ownership of files to "nobody" instead of "cinder" or "root". If you change cinder.conf to include nfs_mount_options=nfsvers=3 it will use NFSv3 and work as expected. You need to configure the NetApp machine correctly for NFSv4. (I am not sure on the details of how to do this.)
Hi , I add to cinder conf nfs_mount_options=nfsvers=3 , it works ! Thanks Benny
With NFSv4, you have to configure user mapping or you end up with the nobody/nogroup situation. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-nfs.html Closing the bug Sean