Bug 996048

Description of problem: As per http://docs.openstack.org/trunk/openstack-block-storage/admin/content/GlusterFS-driver.html description for "glusterfs_disk_util" use "Use du or df for free space calculation." and the default being "df". However, when you provide "du" as a config option and restart cinder volume services, the process fails to start with "openstack-cinder-volume dead but pid file exists" and traceback messages getting logged in /var/log/cinder/volume.log.


Version-Release number of selected component (if applicable):
openstack-cinder-2013.1.2-3.el6ost.noarch
glusterfs-3.4.0.18rhs-1.el6_4.x86_64

How reproducible: Always


Steps to Reproduce:
1. Create a 6X2 distribute replicate volume, using 4 RHS servers and 3 bricks per RHS server
   (i.e) gluster volume create cinder-vol replica 2 <brick1> .... <brick12>

2. Tag this volume with group virt
   (i.e) gluster volume set cinder-vol group virt

3. Set storage.owner-uid & storage.owner-gid to 165:165
   (i.e) gluster volume set cinder.vol storage.owner-uid 165
         gluster volume set cinder.vol storage.owner-gid 165

4. Configure cinder to use the above created gluster volume
   This can be done by editing /etc/cinder/cinder.conf file
   (i.e) openstack-config --set /etc/cinder/cinder.conf volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver
         openstack-config --set /etc/cinder/cinder.conf glusterfs_shares_config /etc/cinder/shares.conf
         openstack-config --set /etc/cinder/cinder.conf glusterfs_mount_point_base /var/lib/cinder/images

5. Add an volume entry for the volume created in step 1 to /etc/cinder/shares.conf. The content of this file should be,
    10.70.43.44:cinder-vol
   
6. Restart Cinder services

7. See to that gluster volumes are mounted automatically using mount command

8. Create a cinder volume now. 

9. Check the cinder volume
    (i.e) cinder list
    Here you can see the status of cinder volume as "available"

10. In /etc/cinder/cinder.conf, change the default value of glusterfs_disk_util from "df" to "du".

11. Restart Cinder services.

12. Observe /var/log/cinder/volume.log and the status of openstack-cinder-volume service.


Actual results:
1. # service openstack-cinder-volume status
openstack-cinder-volume dead but pid file exists

2. ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
Exit code: 99


Expected results: Cinder services should restart successfully with no tracebacks.


Additional info:
==> /var/log/cinder/volume.log <==
2013-08-12 09:25:12     INFO [cinder.service] Starting 1 workers
2013-08-12 09:25:12     INFO [cinder.service] Started child 2961
2013-08-12 09:25:12    AUDIT [cinder.service] Starting cinder-volume node (version 2013.1.2)
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): mount.glusterfs
2013-08-12 09:25:12    DEBUG [cinder.utils] Result was 1
2013-08-12 09:25:12    DEBUG [cinder.utils] backend <module 'cinder.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.pyc'>
2013-08-12 09:25:12    DEBUG [cinder.volume.manager] Re-exporting 2 volumes
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.44:cinder-vol /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.44:cinder-vol /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.volume.manager] Resuming any in progress delete operations
2013-08-12 09:25:13     INFO [cinder.volume.manager] Updating volume status
2013-08-12 09:25:13    DEBUG [cinder.volume.drivers.nfs] shares loaded: {'10.70.43.44:cinder-vol': '-o backupvolfile-server=10.70.43.47,fetch-attempts=3,acl,selinux', '10.70.43.76:cinder-vol3': '-o selinux'}
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.44:cinder-vol /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd -o backupvolfile-server=10.70.43.47,fetch-attempts=3,acl,selinux
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/b4fba7359b52d3bd7483f7493451ee7d
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.76:cinder-vol3 /var/lib/cinder/volumes/b4fba7359b52d3bd7483f7493451ee7d -o selinux
2013-08-12 09:25:13    DEBUG [cinder.volume.drivers.glusterfs] Available shares ['10.70.43.44:cinder-vol', '10.70.43.76:cinder-vol3']
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf df --portability --block-size 1 /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Result was 99
2013-08-12 09:25:13    ERROR [cinder.service] Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 227, in _start_child
    self._child_process(wrap.server)
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 204, in _child_process
    launcher.run_server(server)
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 95, in run_server
    server.start()
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 355, in start
    self.manager.init_host()
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 160, in init_host
    self.publish_service_capabilities(ctxt)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 705, in publish_service_capabilities
    self._report_driver_status(context)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 697, in _report_driver_status
    volume_stats = self.driver.get_volume_stats(refresh=True)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 268, in get_volume_stats
    self._update_volume_stats()
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 287, in _update_volume_stats
    free, capacity = self._get_available_capacity(nfs_share)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 239, in _get_available_capacity
    run_as_root=True)
  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
Exit code: 99
Stdout: '/usr/bin/cinder-rootwrap: Unauthorized command: du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd (no filter matched)\n'
Stderr: ''
2013-08-12 09:25:13     INFO [cinder.service] Child 2961 exited with status 2
2013-08-12 09:25:13     INFO [cinder.service] _wait_child 1
2013-08-12 09:25:13     INFO [cinder.service] wait wrap.failed True
^C
[root@rhs-hpc-srv1 ~]#