Bug 996048 - [RHS-RHOS] "openstack-cinder-volume dead but pid file exists" when restarted openstack-cinder-volume with glusterfs_disk_util as "du" in cinder.conf
[RHS-RHOS] "openstack-cinder-volume dead but pid file exists" when restarted ...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity urgent
: beta
: 4.0
Assigned To: Eric Harney
Dafna Ron
storage
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-12 06:04 EDT by Gowrishankar Rajaiyan
Modified: 2016-04-26 22:25 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Prior to this update, the 'du' command was missing as an allowed command from Block Storage rootwrap configuration. Consequently, the GlusterFS driver would not function when a user set glusterfs_disk_util="du" in cinder.conf. This update adds the 'du' command to Block Storage's rootwrap configuration, and the glusterfs_disk_util option is now usable. Please also note that this option will be deprecated in a future release.
Story Points: ---
Clone Of:
Environment:
virt rhos cinder rhs integration
Last Closed: 2013-12-19 19:19:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 26032 None None None Never

  None (edit)
Description Gowrishankar Rajaiyan 2013-08-12 06:04:50 EDT
Description of problem: As per http://docs.openstack.org/trunk/openstack-block-storage/admin/content/GlusterFS-driver.html description for "glusterfs_disk_util" use "Use du or df for free space calculation." and the default being "df". However, when you provide "du" as a config option and restart cinder volume services, the process fails to start with "openstack-cinder-volume dead but pid file exists" and traceback messages getting logged in /var/log/cinder/volume.log.


Version-Release number of selected component (if applicable):
openstack-cinder-2013.1.2-3.el6ost.noarch
glusterfs-3.4.0.18rhs-1.el6_4.x86_64

How reproducible: Always


Steps to Reproduce:
1. Create a 6X2 distribute replicate volume, using 4 RHS servers and 3 bricks per RHS server
   (i.e) gluster volume create cinder-vol replica 2 <brick1> .... <brick12>

2. Tag this volume with group virt
   (i.e) gluster volume set cinder-vol group virt

3. Set storage.owner-uid & storage.owner-gid to 165:165
   (i.e) gluster volume set cinder.vol storage.owner-uid 165
         gluster volume set cinder.vol storage.owner-gid 165

4. Configure cinder to use the above created gluster volume
   This can be done by editing /etc/cinder/cinder.conf file
   (i.e) openstack-config --set /etc/cinder/cinder.conf volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver
         openstack-config --set /etc/cinder/cinder.conf glusterfs_shares_config /etc/cinder/shares.conf
         openstack-config --set /etc/cinder/cinder.conf glusterfs_mount_point_base /var/lib/cinder/images

5. Add an volume entry for the volume created in step 1 to /etc/cinder/shares.conf. The content of this file should be,
    10.70.43.44:cinder-vol
   
6. Restart Cinder services

7. See to that gluster volumes are mounted automatically using mount command

8. Create a cinder volume now. 

9. Check the cinder volume
    (i.e) cinder list
    Here you can see the status of cinder volume as "available"

10. In /etc/cinder/cinder.conf, change the default value of glusterfs_disk_util from "df" to "du".

11. Restart Cinder services.

12. Observe /var/log/cinder/volume.log and the status of openstack-cinder-volume service.


Actual results:
1. # service openstack-cinder-volume status
openstack-cinder-volume dead but pid file exists

2. ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
Exit code: 99


Expected results: Cinder services should restart successfully with no tracebacks.


Additional info:
==> /var/log/cinder/volume.log <==
2013-08-12 09:25:12     INFO [cinder.service] Starting 1 workers
2013-08-12 09:25:12     INFO [cinder.service] Started child 2961
2013-08-12 09:25:12    AUDIT [cinder.service] Starting cinder-volume node (version 2013.1.2)
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): mount.glusterfs
2013-08-12 09:25:12    DEBUG [cinder.utils] Result was 1
2013-08-12 09:25:12    DEBUG [cinder.utils] backend <module 'cinder.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.pyc'>
2013-08-12 09:25:12    DEBUG [cinder.volume.manager] Re-exporting 2 volumes
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.44:cinder-vol /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:12    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.44:cinder-vol /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.volume.manager] Resuming any in progress delete operations
2013-08-12 09:25:13     INFO [cinder.volume.manager] Updating volume status
2013-08-12 09:25:13    DEBUG [cinder.volume.drivers.nfs] shares loaded: {'10.70.43.44:cinder-vol': '-o backupvolfile-server=10.70.43.47,fetch-attempts=3,acl,selinux', '10.70.43.76:cinder-vol3': '-o selinux'}
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.44:cinder-vol /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd -o backupvolfile-server=10.70.43.47,fetch-attempts=3,acl,selinux
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/b4fba7359b52d3bd7483f7493451ee7d
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.43.76:cinder-vol3 /var/lib/cinder/volumes/b4fba7359b52d3bd7483f7493451ee7d -o selinux
2013-08-12 09:25:13    DEBUG [cinder.volume.drivers.glusterfs] Available shares ['10.70.43.44:cinder-vol', '10.70.43.76:cinder-vol3']
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf df --portability --block-size 1 /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
2013-08-12 09:25:13    DEBUG [cinder.utils] Result was 99
2013-08-12 09:25:13    ERROR [cinder.service] Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 227, in _start_child
    self._child_process(wrap.server)
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 204, in _child_process
    launcher.run_server(server)
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 95, in run_server
    server.start()
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 355, in start
    self.manager.init_host()
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 160, in init_host
    self.publish_service_capabilities(ctxt)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 705, in publish_service_capabilities
    self._report_driver_status(context)
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 697, in _report_driver_status
    volume_stats = self.driver.get_volume_stats(refresh=True)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 268, in get_volume_stats
    self._update_volume_stats()
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 287, in _update_volume_stats
    free, capacity = self._get_available_capacity(nfs_share)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 239, in _get_available_capacity
    run_as_root=True)
  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd
Exit code: 99
Stdout: '/usr/bin/cinder-rootwrap: Unauthorized command: du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/volumes/1d12e17a168a458a2db39ca37ee302fd (no filter matched)\n'
Stderr: ''
2013-08-12 09:25:13     INFO [cinder.service] Child 2961 exited with status 2
2013-08-12 09:25:13     INFO [cinder.service] _wait_child 1
2013-08-12 09:25:13     INFO [cinder.service] wait wrap.failed True
^C
[root@rhs-hpc-srv1 ~]#
Comment 2 Eric Harney 2013-08-12 08:23:52 EDT
"du" is missing from the rootwrap configuration in RHOS 3, it was added for Havana.  It may be possible to workaround this for RHOS 3 by adding
"du: CommandFilter, du, root"
to your a local rootwrap config file in /etc/cinder/rootwrap.d/, but I need to verify this.

Will look into a fix so that this error doesn't halt the volume service.
Comment 4 Divya 2013-09-10 08:07:03 EDT
Eric,

This bug has been identified as a known issue for Big Bend release. Please provide CCFR information in the Doc Text field.
Comment 5 Dafna Ron 2013-10-21 06:52:26 EDT
verified on openstack-cinder-2013.2-0.11.rc1.el6ost.noarch
Comment 7 Scott Lewis 2013-12-16 13:56:38 EST
Auto qa_ack+ for VERIFIED bugs so they can be added to RC erratum
Comment 15 errata-xmlrpc 2013-12-19 19:19:03 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.