Created attachment 780536 [details]
Sosreport for openstack cinder

Description of problem: After updating the /etc/cinder/shares.conf with the new RHS volume credentials and restarting the openstack-cinder services, we see that still the old RHS volume credentails are being used while mounting the client. 


Version-Release number of selected component (if applicable):

Cinder: openstack-cinder-2013.1.2-3.el6ost.noarch
RHS: glusterfs-3.4.0.12rhs.beta6-1.el6rhs.x86_64


How reproducible: Consistently 

Consistently

Steps to Reproduce:
1. Create a new 2x2 Distributed-Replicate RHS volume

2. Umount old RHS cinder volume mount.

3. Updated /etc/cinder/shares.conf with the new RHS credentials
   # cat /etc/cinder/shares.conf
   10.70.37.49:cinder-vol

4. Restart cinder services,
   for i in api scheduler volume; do sudo service openstack-cinder-${i} start; done

Actual results:

openstack-cinder-volume tries mounting with the old RHS volume credentials.

The update RHS server is 10.70.37.49 which is what is mentioned in  /etc/cinder/shares.conf, but the mount command as seen in /var/log/cinder/volume.log is using the old server 10.70.37.66 to mount the RHS volume.

Expected results:

Should mount with the updated RHS volume server credentials

Additional info:
# gluster volume info cinder-vol
 
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: 50f8fa6d-50e6-4212-af3a-ef5b49cb3e94
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.37.49:/cinder4/s1
Brick2: 10.70.37.120:/cinder4/s1
Brick3: 10.70.37.132:/cinder4/s1
Brick4: 10.70.37.208:/cinder4/s1
Options Reconfigured:
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
storage.owner-gid: 165
storage.owner-uid: 165

#gluster peer status
Number of Peers: 3

Hostname: 10.70.37.120
Uuid: 2e35a7ec-6c4b-4942-8470-40649851dfdf
State: Peer in Cluster (Connected)

Hostname: 10.70.37.132
Uuid: d92d9ce8-4d1b-4d35-982b-5b25e55eafa1
State: Peer in Cluster (Connected)

Hostname: 10.70.37.208
Uuid: d7f6783a-d5db-453c-bd59-c5a4e8769935
State: Peer in Cluster (Connected)



# date
Tue Jul 30 12:53:12 IST 2013

# for i in api scheduler volume; do sudo service openstack-cinder-${i} start; done

#tail /var/log/cinder/volume.log

Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.37.66:cinder-vol /var/lib/cinder/volumes/cf55327cba40506e44b37f45f55af5e7
Exit code: 1
Stdout: 'Mount failed. Please check the log file for more details.\n'
Stderr: ''
2013-07-30 12:07:13     INFO [cinder.service] Child 30642 exited with status 2
2013-07-30 12:07:13     INFO [cinder.service] _wait_child 1
2013-07-30 12:07:13     INFO [cinder.service] wait wrap.failed True
2013-07-30 12:53:20     INFO [cinder.service] Starting 1 workers
2013-07-30 12:53:20     INFO [cinder.service] Started child 24501
2013-07-30 12:53:20    AUDIT [cinder.service] Starting cinder-volume node (version 2013.1.2)
2013-07-30 12:53:20    DEBUG [cinder.utils] Running cmd (subprocess): mount.glusterfs
2013-07-30 12:53:20    DEBUG [cinder.utils] Result was 1
2013-07-30 12:53:21    DEBUG [cinder.utils] backend <module 'cinder.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.pyc'>
2013-07-30 12:53:21    DEBUG [cinder.volume.manager] Re-exporting 5 volumes
2013-07-30 12:53:21    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat /var/lib/cinder/volumes/cf55327cba40506e44b37f45f55af5e7
2013-07-30 12:53:21    DEBUG [cinder.utils] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.37.66:cinder-vol /var/lib/cinder/volumes/cf55327cba40506e44b37f45f55af5e7
2013-07-30 12:53:24    DEBUG [cinder.utils] Result was 1
2013-07-30 12:53:24    ERROR [cinder.service] Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 227, in _start_child
    self._child_process(wrap.server)
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 204, in _child_process
    launcher.run_server(server)
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 95, in run_server
    server.start()
  File "/usr/lib/python2.6/site-packages/cinder/service.py", line 355, in start
    self.manager.init_host()
  File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 149, in init_host
    self.driver.ensure_export(ctxt, volume)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 128, in ensure_export
    self._ensure_share_mounted(volume['provider_location'])
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 189, in _ensure_share_mounted
    self._mount_glusterfs(glusterfs_share, mount_path, ensure=True)
  File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 256, in _mount_glusterfs
    self._execute(*command, run_as_root=True)
  File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 190, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.70.37.66:cinder-vol /var/lib/cinder/volumes/cf55327cba40506e44b37f45f55af5e7
Exit code: 1
Stdout: 'Mount failed. Please check the log file for more details.\n'
Stderr: ''
2013-07-30 12:53:24     INFO [cinder.service] Child 24501 exited with status 2
2013-07-30 12:53:24     INFO [cinder.service] _wait_child 1
2013-07-30 12:53:24     INFO [cinder.service] wait wrap.failed True