Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1251144

Summary:	openstack-cinder-volume service fails to start on the Blockstorage-Node because of missing DB password in the cinder.conf
Product:	Red Hat OpenStack	Reporter:	Omri Hochman <ohochman>
Component:	openstack-tripleo-heat-templates	Assignee:	Giulio Fidente <gfidente>
Status:	CLOSED ERRATA	QA Contact:	lkuchlan <lkuchlan>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	7.0 (Kilo)	CC:	dbecker, hbrock, jcoufal, jschluet, lkuchlan, mburns, mlopes, nbarcet, rhel-osp-director-maint, srevivo
Target Milestone:	rc	Keywords:	Triaged
Target Release:	10.0 (Newton)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 15:14:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Omri Hochman 2015-08-06 14:37:32 UTC

openstack-cinder-volume service fails to start on the Blockstorage-Node because of missing DB password in the cinder.conf  

Environment: 
-------------
instack-0.0.7-1.el7ost.noarch
instack-undercloud-2.1.2-22.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-puppet-modules-2015.1.8-8.el7ost.noarch

openstack-cinder-2015.1.0-3.el7ost.noarch

== Cinder services ==
openstack-cinder-api:                   inactive  (disabled on boot)
openstack-cinder-scheduler:             inactive  (disabled on boot)
openstack-cinder-volume:                failed
openstack-cinder-backup:                inactive  (disabled on boot)


messages (from blockstorage-node) :
-----------
Aug  5 19:38:15 localhost os-collect-config: ++ os-apply-config --key completion-handle --type raw --key-default ''
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 CRITICAL cinder [-] OperationalError: (_mysql_exceptions.OperationalError) (1045, "Acce
ss denied for user 'cinder'@'10.19.104.12' (using password: YES)")
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder Traceback (most recent call last):
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/bin/cinder-volume", line 10, in <module>
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     sys.exit(main())
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 71, i
n main
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     binary='cinder-volume')
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 249, in 
create
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     service_name=service_name)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 129, in 
__init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     *args, **kwargs)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1
95, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     *args, **kwargs)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 130, in 
__init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     super(SchedulerDependentManager, self).__init__(host, db_driver)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 80, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     super(Manager, self).__init__(db_driver)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/db/base.py", line 42, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     self.db.dispose_engine()
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 80, in dispose_engine
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     if 'sqlite' not in IMPL.get_engine().name:
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 85, in get_engine
..
...
..
..
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     return Connection(*args, **kwargs)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     super(Connection, self).__init__(*args, **kwargs2)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'cinder'@'10.19.104.12' (using password: YES)")
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder
Aug  5 19:38:15 localhost systemd: openstack-cinder-volume.service: main process exited, code=exited, status=1/FAILURE
Aug  5 19:38:15 localhost systemd: Unit openstack-cinder-volume.service entered failed state.



diff cinder.conf from controller-node VS blockstorage-node:
------------------------------------------------------------
148d147
< osapi_volume_listen=10.19.104.12
157d155
< osapi_volume_workers=12
385a384
> swift_catalog_info=object-store:swift:internalURL
626d624
< auth_strategy=keystone
632c630
< enabled_backends=tripleo_ceph
---
> enabled_backends=tripleo_iscsi
680a679
> nova_catalog_info=compute:Compute Service:internalURL
842d840
< scheduler_driver=cinder.scheduler.filter_scheduler.FilterScheduler
2385c2383
< connection=mysql://cinder:fcf47825be3fa35c9999d386f387c8d426982be1.104.10/cinder
---
> connection=mysql://cinder:unset.104.10/cinder
2862,2871c2860,2867
< [tripleo_ceph]
< host=hostgroup
< volume_driver=cinder.volume.drivers.rbd.RBDDriver
< rbd_flatten_volume_from_snapshot=False
< rbd_max_clone_depth=5
< rbd_pool=volumes
< rbd_secret_uuid=1133fc6e-3bb5-11e5-a6ae-c81f66c883ca
< volume_backend_name=tripleo_ceph
< rbd_user=openstack
< rbd_ceph_conf=/etc/ceph/ceph.conf
---
> [tripleo_iscsi]
> volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
> volumes_dir=/var/lib/cinder/volumes
> iscsi_protocol=iscsi
> iscsi_ip_address=10.19.105.16
> volume_backend_name=tripleo_iscsi
> volume_group=cinder-volumes
> iscsi_helper=lioadm

Comment 3 Omri Hochman 2015-08-06 16:42:52 UTC

*** Bug 1251143 has been marked as a duplicate of this bug. ***

Comment 4 Omri Hochman 2015-08-06 16:47:14 UTC

To reproduce : attempt to deploy (with :  --block-storage-scale 1 --swift-storage-scale 1 ) 

 openstack overcloud deploy  --plan overcloud --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 1 --swift-storage-scale 1 /home/stack/network-environment.yaml --ntp-server 10.5.26.10 --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout 90 



(*) ssh the blockstorage-node and check that all cinder services are up .

Comment 6 Omri Hochman 2015-08-06 20:16:11 UTC

(In reply to Omri Hochman from comment #4)
> To reproduce : attempt to deploy (with :  --block-storage-scale 1
> --swift-storage-scale 1 ) 
> 

Correction : 
------------
to reproduce run deployment with:  --block-storage-scale 1 

( *No* need to deploy --swift-storage-scale 1 )

Comment 10 Mike Burns 2016-04-07 20:47:27 UTC

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 13 lkuchlan 2016-11-10 06:51:34 UTC

Tested using:
openstack-tripleo-heat-templates-5.0.0-1.4.el7ost.noarch

Verification flow:

Deployment:
openstack overcloud deploy --templates --compute-scale 1 --block-storage-scale 1 --ntp-server clock.redhat.com
--timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --ceph-storage-scale 0

[root@overcloud-controller-0 ~]# systemctl -a | grep cinder
  openstack-cinder-api.service                          loaded    active   running   OpenStack Cinder API Server
  openstack-cinder-scheduler.service                    loaded    active   running   OpenStack Cinder Scheduler Server
  openstack-cinder-volume.service                       loaded    active   running   Cluster Controlled openstack-cinder-volume

Comment 15 errata-xmlrpc 2016-12-14 15:14:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html