Bug 1251144 - openstack-cinder-volume service fails to start on the Blockstorage-Node because of missing DB password in the cinder.conf
openstack-cinder-volume service fails to start on the Blockstorage-Node becau...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
x86_64 Linux
urgent Severity urgent
: rc
: 10.0 (Newton)
Assigned To: Giulio Fidente
lkuchlan
: Triaged
: 1251143 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-06 10:37 EDT by Omri Hochman
Modified: 2016-12-14 10:14 EST (History)
10 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-14 10:14:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 322858 None None None 2016-06-23 04:25 EDT

  None (edit)
Description Omri Hochman 2015-08-06 10:37:32 EDT
openstack-cinder-volume service fails to start on the Blockstorage-Node because of missing DB password in the cinder.conf  

Environment: 
-------------
instack-0.0.7-1.el7ost.noarch
instack-undercloud-2.1.2-22.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-puppet-modules-2015.1.8-8.el7ost.noarch

openstack-cinder-2015.1.0-3.el7ost.noarch

== Cinder services ==
openstack-cinder-api:                   inactive  (disabled on boot)
openstack-cinder-scheduler:             inactive  (disabled on boot)
openstack-cinder-volume:                failed
openstack-cinder-backup:                inactive  (disabled on boot)


messages (from blockstorage-node) :
-----------
Aug  5 19:38:15 localhost os-collect-config: ++ os-apply-config --key completion-handle --type raw --key-default ''
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 CRITICAL cinder [-] OperationalError: (_mysql_exceptions.OperationalError) (1045, "Acce
ss denied for user 'cinder'@'10.19.104.12' (using password: YES)")
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder Traceback (most recent call last):
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/bin/cinder-volume", line 10, in <module>
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     sys.exit(main())
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 71, i
n main
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     binary='cinder-volume')
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 249, in 
create
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     service_name=service_name)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 129, in 
__init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     *args, **kwargs)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1
95, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     *args, **kwargs)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 130, in 
__init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     super(SchedulerDependentManager, self).__init__(host, db_driver)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 80, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     super(Manager, self).__init__(db_driver)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/db/base.py", line 42, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     self.db.dispose_engine()
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 80, in dispose_engine
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     if 'sqlite' not in IMPL.get_engine().name:
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 85, in get_engine
..
...
..
..
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     return Connection(*args, **kwargs)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder     super(Connection, self).__init__(*args, **kwargs2)
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'cinder'@'10.19.104.12' (using password: YES)")
Aug  5 19:38:15 localhost cinder-volume: 2015-08-05 19:38:15.260 15450 TRACE cinder
Aug  5 19:38:15 localhost systemd: openstack-cinder-volume.service: main process exited, code=exited, status=1/FAILURE
Aug  5 19:38:15 localhost systemd: Unit openstack-cinder-volume.service entered failed state.



diff cinder.conf from controller-node VS blockstorage-node:
------------------------------------------------------------
148d147
< osapi_volume_listen=10.19.104.12
157d155
< osapi_volume_workers=12
385a384
> swift_catalog_info=object-store:swift:internalURL
626d624
< auth_strategy=keystone
632c630
< enabled_backends=tripleo_ceph
---
> enabled_backends=tripleo_iscsi
680a679
> nova_catalog_info=compute:Compute Service:internalURL
842d840
< scheduler_driver=cinder.scheduler.filter_scheduler.FilterScheduler
2385c2383
< connection=mysql://cinder:fcf47825be3fa35c9999d386f387c8d426982be1@10.19.104.10/cinder
---
> connection=mysql://cinder:unset@10.19.104.10/cinder
2862,2871c2860,2867
< [tripleo_ceph]
< host=hostgroup
< volume_driver=cinder.volume.drivers.rbd.RBDDriver
< rbd_flatten_volume_from_snapshot=False
< rbd_max_clone_depth=5
< rbd_pool=volumes
< rbd_secret_uuid=1133fc6e-3bb5-11e5-a6ae-c81f66c883ca
< volume_backend_name=tripleo_ceph
< rbd_user=openstack
< rbd_ceph_conf=/etc/ceph/ceph.conf
---
> [tripleo_iscsi]
> volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver
> volumes_dir=/var/lib/cinder/volumes
> iscsi_protocol=iscsi
> iscsi_ip_address=10.19.105.16
> volume_backend_name=tripleo_iscsi
> volume_group=cinder-volumes
> iscsi_helper=lioadm
Comment 3 Omri Hochman 2015-08-06 12:42:52 EDT
*** Bug 1251143 has been marked as a duplicate of this bug. ***
Comment 4 Omri Hochman 2015-08-06 12:47:14 EDT
To reproduce : attempt to deploy (with :  --block-storage-scale 1 --swift-storage-scale 1 ) 

 openstack overcloud deploy  --plan overcloud --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 1 --swift-storage-scale 1 /home/stack/network-environment.yaml --ntp-server 10.5.26.10 --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout 90 



(*) ssh the blockstorage-node and check that all cinder services are up .
Comment 6 Omri Hochman 2015-08-06 16:16:11 EDT
(In reply to Omri Hochman from comment #4)
> To reproduce : attempt to deploy (with :  --block-storage-scale 1
> --swift-storage-scale 1 ) 
> 

Correction : 
------------
to reproduce run deployment with:  --block-storage-scale 1 

( *No* need to deploy --swift-storage-scale 1 )
Comment 10 Mike Burns 2016-04-07 16:47:27 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 13 lkuchlan 2016-11-10 01:51:34 EST
Tested using:
openstack-tripleo-heat-templates-5.0.0-1.4.el7ost.noarch

Verification flow:

Deployment:
openstack overcloud deploy --templates --compute-scale 1 --block-storage-scale 1 --ntp-server clock.redhat.com
--timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --ceph-storage-scale 0

[root@overcloud-controller-0 ~]# systemctl -a | grep cinder
  openstack-cinder-api.service                          loaded    active   running   OpenStack Cinder API Server
  openstack-cinder-scheduler.service                    loaded    active   running   OpenStack Cinder Scheduler Server
  openstack-cinder-volume.service                       loaded    active   running   Cluster Controlled openstack-cinder-volume
Comment 15 errata-xmlrpc 2016-12-14 10:14:32 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html

Note You need to log in before you can comment on or make changes to this bug.