Bug 1391551

Summary: Upgrade fails on convergence step: Services misconfigured to use sqlite connections
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Omri Hochman <ohochman>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: augol, dbecker, jslagle, lbezdick, mandreou, mburns, morazi, pkilambi, rhel-osp-director-maint, yprokule
Target Milestone: rc   
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-07 12:28:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Yurii Prokulevych 2016-11-03 14:21:24 UTC
Description of problem:
-----------------------
Upgrade from RHOS-9 to RHOS-10 fails during convergence step.

time openstack overcloud deploy \
        --templates \
        --libvirt-type kvm \
        --ntp-server clock.redhat.com \
        --neutron-network-type vxlan \
        --neutron-tunnel-types vxlan \
        --control-scale 3 \
        --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \
        --compute-scale 1 \
        --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
        --ceph-storage-scale 1 \
        --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 \
        -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
        -e /home/stack/virt/ceph.yaml \
        -e /home/stack/virt/network/network-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
        -e /home/stack/virt/enable-tls.yaml \
        -e /home/stack/virt/inject-trust-anchor.yaml \
        -e /home/stack/virt/hostnames.yml \
        -e /home/stack/virt/debug.yaml \
        -e /home/stack/virt/gnocchi_swift.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml

Excerpt from failed deployment:
...
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.416 7245 WARNING stevedore.named [-] Could not load sqlite
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 CRITICAL gnocchi [-] NoMatches: No 'gnocchi.indexer' driver found, looking for 'sqlite'
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi Traceback (most recent call last):
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi   File "/usr/bin/gnocchi-upgrade", line 10, in <module>
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi     sys.exit(upgrade())
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 55, in upgrade
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi     index = indexer.get_driver(conf)
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/indexer/__init__.py", line 78, in get_driver
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi     split.scheme).driver
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 46, in __init__
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi     verify_requirements=verify_requirements,
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 77, in __init__
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi     self._init_plugins(extensions)
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 98, in _init_plugins
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi     (self.namespace, name))
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi NoMatches: No 'gnocchi.indexer' driver found, looking for 'sqlite'
Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]/returns: 2016-11-03 12:55:00.417 7245 ERROR gnocchi 

Indeed gnocchi/glance-registry/aodh... are misconfigured to use 'default' connection types:

connection = sqlite:////var/lib/aodh/aodh.sqlite
connection = sqlite:///var/lib/glance/glance.sqlite
url = sqlite:////var/lib/gnocchi/gnocchi.sqlite


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-tripleo-image-elements-5.0.0-2.el7ost.noarch
puppet-tripleo-5.3.0-4.el7ost.noarch
openstack-tripleo-puppet-elements-5.0.0-1.el7ost.noarch
openstack-tripleo-validations-5.1.0-1.el7ost.noarch
openstack-tripleo-common-5.3.0-3.el7ost.noarch
python-tripleoclient-5.3.0-4.el7ost.noarch
openstack-tripleo-ui-1.0.4-2.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-1.2.el7ost.noarch
openstack-tripleo-heat-templates-compat-2.0.0-34.3.el7ost.noarch

Comment 2 Lukas Bezdicka 2016-11-03 17:59:50 UTC
gnocchi::db::database_connection: ://gnocchi:PjDXcvZCjw934DQk8qHnUzTMg@/gnocchi instead of
gnocchi::db::database_connection: mysql+pymysql://gnocchi:PjDXcvZCjw934DQk8qHnUzTMg@/gnocchi

Comment 3 Yurii Prokulevych 2016-11-04 14:38:42 UTC
As discussed on irc possible culprit might be enable-tls.yaml missing MysqlInternal record for EndpointMap paramater.

Comment 4 Marios Andreou 2016-11-07 10:57:31 UTC
@yprokule hey o/ any update here? Are we reproducing this or was it indeed environmental as we discussed on friday scrum?

Please update asap, I'd love to either point at a fix here (and move to POST) or close this as notabug.

Comment 5 Yurii Prokulevych 2016-11-07 11:21:19 UTC
Hi Marios,

Indeed it looks like misconfiguration on my side, cause since passing '/usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml' to upgrade command explicitly eliminates the issue.

---
Yurii

Comment 6 Marios Andreou 2016-11-07 12:28:14 UTC
Awesome thanks for the update Yurii closing for now we can revisit if necessary.