Bug 902871

Summary: keystone-manage db_sync fails
Product: [Fedora] Fedora EPEL Reporter: Sandro Mathys <sandro>
Component: openstack-packstackAssignee: Derek Higgins <derekh>
Status: CLOSED DEFERRED QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: el6CC: derekh, mmagr, p
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-23 09:49:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
debug log (for comment 2)
none
mysql.pp.log (for comment 5) none

Description Sandro Mathys 2013-01-22 15:15:06 UTC
Description of problem:
err: /Stage[main]/Keystone/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0] at /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/keystone/manifests/init.pp:173


Version-Release number of selected component (if applicable):
openstack-packstack-2012.2.2-0.5.dev318.el6.noarch


How reproducible:
Unknown

Steps to Reproduce:
1. Run packstack with an answer file
2. Watch it fail :)
  
Additional info:
Everything seems to work fine until:
ERROR:root:Error  during remote puppet apply of /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/manifests/10.112.1.29_keystone.pp
ERROR:root:warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 142 in file /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/nova/manifests/init.pp
warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 152 in file /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/nova/manifests/init.pp
warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 161 in file /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/nova/manifests/init.pp
[...]some positive notices[/...]
err: /Stage[main]/Keystone/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0] at /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/keystone/manifests/init.pp:173

After that, obviously all keystone commands fail.

ans.txt:
CONFIG_GLANCE_INSTALL=y
CONFIG_CINDER_INSTALL=n
CONFIG_NOVA_INSTALL=y
CONFIG_HORIZON_INSTALL=y
CONFIG_SWIFT_INSTALL=n
CONFIG_CLIENT_INSTALL=y
CONFIG_SSH_KEY=/root/.ssh/id_rsa.pub
CONFIG_MYSQL_HOST=10.112.1.29
CONFIG_MYSQL_USER=root
CONFIG_MYSQL_PW=***
CONFIG_QPID_HOST=10.112.1.29
CONFIG_KEYSTONE_HOST=10.112.1.29
CONFIG_KEYSTONE_ADMINTOKEN=***
CONFIG_KEYSTONE_ADMINPASSWD=***
CONFIG_GLANCE_HOST=10.112.1.29
CONFIG_CINDER_HOST=10.112.1.29
CONFIG_NOVA_API_HOST=10.112.1.29
CONFIG_NOVA_CERT_HOST=10.112.1.29
CONFIG_NOVA_VNCPROXY_HOST=10.112.1.29
CONFIG_NOVA_COMPUTE_HOSTS=10.112.1.28
CONFIG_NOVA_COMPUTE_PRIVIF=em2.1101
CONFIG_NOVA_NETWORK_HOST=10.112.1.29
CONFIG_NOVA_NETWORK_PUBIF=eth1
CONFIG_NOVA_NETWORK_PRIVIF=eth2
CONFIG_NOVA_NETWORK_FIXEDRANGE=192.168.32.0/22
CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.4.0/22
CONFIG_NOVA_SCHED_HOST=10.112.1.29
CONFIG_OSCLIENT_HOST=10.112.1.29
CONFIG_HORIZON_HOST=10.112.1.29
CONFIG_SWIFT_PROXY_HOSTS=10.112.1.29
CONFIG_SWIFT_STORAGE_HOSTS=10.112.1.29
CONFIG_SWIFT_STORAGE_ZONES=1
CONFIG_SWIFT_STORAGE_REPLICAS=1
CONFIG_SWIFT_STORAGE_FSTYPE=ext4
CONFIG_USE_EPEL=y
CONFIG_REPO=
CONFIG_RH_USERNAME=
CONFIG_RH_PASSWORD=

Repos:
- RHEL 6.4 Beta
- EPEL 6
- Grizzly Side Repo

Comment 1 Martin Magr 2013-01-22 16:55:46 UTC
Could you please run packstack with --debug option and upload logs? Also please paste here versions of openstack* packages.

Comment 2 Sandro Mathys 2013-01-23 08:12:35 UTC
I now also activated EPEL-Testing. The primary host (the one packstack is executed on) is a fresh install, the other node has been left in the state from the last test.

# rpm -qa | grep openstack
openstack-utils-2013.1-1.el6.noarch
openstack-packstack-2012.2.2-0.5.dev318.el6.noarch
openstack-keystone-2013.1-0.2.g2.el6.noarch
openstack-glance-2013.1-0.2.g2.el6.noarch

Also, uploading debug log.

Comment 3 Sandro Mathys 2013-01-23 08:13:22 UTC
Created attachment 685698 [details]
debug log (for comment 2)

Comment 4 Derek Higgins 2013-01-23 08:32:13 UTC
Hi Sandro,
    If your up for it can you try running the command below(as root) on the other node (where openstack is being installed to), 

NOTE : this command wipes lots of things off the server including mysql databases, running vm's and data in various places, so only run it if your sure you don't have anything to loose on the server.

yum remove -y "*openstack*" "*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" mysql mysql-server httpd  perl-DBI perl-DBD-MySQL ; ps -ef | grep -i repli | grep swift | awk '{print $2}' | xargs kill ; rm -rf /root/.my.cnf /var/lib/mysql/ /var/lib/nova /etc/nova /etc/swift /srv/node/device*/* ; umount /srv/node/device* ; killall dnsmasq ; for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do virsh destroy $x ; virsh undefine $x ; done ; setenforce 1

Also can you verify you have the file /etc/my.cnf if not remove and re-install the package mysql-libs

Finally on your packstack server you should have a file named
/var/tmp/16b33222-b8dd-418c-b977-d30d208e68b3/manifests/10.112.1.29_mysql.pp.log
can you see if it contains any errors?

Note: this version of packstack is intended to work with openstack folsom (and not tested on grizzly) but some of the suggestions above may help progress.

thanks,
Derek.

Comment 5 Sandro Mathys 2013-01-23 08:42:35 UTC
Sure, can go and try that. Since it's a very fresh install and is not working at all like that, I don't have much to lose ;)

*mysql.pp.log shows one error, that sure worries me some:

err: Could not prefetch database_grant provider 'mysql': Command mysql is missing

Figure some kind of dependency is missing there in puppet. Going to attach that log, might be helpful.

Comment 6 Sandro Mathys 2013-01-23 08:43:21 UTC
Created attachment 685704 [details]
mysql.pp.log (for comment 5)

Comment 7 Derek Higgins 2013-01-23 08:49:15 UTC
Thanks.

(In reply to comment #5)
> *mysql.pp.log shows one error, that sure worries me some:
> 
> err: Could not prefetch database_grant provider 'mysql': Command mysql is
> missing

thats an expected message, and shouldn't be causing any problems

Comment 8 Sandro Mathys 2013-01-23 09:03:19 UTC
I see. Executing the remove-everything cmd and trying again didn't change anything, unsurprisingly.

I know it supports Folsom only at this point but wanted to give it a try with Grizzly (it should work one day, right?)...but might as well go and see if it works with Folsom or if I can reproduce the issue there as well.

Comment 9 Sandro Mathys 2013-01-23 09:34:51 UTC
Can't reproduce with Folsom so this bug can wait until Grizzly is going to be supported... ;)

Comment 10 Derek Higgins 2013-01-23 09:49:16 UTC
Thanks Sandro, yes grizzly will be supported some day, I'm going to close this as DEFERRED, but will try to reproduce it to see if I can find out what is going wrong. If I get anywhere I'll let you know.