Bug 902871 - keystone-manage db_sync fails
Summary: keystone-manage db_sync fails
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: openstack-packstack
Version: el6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Derek Higgins
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-01-22 15:15 UTC by Sandro Mathys
Modified: 2013-01-23 09:49 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2013-01-23 09:49:16 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
debug log (for comment 2) (98.62 KB, text/plain)
2013-01-23 08:13 UTC, Sandro Mathys
no flags Details
mysql.pp.log (for comment 5) (4.77 KB, text/plain)
2013-01-23 08:43 UTC, Sandro Mathys
no flags Details

Description Sandro Mathys 2013-01-22 15:15:06 UTC
Description of problem:
err: /Stage[main]/Keystone/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0] at /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/keystone/manifests/init.pp:173


Version-Release number of selected component (if applicable):
openstack-packstack-2012.2.2-0.5.dev318.el6.noarch


How reproducible:
Unknown

Steps to Reproduce:
1. Run packstack with an answer file
2. Watch it fail :)
  
Additional info:
Everything seems to work fine until:
ERROR:root:Error  during remote puppet apply of /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/manifests/10.112.1.29_keystone.pp
ERROR:root:warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 142 in file /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/nova/manifests/init.pp
warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 152 in file /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/nova/manifests/init.pp
warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 161 in file /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/nova/manifests/init.pp
[...]some positive notices[/...]
err: /Stage[main]/Keystone/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0] at /var/tmp/2d0e042a-0173-4bb7-901e-1eb339feb4e9/modules/keystone/manifests/init.pp:173

After that, obviously all keystone commands fail.

ans.txt:
CONFIG_GLANCE_INSTALL=y
CONFIG_CINDER_INSTALL=n
CONFIG_NOVA_INSTALL=y
CONFIG_HORIZON_INSTALL=y
CONFIG_SWIFT_INSTALL=n
CONFIG_CLIENT_INSTALL=y
CONFIG_SSH_KEY=/root/.ssh/id_rsa.pub
CONFIG_MYSQL_HOST=10.112.1.29
CONFIG_MYSQL_USER=root
CONFIG_MYSQL_PW=***
CONFIG_QPID_HOST=10.112.1.29
CONFIG_KEYSTONE_HOST=10.112.1.29
CONFIG_KEYSTONE_ADMINTOKEN=***
CONFIG_KEYSTONE_ADMINPASSWD=***
CONFIG_GLANCE_HOST=10.112.1.29
CONFIG_CINDER_HOST=10.112.1.29
CONFIG_NOVA_API_HOST=10.112.1.29
CONFIG_NOVA_CERT_HOST=10.112.1.29
CONFIG_NOVA_VNCPROXY_HOST=10.112.1.29
CONFIG_NOVA_COMPUTE_HOSTS=10.112.1.28
CONFIG_NOVA_COMPUTE_PRIVIF=em2.1101
CONFIG_NOVA_NETWORK_HOST=10.112.1.29
CONFIG_NOVA_NETWORK_PUBIF=eth1
CONFIG_NOVA_NETWORK_PRIVIF=eth2
CONFIG_NOVA_NETWORK_FIXEDRANGE=192.168.32.0/22
CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.4.0/22
CONFIG_NOVA_SCHED_HOST=10.112.1.29
CONFIG_OSCLIENT_HOST=10.112.1.29
CONFIG_HORIZON_HOST=10.112.1.29
CONFIG_SWIFT_PROXY_HOSTS=10.112.1.29
CONFIG_SWIFT_STORAGE_HOSTS=10.112.1.29
CONFIG_SWIFT_STORAGE_ZONES=1
CONFIG_SWIFT_STORAGE_REPLICAS=1
CONFIG_SWIFT_STORAGE_FSTYPE=ext4
CONFIG_USE_EPEL=y
CONFIG_REPO=
CONFIG_RH_USERNAME=
CONFIG_RH_PASSWORD=

Repos:
- RHEL 6.4 Beta
- EPEL 6
- Grizzly Side Repo

Comment 1 Martin Magr 2013-01-22 16:55:46 UTC
Could you please run packstack with --debug option and upload logs? Also please paste here versions of openstack* packages.

Comment 2 Sandro Mathys 2013-01-23 08:12:35 UTC
I now also activated EPEL-Testing. The primary host (the one packstack is executed on) is a fresh install, the other node has been left in the state from the last test.

# rpm -qa | grep openstack
openstack-utils-2013.1-1.el6.noarch
openstack-packstack-2012.2.2-0.5.dev318.el6.noarch
openstack-keystone-2013.1-0.2.g2.el6.noarch
openstack-glance-2013.1-0.2.g2.el6.noarch

Also, uploading debug log.

Comment 3 Sandro Mathys 2013-01-23 08:13:22 UTC
Created attachment 685698 [details]
debug log (for comment 2)

Comment 4 Derek Higgins 2013-01-23 08:32:13 UTC
Hi Sandro,
    If your up for it can you try running the command below(as root) on the other node (where openstack is being installed to), 

NOTE : this command wipes lots of things off the server including mysql databases, running vm's and data in various places, so only run it if your sure you don't have anything to loose on the server.

yum remove -y "*openstack*" "*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" mysql mysql-server httpd  perl-DBI perl-DBD-MySQL ; ps -ef | grep -i repli | grep swift | awk '{print $2}' | xargs kill ; rm -rf /root/.my.cnf /var/lib/mysql/ /var/lib/nova /etc/nova /etc/swift /srv/node/device*/* ; umount /srv/node/device* ; killall dnsmasq ; for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do virsh destroy $x ; virsh undefine $x ; done ; setenforce 1

Also can you verify you have the file /etc/my.cnf if not remove and re-install the package mysql-libs

Finally on your packstack server you should have a file named
/var/tmp/16b33222-b8dd-418c-b977-d30d208e68b3/manifests/10.112.1.29_mysql.pp.log
can you see if it contains any errors?

Note: this version of packstack is intended to work with openstack folsom (and not tested on grizzly) but some of the suggestions above may help progress.

thanks,
Derek.

Comment 5 Sandro Mathys 2013-01-23 08:42:35 UTC
Sure, can go and try that. Since it's a very fresh install and is not working at all like that, I don't have much to lose ;)

*mysql.pp.log shows one error, that sure worries me some:

err: Could not prefetch database_grant provider 'mysql': Command mysql is missing

Figure some kind of dependency is missing there in puppet. Going to attach that log, might be helpful.

Comment 6 Sandro Mathys 2013-01-23 08:43:21 UTC
Created attachment 685704 [details]
mysql.pp.log (for comment 5)

Comment 7 Derek Higgins 2013-01-23 08:49:15 UTC
Thanks.

(In reply to comment #5)
> *mysql.pp.log shows one error, that sure worries me some:
> 
> err: Could not prefetch database_grant provider 'mysql': Command mysql is
> missing

thats an expected message, and shouldn't be causing any problems

Comment 8 Sandro Mathys 2013-01-23 09:03:19 UTC
I see. Executing the remove-everything cmd and trying again didn't change anything, unsurprisingly.

I know it supports Folsom only at this point but wanted to give it a try with Grizzly (it should work one day, right?)...but might as well go and see if it works with Folsom or if I can reproduce the issue there as well.

Comment 9 Sandro Mathys 2013-01-23 09:34:51 UTC
Can't reproduce with Folsom so this bug can wait until Grizzly is going to be supported... ;)

Comment 10 Derek Higgins 2013-01-23 09:49:16 UTC
Thanks Sandro, yes grizzly will be supported some day, I'm going to close this as DEFERRED, but will try to reproduce it to see if I can find out what is going wrong. If I get anywhere I'll let you know.


Note You need to log in before you can comment on or make changes to this bug.