Bug 1275803

Summary: packstack --allinone fails on Fedora 22-3 during _keystone.pp
Product: [Community] RDO Reporter: Sergei <svorobyov>
Component: openstack-packstackAssignee: Martin Magr <mmagr>
Status: CLOSED EOL QA Contact: Shai Revivo <srevivo>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: ag, aortega, derekh, srevivo, svorobyov
Target Milestone: ---   
Target Release: Kilo   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-19 15:38:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sergei 2015-10-27 19:20:50 UTC
Description of problem:

packstack --allinone

fails with

192.168.1.158_mariadb.pp:                            [ DONE ]       
Applying 192.168.1.158_keystone.pp
Applying 192.168.1.158_glance.pp
Applying 192.168.1.158_cinder.pp
192.168.1.158_keystone.pp:                        [ ERROR ]          
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.1.158_keystone.pp
Error: Failed to apply catalog: Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Internal Server Error (HTTP 500)


Version-Release number of selected component (if applicable):

Used fresh installation of Fedora 22-3 Workstation.
Fetched the latest version of OpenStack as described on
https://www.rdoproject.org/install/quickstart/

How reproducible:

Just followed the steps in https://www.rdoproject.org/install/quickstart/

Steps to Reproduce:
1. follow the steps in https://www.rdoproject.org/install/quickstart/
2. run packstack --allinone

Actual results:

192.168.1.158_keystone.pp:                        [ ERROR ]          
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.1.158_keystone.pp
Error: Failed to apply catalog: Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Internal Server Error (HTTP 500)

Expected results:

No installation errors

Additional info:

[root@so ~]# cat /var/tmp/packstack/20151027-193859-XXZqoI/openstack-setup.log
2015-10-27 19:38:59::INFO::shell::94::root:: [localhost] Executing script:
DEVICE=($(ip -o address show to 192.168.1.158 | cut -f 2 -d ' '))
if [ ! -z ${DISPLAY[1]} ]; then false; fi
ip link show "$DEVICE" > /dev/null
echo $DEVICE
2015-10-27 19:39:00::INFO::shell::94::root:: [localhost] Executing script:
rm -rf /var/tmp/packstack/20151027-193859-XXZqoI/manifests/*pp
2015-10-27 19:39:00::INFO::shell::94::root:: [localhost] Executing script:
mkdir -p ~/.ssh
chmod 500 ~/.ssh
grep 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYj1Sz7MS/JZcjTTMvkUnDqcCYoxfR1nganMdiyM3cnrjYLVv+PWgpIEY413njcdn1ampm6cHyctTAwiWCtYVTuRm/OCIuts0n9uWegcBU/qjCU/lX55GOWFhClpwMEZSEmPTig1Ks3hEXeRpRJ71as+mxk3KbF9gxG6PNAmPGyZnFMewKbadJbNEyJGnqVF8+j2cIQVYThhAQGgzNzICSBwDuEFvcAwWxYPS6tHukG6p2puuJesopKqiK5pW+sxDWex1yjd6UQy9Z8dcWBrdVLqM/rlaUwo5kpC6Tvzrc5n9rIMoUTG4sPq1gDWbK+ijP0dSTx1ft72aiHBfyKzMT root.se' ~/.ssh/authorized_keys > /dev/null 2>&1 || echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYj1Sz7MS/JZcjTTMvkUnDqcCYoxfR1nganMdiyM3cnrjYLVv+PWgpIEY413njcdn1ampm6cHyctTAwiWCtYVTuRm/OCIuts0n9uWegcBU/qjCU/lX55GOWFhClpwMEZSEmPTig1Ks3hEXeRpRJ71as+mxk3KbF9gxG6PNAmPGyZnFMewKbadJbNEyJGnqVF8+j2cIQVYThhAQGgzNzICSBwDuEFvcAwWxYPS6tHukG6p2puuJesopKqiK5pW+sxDWex1yjd6UQy9Z8dcWBrdVLqM/rlaUwo5kpC6Tvzrc5n9rIMoUTG4sPq1gDWbK+ijP0dSTx1ft72aiHBfyKzMT root.se >> ~/.ssh/authorized_keys
chmod 400 ~/.ssh/authorized_keys
restorecon -r ~/.ssh
2015-10-27 19:39:00::INFO::shell::94::root:: [192.168.1.158] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
2015-10-27 19:39:01::INFO::shell::48::root:: Executing command:
rpm -q rdo-release --qf='%{version}-%{release}.%{arch}
'
2015-10-27 19:39:02::INFO::shell::94::root:: [192.168.1.158] Executing script:
(rpm -q 'rdo-release-liberty' || yum install -y --nogpg http://rdo.fedorapeople.org/openstack/openstack-liberty/rdo-release-liberty-2.noarch.rpm) || true
2015-10-27 19:39:04::INFO::shell::94::root:: [192.168.1.158] Executing script:
yum-config-manager --enable openstack-liberty
2015-10-27 19:39:05::INFO::shell::94::root:: [192.168.1.158] Executing script:
python -c "import platform; print platform.linux_distribution(full_distribution_name=0)[0]+','+platform.linux_distribution()[1]"
2015-10-27 19:39:05::INFO::shell::94::root:: [192.168.1.158] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
yum clean metadata
2015-10-27 19:39:07::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-puppet-modules | egrep -v "^(rpmlib|\/|perl)"
2015-10-27 19:39:08::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-packstack-puppet | egrep -v "^(rpmlib|\/|perl)"
2015-10-27 19:39:09::INFO::shell::94::root:: [192.168.1.158] Executing script:
yum install -y puppet hiera openssh-clients tar nc rubygem-json
yum update -y puppet hiera openssh-clients tar nc rubygem-json
rpm -q --whatprovides puppet
rpm -q --whatprovides hiera
rpm -q --whatprovides openssh-clients
rpm -q --whatprovides tar
rpm -q --whatprovides nc
rpm -q --whatprovides rubygem-json
2015-10-27 19:40:41::INFO::shell::94::root:: [192.168.1.158] Executing script:
mkdir -p /var/tmp/packstack
mkdir --mode 0700 /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1
mkdir --mode 0700 /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/modules
mkdir --mode 0700 /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/resources
2015-10-27 19:40:41::INFO::shell::94::root:: [192.168.1.158] Executing script:
facter -p
2015-10-27 19:40:43::INFO::shell::94::root:: [192.168.1.158] Executing script:
[[ ! -L /etc/puppet/hiera.yaml ]] && ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml || echo "hiera.yaml symlink already created"
sed -i 's;:datadir:.*;:datadir: /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/hieradata;g' /etc/puppet/hiera.yaml
2015-10-27 19:40:43::INFO::shell::94::root:: [192.168.1.158] Executing script:
vgdisplay cinder-volumes
2015-10-27 19:40:44::INFO::shell::94::root:: [localhost] Executing script:
ssh-keygen -t rsa -b 2048 -f "/var/tmp/packstack/20151027-193859-XXZqoI/nova_migration_key" -N ""
2015-10-27 19:40:44::INFO::shell::94::root:: [localhost] Executing script:
ssh-keyscan 192.168.1.158
2015-10-27 19:40:44::INFO::shell::94::root:: [192.168.1.158] Executing script:
systemctl
2015-10-27 19:40:45::INFO::shell::94::root:: [192.168.1.158] Executing script:
systemctl is-enabled NetworkManager
2015-10-27 19:40:45::INFO::shell::94::root:: [192.168.1.158] Executing script:
systemctl is-active NetworkManager
2015-10-27 19:40:45::INFO::shell::94::root:: [192.168.1.158] Executing script:
echo $HOME
2015-10-27 19:40:46::INFO::shell::94::root:: [localhost] Executing script:
cd /var/tmp/packstack/20151027-193859-XXZqoI/hieradata
tar --dereference -cpzf - ../hieradata | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.158 tar -C /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1 -xpzf -
cd /usr/lib/python2.7/site-packages/packstack/puppet
cd /var/tmp/packstack/20151027-193859-XXZqoI/manifests
tar --dereference -cpzf - ../manifests | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.158 tar -C /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1 -xpzf -
cd /usr/share/openstack-puppet/modules
tar --dereference -cpzf - apache ceilometer certmonger cinder concat firewall glance galera heat horizon inifile ironic keystone manila memcached mongodb mysql neutron nova nssdb openstack packstack qpid rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest trove vcsrepo vlan vswitch xinetd openstacklib | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.158 tar -C /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/modules -xpzf -
2015-10-27 19:41:42::ERROR::run_setup::1017::root:: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 992, in main
    single_step_aio_install(options, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 760, in single_step_aio_install
    single_step_install(options, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 791, in single_step_install
    _main(options, answerfilepath, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 660, in _main
    runSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 627, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/setup_controller.py", line 81, in runAllSequences
    sequence.run(config=self.CONF, messages=self.MESSAGES)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 109, in run
    step.run(config=config, messages=messages)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 50, in run
    self.function(config, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 209, in apply_puppet_manifest
    wait_for_puppet(currently_running, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 123, in wait_for_puppet
    validate_logfile(log)
  File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 107, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: 192.168.1.158_keystone.pp
Error: Failed to apply catalog: Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Internal Server Error (HTTP 500)
You will find full trace in log /var/tmp/packstack/20151027-193859-XXZqoI/manifests/192.168.1.158_keystone.pp.log

2015-10-27 19:41:42::INFO::shell::94::root:: [192.168.1.158] Executing script:
rm -rf /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1




[root@so ~]# cat /var/tmp/packstack/20151027-193859-XXZqoI/openstack-setup.log
2015-10-27 19:38:59::INFO::shell::94::root:: [localhost] Executing script:
DEVICE=($(ip -o address show to 192.168.1.158 | cut -f 2 -d ' '))
if [ ! -z ${DISPLAY[1]} ]; then false; fi
ip link show "$DEVICE" > /dev/null
echo $DEVICE
2015-10-27 19:39:00::INFO::shell::94::root:: [localhost] Executing script:
rm -rf /var/tmp/packstack/20151027-193859-XXZqoI/manifests/*pp
2015-10-27 19:39:00::INFO::shell::94::root:: [localhost] Executing script:
mkdir -p ~/.ssh
chmod 500 ~/.ssh
grep 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYj1Sz7MS/JZcjTTMvkUnDqcCYoxfR1nganMdiyM3cnrjYLVv+PWgpIEY413njcdn1ampm6cHyctTAwiWCtYVTuRm/OCIuts0n9uWegcBU/qjCU/lX55GOWFhClpwMEZSEmPTig1Ks3hEXeRpRJ71as+mxk3KbF9gxG6PNAmPGyZnFMewKbadJbNEyJGnqVF8+j2cIQVYThhAQGgzNzICSBwDuEFvcAwWxYPS6tHukG6p2puuJesopKqiK5pW+sxDWex1yjd6UQy9Z8dcWBrdVLqM/rlaUwo5kpC6Tvzrc5n9rIMoUTG4sPq1gDWbK+ijP0dSTx1ft72aiHBfyKzMT root.se' ~/.ssh/authorized_keys > /dev/null 2>&1 || echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYj1Sz7MS/JZcjTTMvkUnDqcCYoxfR1nganMdiyM3cnrjYLVv+PWgpIEY413njcdn1ampm6cHyctTAwiWCtYVTuRm/OCIuts0n9uWegcBU/qjCU/lX55GOWFhClpwMEZSEmPTig1Ks3hEXeRpRJ71as+mxk3KbF9gxG6PNAmPGyZnFMewKbadJbNEyJGnqVF8+j2cIQVYThhAQGgzNzICSBwDuEFvcAwWxYPS6tHukG6p2puuJesopKqiK5pW+sxDWex1yjd6UQy9Z8dcWBrdVLqM/rlaUwo5kpC6Tvzrc5n9rIMoUTG4sPq1gDWbK+ijP0dSTx1ft72aiHBfyKzMT root.se >> ~/.ssh/authorized_keys
chmod 400 ~/.ssh/authorized_keys
restorecon -r ~/.ssh
2015-10-27 19:39:00::INFO::shell::94::root:: [192.168.1.158] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
2015-10-27 19:39:01::INFO::shell::48::root:: Executing command:
rpm -q rdo-release --qf='%{version}-%{release}.%{arch}
'
2015-10-27 19:39:02::INFO::shell::94::root:: [192.168.1.158] Executing script:
(rpm -q 'rdo-release-liberty' || yum install -y --nogpg http://rdo.fedorapeople.org/openstack/openstack-liberty/rdo-release-liberty-2.noarch.rpm) || true
2015-10-27 19:39:04::INFO::shell::94::root:: [192.168.1.158] Executing script:
yum-config-manager --enable openstack-liberty
2015-10-27 19:39:05::INFO::shell::94::root:: [192.168.1.158] Executing script:
python -c "import platform; print platform.linux_distribution(full_distribution_name=0)[0]+','+platform.linux_distribution()[1]"
2015-10-27 19:39:05::INFO::shell::94::root:: [192.168.1.158] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
yum clean metadata
2015-10-27 19:39:07::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-puppet-modules | egrep -v "^(rpmlib|\/|perl)"
2015-10-27 19:39:08::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-packstack-puppet | egrep -v "^(rpmlib|\/|perl)"
2015-10-27 19:39:09::INFO::shell::94::root:: [192.168.1.158] Executing script:
yum install -y puppet hiera openssh-clients tar nc rubygem-json
yum update -y puppet hiera openssh-clients tar nc rubygem-json
rpm -q --whatprovides puppet
rpm -q --whatprovides hiera
rpm -q --whatprovides openssh-clients
rpm -q --whatprovides tar
rpm -q --whatprovides nc
rpm -q --whatprovides rubygem-json
2015-10-27 19:40:41::INFO::shell::94::root:: [192.168.1.158] Executing script:
mkdir -p /var/tmp/packstack
mkdir --mode 0700 /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1
mkdir --mode 0700 /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/modules
mkdir --mode 0700 /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/resources
2015-10-27 19:40:41::INFO::shell::94::root:: [192.168.1.158] Executing script:
facter -p
2015-10-27 19:40:43::INFO::shell::94::root:: [192.168.1.158] Executing script:
[[ ! -L /etc/puppet/hiera.yaml ]] && ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml || echo "hiera.yaml symlink already created"
sed -i 's;:datadir:.*;:datadir: /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/hieradata;g' /etc/puppet/hiera.yaml
2015-10-27 19:40:43::INFO::shell::94::root:: [192.168.1.158] Executing script:
vgdisplay cinder-volumes
2015-10-27 19:40:44::INFO::shell::94::root:: [localhost] Executing script:
ssh-keygen -t rsa -b 2048 -f "/var/tmp/packstack/20151027-193859-XXZqoI/nova_migration_key" -N ""
2015-10-27 19:40:44::INFO::shell::94::root:: [localhost] Executing script:
ssh-keyscan 192.168.1.158
2015-10-27 19:40:44::INFO::shell::94::root:: [192.168.1.158] Executing script:
systemctl
2015-10-27 19:40:45::INFO::shell::94::root:: [192.168.1.158] Executing script:
systemctl is-enabled NetworkManager
2015-10-27 19:40:45::INFO::shell::94::root:: [192.168.1.158] Executing script:
systemctl is-active NetworkManager
2015-10-27 19:40:45::INFO::shell::94::root:: [192.168.1.158] Executing script:
echo $HOME
2015-10-27 19:40:46::INFO::shell::94::root:: [localhost] Executing script:
cd /var/tmp/packstack/20151027-193859-XXZqoI/hieradata
tar --dereference -cpzf - ../hieradata | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.158 tar -C /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1 -xpzf -
cd /usr/lib/python2.7/site-packages/packstack/puppet
cd /var/tmp/packstack/20151027-193859-XXZqoI/manifests
tar --dereference -cpzf - ../manifests | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.158 tar -C /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1 -xpzf -
cd /usr/share/openstack-puppet/modules
tar --dereference -cpzf - apache ceilometer certmonger cinder concat firewall glance galera heat horizon inifile ironic keystone manila memcached mongodb mysql neutron nova nssdb openstack packstack qpid rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest trove vcsrepo vlan vswitch xinetd openstacklib | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.158 tar -C /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1/modules -xpzf -
2015-10-27 19:41:42::ERROR::run_setup::1017::root:: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 992, in main
    single_step_aio_install(options, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 760, in single_step_aio_install
    single_step_install(options, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 791, in single_step_install
    _main(options, answerfilepath, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 660, in _main
    runSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 627, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/setup_controller.py", line 81, in runAllSequences
    sequence.run(config=self.CONF, messages=self.MESSAGES)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 109, in run
    step.run(config=config, messages=messages)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 50, in run
    self.function(config, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 209, in apply_puppet_manifest
    wait_for_puppet(currently_running, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 123, in wait_for_puppet
    validate_logfile(log)
  File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 107, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: 192.168.1.158_keystone.pp
Error: Failed to apply catalog: Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Internal Server Error (HTTP 500)
You will find full trace in log /var/tmp/packstack/20151027-193859-XXZqoI/manifests/192.168.1.158_keystone.pp.log

2015-10-27 19:41:42::INFO::shell::94::root:: [192.168.1.158] Executing script:
rm -rf /var/tmp/packstack/34436bb2b9014b688f05d141f2f44ac1

Comment 1 Adam Gordon 2015-12-03 23:08:04 UTC
Can confirm this error today on fedora 23

Comment 3 Chandan Kumar 2016-05-19 15:38:00 UTC
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.