Bug 1276277 - packstack --allinone fails on CentOS 7 x86_64 1503-01
packstack --allinone fails on CentOS 7 x86_64 1503-01
Status: CLOSED EOL
Product: RDO
Classification: Community
Component: openstack-packstack (Show other bugs)
Liberty
x86_64 Linux
unspecified Severity high
: ---
: Kilo
Assigned To: Martin Magr
Shai Revivo
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-29 06:29 EDT by Sergei
Modified: 2016-05-19 12:01 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-19 12:01:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sergei 2015-10-29 06:29:07 EDT
Description of problem:

Quickstart recipe with packstack --allinone
fails big time on stock CentOS 7 x86_64 1503-01

Version-Release number of selected component (if applicable):

rdo-release-liberty-2.noarch

How reproducible:
see below

Steps to Reproduce:
1. install stock CentOS 7 x86_64 1503-01
2. follow the steps at https://www.rdoproject.org/install/quickstart/
3. run packstack --allinone

Actual results:

ERROR : Error appeared during Puppet run: 192.168.1.158_keystone.pp
Error: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
You will find full trace in log /var/tmp/packstack/20151028-215212-Q5r3xp/manifests/192.168.1.158_keystone.pp.log


Expected results:

seamless installation


Additional info:

do you people ever test your c@#$%pware on any major Linux distributions you recommend yourselves?

Previously I reported it fails big time on Fedora 22-3.

Now it's the same with CentOS, the #1 on your recommendation list

To make it work, by painstaking trial and error I had to manually doctor three files in

/etc/httpd/conf.d:

1. fcgid.conf - added line

LoadModule fcgid_module modules/mod_fcgid.so

2. php.conf - added lone

LoadModule php5_module modules/libphp5.so

3. ssl.conf - added two lines:

LoadModule ssl_module modules/mod_ssl.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so

I don't think many people would do it. Your puppets are useless and take forever to finish, as if they solve a heavy computational large hadron collider problem.
Comment 1 Sergei 2015-10-29 10:30:13 EDT
After a system reboot everything stopped working:

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1446065956
[root@so sv]# abrt-cli list --since 1446065956
id 678ac176fca432b62047a5d0d6c92d22e596279a
reason:         __init__.py:925:_open:IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'
time:           Thu 29 Oct 2015 12:01:05 AM CET
cmdline:        /usr/bin/python2 /bin/nova-manage db archive_deleted_rows --max_rows 100
package:        openstack-nova-common-12.0.0-1.el7
uid:            162 (nova)
count:          2
Directory:      /var/spool/abrt/Python-2015-10-29-00:01:05-3572
Comment 2 Sergei 2015-10-29 10:46:38 EDT
is re-running

packstack --allinone

after every reboot is the only and intentional way to proceed?

Then everything done on the openstack installation between reboots is supposed to be scrapped?
Comment 3 Sergei 2015-10-31 05:06:33 EDT
worked yesterday, but this morning (no changes were made):

[root@so ~(keystone_admin)]# openstack service list
Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
Unable to establish connection to http://192.168.1.158:5000/v2.0/tokens

This piece of ...ware is desperately buggy
Comment 4 Sergei 2015-10-31 12:13:59 EDT
another run of

packstack --allinone

got stuck for more than 8 hours at

...
Applying 192.168.1.130_api_nova.pp
192.168.1.130_api_nova.pp:                           [ DONE ]        
Applying 192.168.1.130_nova.pp
Testing if puppet apply is finished: 192.168.1.130_nova.pp  [ | ]

[root@so packstack]# date
Sat Oct 31 17:13:23 CET 2015
[root@so packstack]# cat /var/tmp/packstack/latest/openstack-setup.log 
2015-10-31 11:08:25::INFO::shell::94::root:: [localhost] Executing script:
DEVICE=($(ip -o address show to 192.168.1.130 | cut -f 2 -d ' '))
if [ ! -z ${DISPLAY[1]} ]; then false; fi
ip link show "$DEVICE" > /dev/null
echo $DEVICE
2015-10-31 11:08:25::INFO::shell::94::root:: [localhost] Executing script:
rm -rf /var/tmp/packstack/20151031-110823-h2VdSD/manifests/*pp
2015-10-31 11:08:25::INFO::shell::94::root:: [localhost] Executing script:
mkdir -p ~/.ssh
chmod 500 ~/.ssh
grep 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTEDeV1Bb3MzI3+7Nmoh28tQE2ZcSMV/fNoOOrScwzoEvTKNfAlYain/ORdcFoVfOYGhjiQDyu3qmFD8fvrl79Xioib21agg7Xts7H2qnm4BOXAblvQ7qskOUjBkjt0noSzubx112JpcmrDw6rYKuMW2TzEes6ZUrhQbTGNzPEAxSfvowFGEtTJpwdjl0im1cMR+u6/DHd2vBqgU2dtWwgxoUA8V0Oo23lvXrmrJcjNkdLVczqaOolAiNx5t3ME0vjnoOBEfynNgK1UT9eKGf/XePCxEmI2a4seAxB8/ntgs0Bcyk5Z9H6eWfjLO2xCPIm24ijpy0WDuUqyjFDbS+l root@so.sv.se' ~/.ssh/authorized_keys > /dev/null 2>&1 || echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTEDeV1Bb3MzI3+7Nmoh28tQE2ZcSMV/fNoOOrScwzoEvTKNfAlYain/ORdcFoVfOYGhjiQDyu3qmFD8fvrl79Xioib21agg7Xts7H2qnm4BOXAblvQ7qskOUjBkjt0noSzubx112JpcmrDw6rYKuMW2TzEes6ZUrhQbTGNzPEAxSfvowFGEtTJpwdjl0im1cMR+u6/DHd2vBqgU2dtWwgxoUA8V0Oo23lvXrmrJcjNkdLVczqaOolAiNx5t3ME0vjnoOBEfynNgK1UT9eKGf/XePCxEmI2a4seAxB8/ntgs0Bcyk5Z9H6eWfjLO2xCPIm24ijpy0WDuUqyjFDbS+l root@so.sv.se >> ~/.ssh/authorized_keys
chmod 400 ~/.ssh/authorized_keys
restorecon -r ~/.ssh
2015-10-31 11:08:25::INFO::shell::94::root:: [192.168.1.130] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
2015-10-31 11:08:27::INFO::shell::48::root:: Executing command:
rpm -q rdo-release --qf='%{version}-%{release}.%{arch}
'
2015-10-31 11:08:27::INFO::shell::94::root:: [192.168.1.130] Executing script:
(rpm -q 'rdo-release-liberty' || yum install -y --nogpg http://rdo.fedorapeople.org/openstack/openstack-liberty/rdo-release-liberty-2.noarch.rpm) || true
2015-10-31 11:08:31::INFO::shell::94::root:: [192.168.1.130] Executing script:
yum-config-manager --enable openstack-liberty
2015-10-31 11:08:40::INFO::shell::94::root:: [192.168.1.130] Executing script:
python -c "import platform; print platform.linux_distribution(full_distribution_name=0)[0]+','+platform.linux_distribution()[1]"
2015-10-31 11:08:44::INFO::shell::94::root:: [192.168.1.130] Executing script:
python -c "import platform; print platform.linux_distribution(full_distribution_name=0)[0]+','+platform.linux_distribution()[1]"
2015-10-31 11:08:46::INFO::shell::94::root:: [192.168.1.130] Executing script:
REPOFILE=$(mktemp)
cat /etc/yum.conf > $REPOFILE
echo -e '[packstack-epel]
name=packstack-epel
enabled=1
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch' >> $REPOFILE
( rpm -q --whatprovides epel-release || yum install -y --nogpg -c $REPOFILE epel-release ) || true
rm -rf $REPOFILE
2015-10-31 11:08:50::INFO::shell::94::root:: [192.168.1.130] Executing script:
rpm -q yum-utils || yum -y install yum-utils
yum-config-manager --enable epel
2015-10-31 11:08:53::INFO::shell::94::root:: [192.168.1.130] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
yum clean metadata
2015-10-31 11:08:57::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-puppet-modules | egrep -v "^(rpmlib|\/|perl)"
2015-10-31 11:08:57::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-packstack-puppet | egrep -v "^(rpmlib|\/|perl)"
2015-10-31 11:08:57::INFO::shell::94::root:: [192.168.1.130] Executing script:
yum install -y puppet hiera openssh-clients tar nc rubygem-json
yum update -y puppet hiera openssh-clients tar nc rubygem-json
rpm -q --whatprovides puppet
rpm -q --whatprovides hiera
rpm -q --whatprovides openssh-clients
rpm -q --whatprovides tar
rpm -q --whatprovides nc
rpm -q --whatprovides rubygem-json
2015-10-31 11:09:48::INFO::shell::94::root:: [192.168.1.130] Executing script:
mkdir -p /var/tmp/packstack
mkdir --mode 0700 /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6
mkdir --mode 0700 /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/modules
mkdir --mode 0700 /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/resources
2015-10-31 11:09:51::INFO::shell::94::root:: [192.168.1.130] Executing script:
facter -p
2015-10-31 11:10:05::INFO::shell::94::root:: [192.168.1.130] Executing script:
[[ ! -L /etc/puppet/hiera.yaml ]] && ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml || echo "hiera.yaml symlink already created"
sed -i 's;:datadir:.*;:datadir: /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/hieradata;g' /etc/puppet/hiera.yaml
2015-10-31 11:10:07::INFO::shell::94::root:: [192.168.1.130] Executing script:
vgdisplay cinder-volumes
2015-10-31 11:10:11::INFO::shell::94::root:: [localhost] Executing script:
ssh-keygen -t rsa -b 2048 -f "/var/tmp/packstack/20151031-110823-h2VdSD/nova_migration_key" -N ""
2015-10-31 11:10:11::INFO::shell::94::root:: [localhost] Executing script:
ssh-keyscan 192.168.1.130
2015-10-31 11:10:11::INFO::shell::94::root:: [192.168.1.130] Executing script:
systemctl
2015-10-31 11:10:14::INFO::shell::94::root:: [192.168.1.130] Executing script:
systemctl is-enabled NetworkManager
2015-10-31 11:10:16::INFO::shell::94::root:: [192.168.1.130] Executing script:
systemctl is-active NetworkManager
2015-10-31 11:10:25::INFO::shell::94::root:: [192.168.1.130] Executing script:
echo $HOME
2015-10-31 11:10:27::INFO::shell::94::root:: [localhost] Executing script:
cd /var/tmp/packstack/20151031-110823-h2VdSD/hieradata
tar --dereference -cpzf - ../hieradata | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.1.130 tar -C /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6 -xpzf -
cd /usr/lib/python2.7/site-packages/packstack/puppet
cd /var/tmp/packstack/20151031-110823-h2VdSD/manifests
tar --dereference -cpzf - ../manifests | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.1.130 tar -C /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6 -xpzf -
cd /usr/share/openstack-puppet/modules
tar --dereference -cpzf - apache ceilometer certmonger cinder concat firewall glance galera heat horizon inifile ironic keystone manila memcached mongodb mysql neutron nova nssdb openstack packstack qpid rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest trove vcsrepo vlan vswitch xinetd openstacklib | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.1.130 tar -C /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/modules -xpzf -
Comment 5 Akinobu Fukuzaki 2016-02-29 00:41:23 EST
Hi,

I tried packstack --allinone but failed at Keystone httpd.

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 

openstack-packstack
version:7.0.0
release:0.10.dev1682.g42b3426.el7

it says..

124.39.118.124_keystone.pp:                       [ ERROR ]           
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 124.39.118.124_keystone.pp
Error: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
You will find full trace in log /var/tmp/packstack/20160229-141456-xVn0sB/manifests/124.39.118.124_keystone.pp.log
Please check log file /var/tmp/packstack/20160229-141456-xVn0sB/openstack-setup.log for more information

But I found a lack on httpd.conf. the line to include conf.modules.d/* is missing.
should have a line like bellow:

Include "/etc/httpd/conf.modules.d/*.conf"

It looks httpd.conf is managed by puppet master. While I fix this, packstack will revert it...

Thanks.
Comment 6 cappetta 2016-02-29 23:07:21 EST
I'm hitting this error too and spent a lot of time tonight tracing through the openstack-puppet-modules hoping to unblock the install.  Also Using CentOS-7 but Akinobu is spot on.

in /var/log/messages:

Feb 29 22:18:41 localhost httpd: AH00526: Syntax error on line 31 of /etc/httpd/conf.d/php.conf:

According to: http://serverfault.com/questions/515417/error-start-apache-php-value this is caused by a missing libphp5.so file.  

At first I thought there was a missing "Load Module" line not getting inserted into the config yet after looking closer I see that logic one line 65 of https://github.com/redhat-openstack/openstack-puppet-modules/blob/master/apache/manifests/params.pp changes the module config directory for RHEL/Centos-7. 

I'm not sure this is 100% correct but I watched this multiple times by removing packstack then performing a first 'failed' install where I saw both conf.d and conf.modules.d get created and populated. Once the keystone.pp failed I copied the files from /etc/httpd/conf.modules.d into /etc/httpd/conf.d and the httpd service started right up.  

I then executed another packstack install on top of the system/failure and all completed as expected.

Not sure what could be causing this...
Comment 7 Daniel Baarts 2016-03-08 13:57:44 EST
(In reply to cappetta from comment #6)
> I'm hitting this error too and spent a lot of time tonight tracing through
> the openstack-puppet-modules hoping to unblock the install.  Also Using
> CentOS-7 but Akinobu is spot on.
> 
> in /var/log/messages:
> 
> Feb 29 22:18:41 localhost httpd: AH00526: Syntax error on line 31 of
> /etc/httpd/conf.d/php.conf:
> 
> According to:
> http://serverfault.com/questions/515417/error-start-apache-php-value this is
> caused by a missing libphp5.so file.  
> 
> At first I thought there was a missing "Load Module" line not getting
> inserted into the config yet after looking closer I see that logic one line
> 65 of
> https://github.com/redhat-openstack/openstack-puppet-modules/blob/master/
> apache/manifests/params.pp changes the module config directory for
> RHEL/Centos-7. 
> 
> I'm not sure this is 100% correct but I watched this multiple times by
> removing packstack then performing a first 'failed' install where I saw both
> conf.d and conf.modules.d get created and populated. Once the keystone.pp
> failed I copied the files from /etc/httpd/conf.modules.d into
> /etc/httpd/conf.d and the httpd service started right up.  
> 
> I then executed another packstack install on top of the system/failure and
> all completed as expected.
> 
> Not sure what could be causing this...

I can re-iterate this solution/workaround. I experienced the same problems described here but after copying all the config files from "/etc/httpd/conf.modules.d" to "/etc/httpd/conf.d" and re-running the installation, everything worked fine. 

Because of the missing Include directive in the httpd.conf file the keystone.pp configuration cannot complete (because apache will not re-start after the applied configuration changes) and the packstack installation fails and stops at that point. This leaves many other puppet configurations un-applied and the entire openstack setup non-functional.

It seems to me that this would be fixed by simply adding the Include directive (as mentioned earlier) in the /etc/httpd/conf/httpd.conf file configured by puppet:

Include "/etc/httpd/conf.modules.d/*.conf"

In the meantime, by simply copying all config files from "/etc/httpd/conf.modules.d" to "/etc/httpd/conf.d" and re-running packstack --allinone works.
Comment 9 Chandan Kumar 2016-05-19 12:01:31 EDT
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.

Note You need to log in before you can comment on or make changes to this bug.