Bug 1276277

After a system reboot everything stopped working:

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1446065956
[root@so sv]# abrt-cli list --since 1446065956
id 678ac176fca432b62047a5d0d6c92d22e596279a
reason:         __init__.py:925:_open:IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'
time:           Thu 29 Oct 2015 12:01:05 AM CET
cmdline:        /usr/bin/python2 /bin/nova-manage db archive_deleted_rows --max_rows 100
package:        openstack-nova-common-12.0.0-1.el7
uid:            162 (nova)
count:          2
Directory:      /var/spool/abrt/Python-2015-10-29-00:01:05-3572

is re-running

packstack --allinone

after every reboot is the only and intentional way to proceed?

Then everything done on the openstack installation between reboots is supposed to be scrapped?

worked yesterday, but this morning (no changes were made):

[root@so ~(keystone_admin)]# openstack service list
Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
Unable to establish connection to http://192.168.1.158:5000/v2.0/tokens

This piece of ...ware is desperately buggy

another run of

packstack --allinone

got stuck for more than 8 hours at

...
Applying 192.168.1.130_api_nova.pp
192.168.1.130_api_nova.pp:                           [ DONE ]        
Applying 192.168.1.130_nova.pp
Testing if puppet apply is finished: 192.168.1.130_nova.pp  [ | ]

[root@so packstack]# date
Sat Oct 31 17:13:23 CET 2015
[root@so packstack]# cat /var/tmp/packstack/latest/openstack-setup.log 
2015-10-31 11:08:25::INFO::shell::94::root:: [localhost] Executing script:
DEVICE=($(ip -o address show to 192.168.1.130 | cut -f 2 -d ' '))
if [ ! -z ${DISPLAY[1]} ]; then false; fi
ip link show "$DEVICE" > /dev/null
echo $DEVICE
2015-10-31 11:08:25::INFO::shell::94::root:: [localhost] Executing script:
rm -rf /var/tmp/packstack/20151031-110823-h2VdSD/manifests/*pp
2015-10-31 11:08:25::INFO::shell::94::root:: [localhost] Executing script:
mkdir -p ~/.ssh
chmod 500 ~/.ssh
grep 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTEDeV1Bb3MzI3+7Nmoh28tQE2ZcSMV/fNoOOrScwzoEvTKNfAlYain/ORdcFoVfOYGhjiQDyu3qmFD8fvrl79Xioib21agg7Xts7H2qnm4BOXAblvQ7qskOUjBkjt0noSzubx112JpcmrDw6rYKuMW2TzEes6ZUrhQbTGNzPEAxSfvowFGEtTJpwdjl0im1cMR+u6/DHd2vBqgU2dtWwgxoUA8V0Oo23lvXrmrJcjNkdLVczqaOolAiNx5t3ME0vjnoOBEfynNgK1UT9eKGf/XePCxEmI2a4seAxB8/ntgs0Bcyk5Z9H6eWfjLO2xCPIm24ijpy0WDuUqyjFDbS+l root.se' ~/.ssh/authorized_keys > /dev/null 2>&1 || echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTEDeV1Bb3MzI3+7Nmoh28tQE2ZcSMV/fNoOOrScwzoEvTKNfAlYain/ORdcFoVfOYGhjiQDyu3qmFD8fvrl79Xioib21agg7Xts7H2qnm4BOXAblvQ7qskOUjBkjt0noSzubx112JpcmrDw6rYKuMW2TzEes6ZUrhQbTGNzPEAxSfvowFGEtTJpwdjl0im1cMR+u6/DHd2vBqgU2dtWwgxoUA8V0Oo23lvXrmrJcjNkdLVczqaOolAiNx5t3ME0vjnoOBEfynNgK1UT9eKGf/XePCxEmI2a4seAxB8/ntgs0Bcyk5Z9H6eWfjLO2xCPIm24ijpy0WDuUqyjFDbS+l root.se >> ~/.ssh/authorized_keys
chmod 400 ~/.ssh/authorized_keys
restorecon -r ~/.ssh
2015-10-31 11:08:25::INFO::shell::94::root:: [192.168.1.130] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
2015-10-31 11:08:27::INFO::shell::48::root:: Executing command:
rpm -q rdo-release --qf='%{version}-%{release}.%{arch}
'
2015-10-31 11:08:27::INFO::shell::94::root:: [192.168.1.130] Executing script:
(rpm -q 'rdo-release-liberty' || yum install -y --nogpg http://rdo.fedorapeople.org/openstack/openstack-liberty/rdo-release-liberty-2.noarch.rpm) || true
2015-10-31 11:08:31::INFO::shell::94::root:: [192.168.1.130] Executing script:
yum-config-manager --enable openstack-liberty
2015-10-31 11:08:40::INFO::shell::94::root:: [192.168.1.130] Executing script:
python -c "import platform; print platform.linux_distribution(full_distribution_name=0)[0]+','+platform.linux_distribution()[1]"
2015-10-31 11:08:44::INFO::shell::94::root:: [192.168.1.130] Executing script:
python -c "import platform; print platform.linux_distribution(full_distribution_name=0)[0]+','+platform.linux_distribution()[1]"
2015-10-31 11:08:46::INFO::shell::94::root:: [192.168.1.130] Executing script:
REPOFILE=$(mktemp)
cat /etc/yum.conf > $REPOFILE
echo -e '[packstack-epel]
name=packstack-epel
enabled=1
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch' >> $REPOFILE
( rpm -q --whatprovides epel-release || yum install -y --nogpg -c $REPOFILE epel-release ) || true
rm -rf $REPOFILE
2015-10-31 11:08:50::INFO::shell::94::root:: [192.168.1.130] Executing script:
rpm -q yum-utils || yum -y install yum-utils
yum-config-manager --enable epel
2015-10-31 11:08:53::INFO::shell::94::root:: [192.168.1.130] Executing script:
rpm -q --whatprovides yum-utils || yum install -y yum-utils
yum clean metadata
2015-10-31 11:08:57::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-puppet-modules | egrep -v "^(rpmlib|\/|perl)"
2015-10-31 11:08:57::INFO::shell::94::root:: [localhost] Executing script:
rpm -q --requires openstack-packstack-puppet | egrep -v "^(rpmlib|\/|perl)"
2015-10-31 11:08:57::INFO::shell::94::root:: [192.168.1.130] Executing script:
yum install -y puppet hiera openssh-clients tar nc rubygem-json
yum update -y puppet hiera openssh-clients tar nc rubygem-json
rpm -q --whatprovides puppet
rpm -q --whatprovides hiera
rpm -q --whatprovides openssh-clients
rpm -q --whatprovides tar
rpm -q --whatprovides nc
rpm -q --whatprovides rubygem-json
2015-10-31 11:09:48::INFO::shell::94::root:: [192.168.1.130] Executing script:
mkdir -p /var/tmp/packstack
mkdir --mode 0700 /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6
mkdir --mode 0700 /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/modules
mkdir --mode 0700 /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/resources
2015-10-31 11:09:51::INFO::shell::94::root:: [192.168.1.130] Executing script:
facter -p
2015-10-31 11:10:05::INFO::shell::94::root:: [192.168.1.130] Executing script:
[[ ! -L /etc/puppet/hiera.yaml ]] && ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml || echo "hiera.yaml symlink already created"
sed -i 's;:datadir:.*;:datadir: /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/hieradata;g' /etc/puppet/hiera.yaml
2015-10-31 11:10:07::INFO::shell::94::root:: [192.168.1.130] Executing script:
vgdisplay cinder-volumes
2015-10-31 11:10:11::INFO::shell::94::root:: [localhost] Executing script:
ssh-keygen -t rsa -b 2048 -f "/var/tmp/packstack/20151031-110823-h2VdSD/nova_migration_key" -N ""
2015-10-31 11:10:11::INFO::shell::94::root:: [localhost] Executing script:
ssh-keyscan 192.168.1.130
2015-10-31 11:10:11::INFO::shell::94::root:: [192.168.1.130] Executing script:
systemctl
2015-10-31 11:10:14::INFO::shell::94::root:: [192.168.1.130] Executing script:
systemctl is-enabled NetworkManager
2015-10-31 11:10:16::INFO::shell::94::root:: [192.168.1.130] Executing script:
systemctl is-active NetworkManager
2015-10-31 11:10:25::INFO::shell::94::root:: [192.168.1.130] Executing script:
echo $HOME
2015-10-31 11:10:27::INFO::shell::94::root:: [localhost] Executing script:
cd /var/tmp/packstack/20151031-110823-h2VdSD/hieradata
tar --dereference -cpzf - ../hieradata | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.130 tar -C /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6 -xpzf -
cd /usr/lib/python2.7/site-packages/packstack/puppet
cd /var/tmp/packstack/20151031-110823-h2VdSD/manifests
tar --dereference -cpzf - ../manifests | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.130 tar -C /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6 -xpzf -
cd /usr/share/openstack-puppet/modules
tar --dereference -cpzf - apache ceilometer certmonger cinder concat firewall glance galera heat horizon inifile ironic keystone manila memcached mongodb mysql neutron nova nssdb openstack packstack qpid rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest trove vcsrepo vlan vswitch xinetd openstacklib | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root.1.130 tar -C /var/tmp/packstack/3acdb68b90a14422901fd9a83f3e3cf6/modules -xpzf -

Hi,

I tried packstack --allinone but failed at Keystone httpd.

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 

openstack-packstack
version:7.0.0
release:0.10.dev1682.g42b3426.el7

it says..

124.39.118.124_keystone.pp:                       [ ERROR ]           
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 124.39.118.124_keystone.pp
Error: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
You will find full trace in log /var/tmp/packstack/20160229-141456-xVn0sB/manifests/124.39.118.124_keystone.pp.log
Please check log file /var/tmp/packstack/20160229-141456-xVn0sB/openstack-setup.log for more information

But I found a lack on httpd.conf. the line to include conf.modules.d/* is missing.
should have a line like bellow:

Include "/etc/httpd/conf.modules.d/*.conf"

It looks httpd.conf is managed by puppet master. While I fix this, packstack will revert it...

Thanks.

I'm hitting this error too and spent a lot of time tonight tracing through the openstack-puppet-modules hoping to unblock the install.  Also Using CentOS-7 but Akinobu is spot on.

in /var/log/messages:

Feb 29 22:18:41 localhost httpd: AH00526: Syntax error on line 31 of /etc/httpd/conf.d/php.conf:

According to: http://serverfault.com/questions/515417/error-start-apache-php-value this is caused by a missing libphp5.so file.  

At first I thought there was a missing "Load Module" line not getting inserted into the config yet after looking closer I see that logic one line 65 of https://github.com/redhat-openstack/openstack-puppet-modules/blob/master/apache/manifests/params.pp changes the module config directory for RHEL/Centos-7. 

I'm not sure this is 100% correct but I watched this multiple times by removing packstack then performing a first 'failed' install where I saw both conf.d and conf.modules.d get created and populated. Once the keystone.pp failed I copied the files from /etc/httpd/conf.modules.d into /etc/httpd/conf.d and the httpd service started right up.  

I then executed another packstack install on top of the system/failure and all completed as expected.

Not sure what could be causing this...

(In reply to cappetta from comment #6)
> I'm hitting this error too and spent a lot of time tonight tracing through
> the openstack-puppet-modules hoping to unblock the install.  Also Using
> CentOS-7 but Akinobu is spot on.
> 
> in /var/log/messages:
> 
> Feb 29 22:18:41 localhost httpd: AH00526: Syntax error on line 31 of
> /etc/httpd/conf.d/php.conf:
> 
> According to:
> http://serverfault.com/questions/515417/error-start-apache-php-value this is
> caused by a missing libphp5.so file.  
> 
> At first I thought there was a missing "Load Module" line not getting
> inserted into the config yet after looking closer I see that logic one line
> 65 of
> https://github.com/redhat-openstack/openstack-puppet-modules/blob/master/
> apache/manifests/params.pp changes the module config directory for
> RHEL/Centos-7. 
> 
> I'm not sure this is 100% correct but I watched this multiple times by
> removing packstack then performing a first 'failed' install where I saw both
> conf.d and conf.modules.d get created and populated. Once the keystone.pp
> failed I copied the files from /etc/httpd/conf.modules.d into
> /etc/httpd/conf.d and the httpd service started right up.  
> 
> I then executed another packstack install on top of the system/failure and
> all completed as expected.
> 
> Not sure what could be causing this...

I can re-iterate this solution/workaround. I experienced the same problems described here but after copying all the config files from "/etc/httpd/conf.modules.d" to "/etc/httpd/conf.d" and re-running the installation, everything worked fine. 

Because of the missing Include directive in the httpd.conf file the keystone.pp configuration cannot complete (because apache will not re-start after the applied configuration changes) and the packstack installation fails and stops at that point. This leaves many other puppet configurations un-applied and the entire openstack setup non-functional.

It seems to me that this would be fixed by simply adding the Include directive (as mentioned earlier) in the /etc/httpd/conf/httpd.conf file configured by puppet:

Include "/etc/httpd/conf.modules.d/*.conf"

In the meantime, by simply copying all config files from "/etc/httpd/conf.modules.d" to "/etc/httpd/conf.d" and re-running packstack --allinone works.

This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.