Bug 1096516 - Packstack gets stuck on prescript.pp
Summary: Packstack gets stuck on prescript.pp
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-packstack
Version: 5.0 (RHEL 7)
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 5.0 (RHEL 7)
Assignee: Gilles Dubreuil
QA Contact: Ami Jeain
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-11 13:20 UTC by Tzach Shefi
Modified: 2019-09-10 14:10 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-05-22 12:31:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
packstack logs (146.72 KB, application/x-compressed-tar)
2014-05-11 13:40 UTC, Tzach Shefi
no flags Details
Tested packstack answer file (17.40 KB, text/plain)
2014-05-16 05:28 UTC, Gilles Dubreuil
no flags Details
Tested packstack logs (20.17 KB, application/x-compressed-tar)
2014-05-16 05:38 UTC, Gilles Dubreuil
no flags Details
answer file and packstack recent logs. (24.26 KB, application/x-gzip)
2014-05-20 11:48 UTC, Tzach Shefi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1016773 0 unspecified CLOSED Packstack gets stuck on prescript.pp 2021-02-22 00:41:40 UTC

Description Tzach Shefi 2014-05-11 13:20:05 UTC
Description of problem:  Packstack freezes with this output:

Applying 10.35.162.22_prescript.pp
Testing if puppet apply is finished: 10.35.162.22_prescript.pp  [ / ]
^C

Rest of the output can be found at:
http://pastebin.test.redhat.com/208277

Semi dist' setup on virtual machines,  cc plus two compute nodes. 


Version-Release number of selected component (if applicable):
RHEL 7 
openstack-packstack-2014.1.1-0.11.1.dev1055.el7ost.noarch
openstack-packstack-puppet-2014.1.1-0.11.1.dev1055.el7ost.noarch

How reproducible:
Not sure happened only once as on now. 

Steps to Reproduce:
1. packstack --debug --answer-file=conf.txt 


Additional info:

Guessing the problem is here
  File "/usr/lib/python2.7/site-packages/packstack/installer/utils/shell.py", line 91, in execute
    out, err = obj.communicate(script)

Possible clone of an older bug for RHOS 3 BZ1016773

Workaround terminate packstack process and restart it again.

Comment 2 Tzach Shefi 2014-05-11 13:40:31 UTC
Created attachment 894436 [details]
packstack logs

Note the logs also include the latest packstack run which completed successfully. So for above bug ignore the last run and look at the ones before it.

Comment 3 Gilles Dubreuil 2014-05-15 06:03:22 UTC
Hi Tzach,

Could you please provide your packstack's answer file?

Thanks,
Gilles

Comment 4 Tzach Shefi 2014-05-15 06:39:32 UTC
Hi Gillies, 

Unfortunately I can't setup is gone. 
Mental note to my self always upload answer file on future packstack bugs.. 

If I run into this again, I'll post answer file here. 

Sorry

Comment 5 Nir Magnezi 2014-05-15 07:20:31 UTC
(In reply to Gilles Dubreuil from comment #3)
> Hi Tzach,
> 
> Could you please provide your packstack's answer file?
> 
> Thanks,
> Gilles

(In reply to Tzach Shefi from comment #4)
> Hi Gillies, 
> 
> Unfortunately I can't setup is gone. 
> Mental note to my self always upload answer file on future packstack bugs.. 
> 
> If I run into this again, I'll post answer file here. 
> 
> Sorry


Guys,
the packstack logs are attached to the bug, you can extract the answer file parameters from there:

2014-05-11 15:30:26::DEBUG::run_setup::549::root:: ssh-public-key: /root/.ssh/id_rsa.pub
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-mysql-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-glance-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-cinder-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-nova-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-neutron-install: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-horizon-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-swift-install: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-ceilometer-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-heat-install: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-client-install: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: ntp-servers: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: nagios-install: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: exclude-servers: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-debug-mode: false
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-vmware: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: mysql-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: mysql-user: root
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: mysql-pw: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-server: rabbitmq
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-enable-ssl: false
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-enable-auth: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-nss-certdb-pw: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-ssl-port: 5671
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-ssl-cert-file: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-ssl-key-file: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-auth-user: guest
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: amqp-auth-password: guest
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: keystone-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: keystone-db-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: keystone-admin-token: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: keystone-admin-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: keystone-demo-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: keystone-token-format: PKI
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: glance-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: glance-db-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: glance-ks-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: cinder-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: cinder-db-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: cinder-ks-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: cinder-backend: lvm
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: cinder-volumes-create: y
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: cinder-volumes-size: 20G
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novaapi-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novacert-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novavncproxy-hosts: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novacompute-hosts: 192.168.162.22,192.168.162.47
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novaconductor-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: nova-db-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: nova-ks-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novasched-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novasched-cpu-allocation-ratio: 16.0
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novasched-ram-allocation-ratio: 1.5
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novacompute-privif: eth1
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-hosts: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-manager: nova.network.manager.FlatDHCPManager
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-pubif: eth0
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-privif: eth1
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-fixed-range: 192.168.32.0/22
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-floating-range: 10.3.4.0/22
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-default-floating-pool: nova
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-auto-assign-floating-ip: False
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-vlan-start: 100
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-num-networks: 1
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: novanetwork-network-size: 255
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: osclient-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-horizon-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: os-horizon-ssl: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: provision-demo: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: provision-tempest: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: provision-tempest-repo-uri: https://github.com/openstack/tempest.git
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: provision-tempest-repo-revision: master
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: provision-all-in-one-ovs-bridge: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: ceilometer-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: ceilometer-secret: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: ceilometer-ks-passwd: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: mongodb-host: 192.168.163.28
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: use-epel: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: additional-repo: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: rh-username: 
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: rh-password: ********
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: rh-beta-repo: n
2014-05-11 15:30:26::DEBUG::run_setup::549::root:: rhn-satellite-server:

Comment 6 Gilles Dubreuil 2014-05-15 11:42:33 UTC
(In reply to Nir Magnezi from comment #5)

> Guys,
> the packstack logs are attached to the bug, you can extract the answer file
> parameters from there:
> 

Yeah, true. Although, in future I'd rather not have to do that, thanks for providing answer file.

(In reply to Tzach Shefi from comment #0)
> Semi dist' setup on virtual machines,  cc plus two compute nodes. 
> 

What's semi?
What's cc? Do you mean 1 controller node and two compute nodes?

Thanks, 
Gilles

Comment 7 Tzach Shefi 2014-05-15 12:35:06 UTC
Thanks Nir for tip. 

Gilles, correct
semi -> semi-distributed setup
cc -> controller (nova plus keystone sql..) plus two compute nodes. 
Don't remember if Glance and Cinder were also on controller or one of the compute nodes.

Comment 8 Gilles Dubreuil 2014-05-16 05:28:27 UTC
Created attachment 896165 [details]
Tested packstack answer file

Hi Tzach,

Using latest rhos5 repo:
- http://ayanami.boston.devel.redhat.com/poodles/rhos-devel-ci/5.0.el7/latest/RHOS-5-RHEL-7/$basearch/os
- openstack-puppet-modules-2014.1-11.el7ost.noarch
- openstack-packstack-2014.1.1-0.14.dev1068.el7ost.noarch

The prescript manifests works

Topology: 1 controller and 2 network nodes
Answer file: attached


Could you please re-test with latest?

Regards,
Gilles

Comment 9 Gilles Dubreuil 2014-05-16 05:38:26 UTC
Created attachment 896166 [details]
Tested packstack logs

Comment 10 Tzach Shefi 2014-05-20 11:46:54 UTC
Ran setup again from your source and answer file, updated answer file's IP address as they changed. I'll zip and upload logs in a second. 

10.35.163.123_mysql.pp:                           [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.35.163.123_mysql.pp
Error: mysqladmin -u root  password 'qum5net' returned 255 instead of one of [0]
You will find full trace in log /var/tmp/packstack/20140520-120112-dhYTT_/manifests/10.35.163.123_mysql.pp.log
Please check log file /var/tmp/packstack/20140520-120112-dhYTT_/openstack-setup.log for more information


Reran packstack again (after logs uploaded) failed again same place. 

Any tips?

Comment 11 Tzach Shefi 2014-05-20 11:48:01 UTC
Created attachment 897543 [details]
answer file and packstack recent logs.

Comment 12 Gilles Dubreuil 2014-05-21 01:51:13 UTC
Hi Tzach,

The output you've provided shows the install is failing during the mysql manifest and not the prescript one.

I understand there are different issues during packstack installation at this stage of RHOS5 lifecycle. 
Meanwhile this is another issue.

That said this errors also occurs when running packstack with different answer files. For instance an all-in-one install(AIO) then re-run AIO installation instead of providing generated answer-file from first AIO run.
Since mysql is already set up then the password change fails.

That said if this not what's happening for you, please ask the IRC channels or open a different bug.

If you agree with above then I'll close this ticket.

Thanks,
Gilles

Comment 13 Tzach Shefi 2014-05-21 06:39:08 UTC
Hi Gilles, 

Sure thing close ticket, if prescript fails again I'll revisit this, hopefully with more details. 

Thanks

Comment 14 bkopilov 2014-11-05 18:04:35 UTC
Hi , 
I got same issue , like packstack installation gets stuck.
Somehow it looks like it is stucked on ssh login . 
I saw it under ps auxf , 

I set workaround that solved my problem with RHEL7 with rhos 5 .
Before running packstack :
echo Host * > /root/.ssh/config
echo '   StrictHostKeyChecking no' >> /root/.ssh/config
echo 'UserKnownHostsFile=/dev/null' >> /root/.ssh/config



Thanks , 
Benny

Comment 15 bkopilov 2014-11-05 18:05:03 UTC
Hi , 
I got same issue , packstack installation gets stuck.
Somehow it looks like it is stucked on ssh login . 
I saw it under ps auxf , 

I set workaround that solved my problem with RHEL7 with rhos 5 .
Before running packstack :
echo Host * > /root/.ssh/config
echo '   StrictHostKeyChecking no' >> /root/.ssh/config
echo 'UserKnownHostsFile=/dev/null' >> /root/.ssh/config



Thanks , 
Benny


Note You need to log in before you can comment on or make changes to this bug.