Bug 1122302

Summary: Rubygem-Staypuft: Deployment of nonHA neutron (GRE) gets stuck installing the networker.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-puppet-modulesAssignee: Martin Magr <mmagr>
Status: CLOSED DUPLICATE QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: aberezin, acathrow, lars, mburns, morazi, ohochman, sclewis, yeylon
Target Milestone: rc   
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-31 12:10:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/messages file from the networker. none

Description Alexander Chuzhoy 2014-07-22 21:42:46 UTC
Rubygem-Staypuft:  Deployment of nonHA neutron (GRE) gets stuck installing the networker.

Environment:
openstack-puppet-modules-2014.1-19.1.el6ost.noarch
openstack-foreman-installer-2.0.15-1.el6ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
rhel-osp-installer-0.1.1-1.el6ost.noarch


Steps to reproduce:
1. Install rhel-osp-installer.
2. Create/start a deployment of nonHA neutron GRE/rabbitmq (1 controller+1 networker+1 compute).

Result:
The deployment gets paused with error on 60% installing the networker.
Running the puppet agent manually on the networker shows the following:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Local ip for ovs agent must be set when tunneling is enabled at /etc/puppet/environments/production/modules/neutron/manifests/agents/ovs.pp:32 on node a25400868093.example.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Expected result: successful deployment.

Comment 1 Alexander Chuzhoy 2014-07-22 21:46:32 UTC
Created attachment 920061 [details]
/var/log/messages file from the networker.

Comment 3 Alexander Chuzhoy 2014-07-23 18:06:18 UTC
Got the following results using puddle 2014-07-22.1


The deployment got stuck on 60% installing the networker.
running the puppet agent manually on the networker - completed with no errors.
Resuming the deployment resulted in the following message in UI:
Controller (Neutron) 100%, Neutron Networker 100%, Compute (Neutron) 5%
execution expired.
And then the deployment got paused again.

Comment 4 Alexander Chuzhoy 2014-07-23 20:05:02 UTC
This patch was applied in attempt to overcome the issue:
https://review.openstack.org/#/c/94504/




Running the deployment after applying the patch - the deployment got paused with error on 60% installing the networker. Further investigation revealed the following:

1. the pupet run failed on: Could not start Service[iptables]: Execution of '/usr/bin/systemctl start iptables' returned 1: Job for iptables.service failed. See 'systemctl status iptables.service' and 'journalctl -xn' for details. Wrapped exception: Execution of '/usr/bin/systemctl start iptables' returned 1: Job for iptables.service failed. See 'systemctl status iptables.service' and 'journalctl -xn' for details.

2. checking the puppet - shows this:

http://pastebin.test.redhat.com/223624

Comment 5 Lars Kellogg-Stedman 2014-07-23 20:06:33 UTC
I'm moving the pastebin into the bz because it's small:

[root@maca25400654fdd ~]# journalctl -fu iptables
-- Logs begin at Wed 2014-07-23 19:23:41 UTC. --
Jul 23 19:27:11 maca25400654fdd.example.com systemd[1]: Starting IPv4 firewall with iptables...
Jul 23 19:27:11 maca25400654fdd.example.com iptables.init[10712]: iptables: Applying firewall rules: iptables-restore: line 14 failed
Jul 23 19:27:11 maca25400654fdd.example.com iptables.init[10712]: [FAILED]
Jul 23 19:27:11 maca25400654fdd.example.com systemd[1]: iptables.service: main process exited, code=exited, status=1/FAILURE
Jul 23 19:27:11 maca25400654fdd.example.com systemd[1]: Failed to start IPv4 firewall with iptables.
Jul 23 19:27:11 maca25400654fdd.example.com systemd[1]: Unit iptables.service entered failed state.
Jul 23 19:38:08 maca25400654fdd.example.com systemd[1]: Starting IPv4 firewall with iptables...
Jul 23 19:38:08 maca25400654fdd.example.com iptables.init[12410]: iptables: Applying firewall rules: [  OK  ]
Jul 23 19:38:08 maca25400654fdd.example.com systemd[1]: Started IPv4 firewall with iptables.

Comment 6 Alexander Chuzhoy 2014-07-23 20:09:53 UTC
Note: Successful start of iptables as seen in comment #5 is a result of manual puppet agent run, following the paused deployment.

Comment 7 Mike Orazi 2014-07-25 20:32:32 UTC
Can we reproduce this reliably?

Comment 9 Mike Burns 2014-07-31 12:10:01 UTC
This issue is the same as bug 1123393 and the problem arises due to RHEL OSP installer not having true network management.

*** This bug has been marked as a duplicate of bug 1123393 ***