Bug 1463375
| Summary: | rhosp-director: after rebooting the undercloud node iptables fails to start properly. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> | ||||
| Component: | rhosp-director | Assignee: | Ben Nemec <bnemec> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Gurenko Alex <agurenko> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 12.0 (Pike) | CC: | agurenko, ahrechan, aschultz, bfournie, bnemec, dbecker, emacchi, mburns, mcornea, morazi, rhel-osp-director-maint, sasha, tvignaud | ||||
| Target Milestone: | --- | Keywords: | Triaged, ZStream | ||||
| Target Release: | 12.0 (Pike) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-02-19 15:17:52 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Alexander Chuzhoy
2017-06-20 16:44:32 UTC
Related to http://bugzilla.redhat.com/show_bug.cgi?id=1463227 ? Hmm, I'm not seeing this behavior: [cloud-user@undercloud-rhos ~]$ rpm -qa | grep instack instack-undercloud-7.0.1-0.20170609013145.el7ost.noarch [cloud-user@undercloud-rhos ~]$ sudo iptables -L -t nat [snip] Chain POSTROUTING (policy ACCEPT) target prot opt source destination BOOTSTACK_MASQ all -- anywhere anywhere nova-api-POSTROUTING all -- anywhere anywhere nova-postrouting-bottom all -- anywhere anywhere MASQUERADE all -- 172.17.0.0/16 anywhere Chain BOOTSTACK_MASQ (1 references) target prot opt source destination MASQUERADE all -- 192.168.24.0/24 !192.168.24.0/24 [rest of output snipped] I used the supplied undercloud.conf, except that I had to change to eth1 since eth0 on my VM is the one providing external connectivity. I suppose it's possible that is relevant here. Could you provide the ~/.instack/install-undercloud.log file so I can see what the install is doing with iptables? Hmm. Doesn't always reproduce - I didn't see it on my last deployment. Will add the log once I catch it again. Created attachment 1292072 [details]
install-undercloud.log
Strange. The commands being run to set up iptables in that log are exactly the same as the ones in my deployment where the rules are correctly configured. I don't see any reason that the rule would be missing. The only tiny difference I see is the ordering of the other three POSTROUTING rules. In the bad deployment they are ordered like this: nova-api-POSTROUTING all -- anywhere anywhere MASQUERADE all -- 172.17.0.0/16 anywhere nova-postrouting-bottom all -- anywhere anywhere Whereas in mine they are: nova-api-POSTROUTING all -- anywhere anywhere nova-postrouting-bottom all -- anywhere anywhere MASQUERADE all -- 172.17.0.0/16 anywhere Maybe there's a race in setting up the rules in this chain that causes the undercloud masquerade rule to get dropped? I don't know why that would be the case though. :-/ Is there any external tooling in these environments that might be making changes to the iptables rules after the undercloud deployment completes? Maybe something that adds a masquerade rule for the overcloud external network? Hi, Please check iptables service status. We have the similar issue for deployment osp12, updating osp12, updating & upgrading osp11. https://bugzilla.redhat.com/show_bug.cgi?id=1460116 https://bugzilla.redhat.com/show_bug.cgi?id=1463227 Well, with iptables (vs firewalld), it's only a matter of having the rule. BTW, the w/a I use in automation is a single command re-adding that nat rule. (undercloud) [stack@undercloud-0 ~]$ sudo service iptables status Redirecting to /bin/systemctl status iptables.service ● iptables.service - IPv4 firewall with iptables Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-07-27 10:04:58 EDT; 2h 20min ago Process: 567 ExecStart=/usr/libexec/iptables/iptables.init start (code=exited, status=1/FAILURE) Main PID: 567 (code=exited, status=1/FAILURE) Jul 27 10:04:58 undercloud-0.redhat.local systemd[1]: Starting IPv4 firewall with iptables... Jul 27 10:04:58 undercloud-0.redhat.local iptables.init[567]: iptables: Applying firewall rules: Another app is currently holding the xtables lock. Perhaps you want to use the -w option? Jul 27 10:04:58 undercloud-0.redhat.local iptables.init[567]: [FAILED] Jul 27 10:04:58 undercloud-0.redhat.local systemd[1]: iptables.service: main process exited, code=exited, status=1/FAILURE Jul 27 10:04:58 undercloud-0.redhat.local systemd[1]: Failed to start IPv4 firewall with iptables. Jul 27 10:04:58 undercloud-0.redhat.local systemd[1]: Unit iptables.service entered failed state. Jul 27 10:04:58 undercloud-0.redhat.local systemd[1]: iptables.service failed. Rebooted the undercloud and the service was in failed state again. Chaning the title accordingly. (undercloud) [stack@undercloud-0 ~]$ sudo iptables -t nat -L Chain PREROUTING (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 172.17.0.0/16 anywhere Chain DOCKER (2 references) target prot opt source destination RETURN all -- anywhere anywhere (undercloud) [stack@undercloud-0 ~]$ sudo service iptables start Redirecting to /bin/systemctl start iptables.service (undercloud) [stack@undercloud-0 ~]$ sudo iptables -t nat -L Chain PREROUTING (policy ACCEPT) target prot opt source destination REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 8775 DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT) target prot opt source destination BOOTSTACK_MASQ all -- anywhere anywhere MASQUERADE all -- 172.17.0.0/16 anywhere Chain BOOTSTACK_MASQ (1 references) target prot opt source destination MASQUERADE all -- 192.168.24.0/24 !192.168.24.0/24 Chain DOCKER (2 references) target prot opt source destination RETURN all -- anywhere anywhere Is this 7.4? I want to say there's another bug for iptables not starting correctly on the reboot but I need to find it 7.4 yes. Clean up - closing this as it is a duplicate. *** This bug has been marked as a duplicate of bug 1465382 *** |