Bug 1767160

Summary: Overcloud deployment for spine/leaf stack fails setting firewall rules
Product: Red Hat OpenStack Reporter: Alistair Tonner <atonner>
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Yogev Rabl <yrabl>
Severity: high Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: cjeanner, dhill, gcharot, gfidente, jfrancoa, jjoyce, jschluet, lbezdick, lmiccini, mburns, michele, rszmigie, slinaber, tvignaud
Target Milestone: z4Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190513171772.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-20 16:49:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1770345    

Description Alistair Tonner 2019-10-30 19:48:29 UTC
Description of problem:

 Deploying OSP 14 spine and leaf stack for testing, failure during configuration of firewall rules:

ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_ADDR: 172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24 Permanent and Non-Permanent(immediate) operation


Version-Release number of selected component (if applicable):

openstack-tripleo-validations-9.3.2-0.20190420045628.361061f.el7ost.noarch
openstack-tripleo-common-9.5.0-8.el7ost.noarch
python2-tripleo-common-9.5.0-8.el7ost.noarch
python-tripleoclient-10.6.2-0.20190425150607.el7ost.noarch
openstack-tripleo-common-containers-9.5.0-8.el7ost.noarch
openstack-tripleo-image-elements-9.0.1-0.20181102144447.9f1c800.el7ost.noarch
python-tripleoclient-heat-installer-10.6.2-0.20190425150607.el7ost.noarch
openstack-tripleo-puppet-elements-9.0.1-5.el7ost.noarch
ceph-ansible-3.2.33-1.el7cp.noarch
openstack-heat-engine-11.0.3-0.20190420005637.df958c9.el7ost.noarch



How reproducible:

Consistent

Steps to Reproduce:
1.  https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-14_director-rhel-7.7-virthost-3cont_6comp_6ceph-yes_UC_SSL-yes_OC_SSL-ceph-ipv4-vxlan-localregistry-spineleaf-20180627-1731/
and
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-14_director-rhel-7.7-virthost-3cont_6comp_6ceph-yes_UC_SSL-yes_OC_SSL-ceph-ipv4-vxlan-localregistry-spineleaf-predictableip-20180627-1731/

2.
3.

Actual results:

Job fails in rqci_oc_deploy with iptables failure.

Expected results:

Successful deployment of OC with spine/leaf configuration and testing

Additional info:

Comment 2 Cédric Jeanneret 2019-10-31 10:15:19 UTC
Hello Alistair,

I tried to find the log line you mention in order to get some more context, but.... nothing in stack home, and apparently nothing matches that in /var/log directory. Care to provide some more info?

Thank you!

Cheers,

C.

Comment 3 Alistair Tonner 2019-10-31 10:46:54 UTC
Cedric:
     This can be seen in var/lib/mistral/overcloud/ansible.log
  
 A full block for one of the OC nodes is:

"<192.168.24.23> (1, '\\n{\"msg\": \"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_ADDR: 172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24 Permanent and Non-Permanent(immediate) operation\", \"failed\": true, \"exception\": \"  File \\\\\"/tmp/ansible_g8NKeT/ansible_modlib.zip/ansible/module_utils/firewalld.py\\\\\", line 103, in action_handler\\\\n    return action_func(*action_func_args)\\\\n  File \\\\\"/tmp/ansible_g8NKeT/ansible_module_firewalld.py\\\\\", line 464, in set_enabled_permanent\\\\n    self.update_fw_settings(fw_zone, fw_settings)\\\\n  File \\\\\"/tmp/ansible_g8NKeT/ansible_modlib.zip/ansible/module_utils/firewalld.py\\\\\", line 134, in update_fw_settings\\\\n    fw_zone.update(fw_settings)\\\\n  File \\\\\"<string>\\\\\", line 2, in update\\\\n  File \\\\\"/usr/lib/python2.7/site-packages/slip/dbus/polkit.py\\\\\", line 103, in _enable_proxy\\\\n    return func(*p, **k)\\\\n  File \\\\\"<string>\\\\\", line 2, in update\\\\n  File \\\\\"/usr/lib/python2.7/site-packages/firewall/client.py\\\\\", line 53, in handle_exceptions\\\\n    return func(*args, **kwargs)\\\\n  File \\\\\"/usr/lib/python2.7/site-packages/firewall/client.py\\\\\", line 441, in update\\\\n    self.fw_zone.update(tuple(settings.settings))\\\\n  File \\\\\"/usr/lib/python2.7/site-packages/slip/dbus/proxies.py\\\\\", line 50, in __call__\\\\n    return dbus.proxies._ProxyMethod.__call__(self, *args, **kwargs)\\\\n  File \\\\\"/usr/lib64/python2.7/site-packages/dbus/proxies.py\\\\\", line 145, in __call__\\\\n    **keywords)\\\\n  File \\\\\"/usr/lib64/python2.7/site-packages/dbus/connection.py\\\\\", line 651, in call_blocking\\\\n    message, timeout)\\\\n\", \"invocation\": {\"module_args\": {\"service\": \"ceph\", \"zone\": \"public\", \"masquerade\": null, \"immediate\": true, \"source\": \"172.120.3.0/24,172.117.3.0/24,172.118.3.0/24,172.119.3.0/24\", \"state\": \"enabled\", \"permanent\": true, \"timeout\": 0, \"interface\": null, \"offline\": null, \"port\": null, \"rich_rule\": null}}}\\n', '')",

Comment 4 Cédric Jeanneret 2019-10-31 10:51:08 UTC
Hello Alistair,

Ah, damn, it didn't show up with my `grep -r'... Thank you!
Will investigate deeper, but it might be related to ceph-ansible.

Stay tuned!

C.

Comment 8 Luca Miccini 2019-11-27 14:48:26 UTC
*** Bug 1777318 has been marked as a duplicate of this bug. ***

Comment 9 Jose Luis Franco 2019-12-02 13:48:06 UTC
*** Bug 1777773 has been marked as a duplicate of this bug. ***

Comment 21 Yogev Rabl 2019-12-16 14:15:42 UTC
*** Bug 1783522 has been marked as a duplicate of this bug. ***

Comment 23 errata-xmlrpc 2019-12-20 16:49:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4339