Description of problem: RDO production, f21 Found here: https://prod-rdojenkins.rhcloud.com/view/RDO-Juno/job/khaleesi-rdo-juno-production-fedora-21-aio-packstack-neutron-ml2-vxlan-rabbitmq-tempest-rpm-minimal/lastBuild/artifact/khaleesi/collected_files/rdo-pksk-j9fpz-rhos-ci-27-controller.tar.bz2/*view*/ Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0] https://www.redhat.com/archives/rdo-list/2014-September/msg00089.html ERROR: 2015-02-04 18:59:07::DEBUG::sequences::40::root:: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 38, in run self.function(config, messages) File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 240, in apply_puppet_manifest wait_for_puppet(currently_running, messages) File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 112, in wait_for_puppet validate_logfile(log) File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 95, in validate_logfile raise PuppetError(message) PuppetError: Error appeared during Puppet run: 192.168.30.229_neutron.pp Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0][0m You will find full trace in log /var/tmp/packstack/20150204-183005-jynqWf/manifests/192.168.30.229_neutron.pp.log 2015-02-04 18:59:07::ERROR::run_setup::967::root:: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 962, in main _main(options, confFile, logFile) File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 620, in _main runSequences() File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 591, in runSequences controller.runAllSequences() File "/usr/lib/python2.7/site-packages/packstack/installer/setup_controller.py", line 68, in runAllSequences sequence.run(config=self.CONF, messages=self.MESSAGES) File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 98, in run step.run(config=config, messages=messages) File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 38, in run self.function(config, messages) File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 240, in apply_puppet_manifest wait_for_puppet(currently_running, messages) File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 112, in wait_for_puppet validate_logfile(log) File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 95, in validate_logfile raise PuppetError(message) PuppetError: Error appeared during Puppet run: 192.168.30.229_neutron.pp Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0][0m You will find full trace in log /var/tmp/packstack/20150204-183005-jynqWf/manifests/192.168.30.229_neutron.pp.log
The actual error is: Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[sysctl_refresh]/returns: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[sysctl_refresh]/returns: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[sysctl_refresh]/returns: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0]
Those sysctls are provided by the 'bridge' kernel module. We probably need to make sure that the 'bridge' module is loaded first. Calling "brctl show" is sufficient to do this.
Weird, the neutron::bridge module explicitly loads the 'bridge' kernel module, and that was successful just a few lines before those errors: Debug: Exec[load-bridge](provider=posix): Executing 'modprobe -b bridge' Debug: Executing 'modprobe -b bridge' Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[load-bridge]/returns: executed successfully
That seems to have started happening after kernel was updated from 3.17 to 3.18 in Fedora 21 (from internal job history: sorry, no public link!)
[root@f21clean ~]# uname -r 3.18.3-201.fc21.x86_64 [root@f21clean ~]# modprobe -b bridge [root@f21clean ~]# lsmod|grep bridge bridge 107941 0 stp 12868 1 bridge llc 13941 2 stp,bridge [root@f21clean ~]# ls /proc/sys/net/bridge/ ls: cannot access /proc/sys/net/bridge/: No such file or directory
It looks like in recent kernels the sysctls are provided by the 'br_netfilter' module. # modprobe bridge # ls /proc/sys/net core ipv4 ipv6 netfilter unix # modprobe br_netfilter # ls /proc/sys/net bridge core ipv4 ipv6 netfilter unix # ls /proc/sys/net/bridge bridge-nf-call-arptables bridge-nf-filter-pppoe-tagged bridge-nf-call-ip6tables bridge-nf-filter-vlan-tagged bridge-nf-call-iptables bridge-nf-pass-vlan-input-dev
I proposed a fix for this: https://review.openstack.org/#/c/153393/
So that's a duplicate of bug 1175460 which was supposed to be fixed in openstack-packstack-2014.2-0.13.dev1395.gaabe0a2.fc22 but https://review.openstack.org/143088 checks for Fedora 22+ instead of kernel version! I'll reopen 1175460
*** Bug 1175460 has been marked as a duplicate of this bug. ***
Actually vice-versa is better, proposed fix is better, thanks Lars!
This update is now urgent, both F20 and F21 are now affected.
https://repos.fedorapeople.org/repos/openstack/openstack-juno/fedora-21/openstack-puppet-modules-2014.2.10-1.fc22.noarch.rpm https://repos.fedorapeople.org/repos/openstack/openstack-juno/fedora-21/openstack-packstack-2014.2-0.16.dev1447.g6f4d34b.fc22.noarch.rpm