Bug 1189681

Summary: F20/F21 RDO packstack error: Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0]
Product: [Community] RDO Reporter: wes hayutin <whayutin>
Component: openstack-packstackAssignee: Gaƫl Chamoulaud <gchamoul>
Status: CLOSED CURRENTRELEASE QA Contact: Ami Jeain <ajeain>
Severity: high Docs Contact:
Priority: urgent    
Version: JunoCC: aortega, apevec, apevec, derekh, gchamoul, lars, lbezdick, ltoscano, rjones, yeylon
Target Milestone: ---   
Target Release: Juno   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-packstack-2014.2-0.15.dev1446.gc669c34.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1199082 1199085 (view as bug list) Environment:
Last Closed: 2015-02-16 09:50:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1199082, 1199085    

Description wes hayutin 2015-02-05 14:27:43 UTC
Description of problem:
RDO production, f21

Found here:
https://prod-rdojenkins.rhcloud.com/view/RDO-Juno/job/khaleesi-rdo-juno-production-fedora-21-aio-packstack-neutron-ml2-vxlan-rabbitmq-tempest-rpm-minimal/lastBuild/artifact/khaleesi/collected_files/rdo-pksk-j9fpz-rhos-ci-27-controller.tar.bz2/*view*/

Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0]

https://www.redhat.com/archives/rdo-list/2014-September/msg00089.html



ERROR:
2015-02-04 18:59:07::DEBUG::sequences::40::root:: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 38, in run
    self.function(config, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 240, in apply_puppet_manifest
    wait_for_puppet(currently_running, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 112, in wait_for_puppet
    validate_logfile(log)
  File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 95, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: 192.168.30.229_neutron.pp
Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0][0m
You will find full trace in log /var/tmp/packstack/20150204-183005-jynqWf/manifests/192.168.30.229_neutron.pp.log

2015-02-04 18:59:07::ERROR::run_setup::967::root:: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 962, in main
    _main(options, confFile, logFile)
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 620, in _main
    runSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/run_setup.py", line 591, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.7/site-packages/packstack/installer/setup_controller.py", line 68, in runAllSequences
    sequence.run(config=self.CONF, messages=self.MESSAGES)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 98, in run
    step.run(config=config, messages=messages)
  File "/usr/lib/python2.7/site-packages/packstack/installer/core/sequences.py", line 38, in run
    self.function(config, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 240, in apply_puppet_manifest
    wait_for_puppet(currently_running, messages)
  File "/usr/lib/python2.7/site-packages/packstack/plugins/puppet_950.py", line 112, in wait_for_puppet
    validate_logfile(log)
  File "/usr/lib/python2.7/site-packages/packstack/modules/puppet.py", line 95, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: 192.168.30.229_neutron.pp
Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0][0m
You will find full trace in log /var/tmp/packstack/20150204-183005-jynqWf/manifests/192.168.30.229_neutron.pp.log

Comment 1 Lars Kellogg-Stedman 2015-02-05 14:53:07 UTC
The actual error is:

Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[sysctl_refresh]/returns: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[sysctl_refresh]/returns: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[sysctl_refresh]/returns: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
Error: sysctl -p /etc/sysctl.conf returned 255 instead of one of [0]

Comment 2 Lars Kellogg-Stedman 2015-02-05 15:06:36 UTC
Those sysctls are provided by the 'bridge' kernel module.  We probably need to make sure that the 'bridge' module is loaded first.  Calling "brctl show" is sufficient to do this.

Comment 3 Lars Kellogg-Stedman 2015-02-05 15:15:59 UTC
Weird, the neutron::bridge module explicitly loads the 'bridge' kernel module, and that was successful just a few lines before those errors:

Debug: Exec[load-bridge](provider=posix): Executing 'modprobe -b bridge'
Debug: Executing 'modprobe -b bridge'
Notice: /Stage[main]/Packstack::Neutron::Bridge/Exec[load-bridge]/returns: executed successfully

Comment 4 Alan Pevec (Fedora) 2015-02-05 17:25:13 UTC
That seems to have started happening after kernel was updated from 3.17 to 3.18 in Fedora 21 (from internal job history: sorry, no public link!)

Comment 5 Alan Pevec (Fedora) 2015-02-05 18:04:26 UTC
[root@f21clean ~]# uname -r
3.18.3-201.fc21.x86_64
[root@f21clean ~]# modprobe -b bridge
[root@f21clean ~]# lsmod|grep bridge
bridge                107941  0 
stp                    12868  1 bridge
llc                    13941  2 stp,bridge
[root@f21clean ~]# ls /proc/sys/net/bridge/
ls: cannot access /proc/sys/net/bridge/: No such file or directory

Comment 6 Lars Kellogg-Stedman 2015-02-05 19:42:06 UTC
It looks like in recent kernels the sysctls are provided by the 'br_netfilter' module.

# modprobe bridge
# ls /proc/sys/net
core  ipv4  ipv6  netfilter  unix
# modprobe br_netfilter
# ls /proc/sys/net
bridge  core  ipv4  ipv6  netfilter  unix
# ls /proc/sys/net/bridge
bridge-nf-call-arptables  bridge-nf-filter-pppoe-tagged
bridge-nf-call-ip6tables  bridge-nf-filter-vlan-tagged
bridge-nf-call-iptables   bridge-nf-pass-vlan-input-dev

Comment 7 Lars Kellogg-Stedman 2015-02-05 21:58:29 UTC
I proposed a fix for this:

https://review.openstack.org/#/c/153393/

Comment 8 Alan Pevec (Fedora) 2015-02-05 22:05:27 UTC
So that's a duplicate of bug 1175460 which was supposed to be fixed in openstack-packstack-2014.2-0.13.dev1395.gaabe0a2.fc22
but https://review.openstack.org/143088 checks for Fedora 22+ instead of kernel version!
I'll reopen 1175460

Comment 9 Alan Pevec (Fedora) 2015-02-05 22:11:01 UTC
*** Bug 1175460 has been marked as a duplicate of this bug. ***

Comment 10 Alan Pevec (Fedora) 2015-02-05 22:12:15 UTC
Actually vice-versa is better, proposed fix is better, thanks Lars!

Comment 11 Alan Pevec 2015-02-09 13:27:25 UTC
This update is now urgent, both F20 and F21 are now affected.