Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1065322

Summary: host deploy fails because iptables cannot be stopped while trying to unload kernel modules
Product: [oVirt] ovirt-host-deploy Reporter: Jiri Belka <jbelka>
Component: Plugins.VDSMAssignee: Alon Bar-Lev <alonbl>
Status: CLOSED WORKSFORME QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 1.1.0CC: acathrow, alonbl, bazulay, bugs, dougsland, gklein, iheim, jbelka, Rhev-m-bugs, srevivo, yeylon
Target Milestone: ---Flags: jbelka: devel_ack?
Target Release: 1.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-10 10:40:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
setup logs none

Description Jiri Belka 2014-02-14 11:05:19 UTC
Created attachment 863216 [details]
setup logs

Description of problem:

RHEL host is installed and has _only_ RHN channel registered. Relevant channels to RHEV:

rhel-x86_64-rhev-mgmt-agent-6
rhel-x86_64-server-6

The RHEL is updated to latest rpms, then it is being added from Admin Portal into RHEV env. But the installation fails:

...
2014-02-14 11:15:20 DEBUG otopi.plugins.otopi.services.rhel plugin.executeRaw:364 execute-result: ('/sbin/service', 'iptables', 'stop'), rc=1
2014-02-14 11:15:20 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:412 execute-output: ('/sbin/service', 'iptables', 'stop') stdout:
iptables: Setting chains to policy ACCEPT: nat mangle filter [  OK  ]
iptables: Flushing firewall rules: [  OK  ]
iptables: Unloading modules:  ip_tables[FAILED]

2014-02-14 11:15:20 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:417 execute-output: ('/sbin/service', 'iptables', 'stop') stderr:


2014-02-14 11:15:20 DEBUG otopi.context context._executeMethod:130 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-jc2cALxday/pythonlib/otopi/context.py", line 120, in _executeMethod
    method['method']()
  File "/tmp/ovirt-jc2cALxday/otopi-plugins/otopi/network/iptables.py", line 111, in _closeup
    self.services.state('iptables', False)
  File "/tmp/ovirt-jc2cALxday/otopi-plugins/otopi/services/rhel.py", line 184, in state
    'start' if state else 'stop'
  File "/tmp/ovirt-jc2cALxday/otopi-plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand
    raiseOnError=raiseOnError
  File "/tmp/ovirt-jc2cALxday/pythonlib/otopi/plugin.py", line 422, in execute
    command=args[0],
RuntimeError: Command '/sbin/service' failed to execute
2014-02-14 11:15:20 ERROR otopi.context context._executeMethod:139 Failed to execute stage 'Closing up': Command '/sbin/service' failed to execute
...

Interesting that after couple of removing (Remove button) of the failed host from setup and re-adding it again (New button), no success but then I tried to 'service iptables restart' and re-added it again it it passed, reboot of the host appeared and the host was set up after some time.

Version-Release number of selected component (if applicable):
vdsm-python-4.13.2-0.9.el6ev.x86_64
vdsm-python-cpopen-4.13.2-0.9.el6ev.x86_64
vdsm-cli-4.13.2-0.9.el6ev.noarch
vdsm-4.13.2-0.9.el6ev.x86_64
vdsm-xmlrpc-4.13.2-0.9.el6ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. install RHEL from RHN with the channels written above, no other repos on the system!
2. add into 3.2 setup rhevm-3.2.5-0.49.el6ev.noarch
3.

Actual results:
fail, fail, fail, (passed??)

Expected results:
pass

Additional info:

Comment 1 Alon Bar-Lev 2014-02-14 11:59:47 UTC
this is probably a bug in iptables, not sure why you open it for rhev product...

service stop iptables should succeed.

Comment 2 Jiri Belka 2014-02-14 12:35:10 UTC
It seems to me that problem can be found here:

  /sbin/initctl', 'status', 'iptables'), rc=1

# /sbin/initctl status iptables ; echo $?
initctl: Unknown job: iptables
1

Not sure why "you" start iptables with '/sbin/service' but "you" query status with "/sbin/initctl", when '/sbin/service' has status as well.

Comment 3 Alon Bar-Lev 2014-02-14 17:24:45 UTC
(In reply to Jiri Belka from comment #2)
> It seems to me that problem can be found here:
> 
>   /sbin/initctl', 'status', 'iptables'), rc=1
> 
> # /sbin/initctl status iptables ; echo $?
> initctl: Unknown job: iptables
> 1

this is perfectly ok, then it falls back to sysv.

> 
> Not sure why "you" start iptables with '/sbin/service' but "you" query
> status with "/sbin/initctl", when '/sbin/service' has status as well.

The problem per what you wrote in commen#0:

2014-02-14 11:15:20 DEBUG otopi.plugins.otopi.services.rhel plugin.executeRaw:364 execute-result: ('/sbin/service', 'iptables', 'stop'), rc=1
2014-02-14 11:15:20 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:412 execute-output: ('/sbin/service', 'iptables', 'stop') stdout:
iptables: Setting chains to policy ACCEPT: nat mangle filter [  OK  ]
iptables: Flushing firewall rules: [  OK  ]
iptables: Unloading modules:  ip_tables[FAILED]

Please open a regression bug for iptables.

Comment 4 Alon Bar-Lev 2014-02-14 18:45:17 UTC
"""
The Problem you have is: Some open connection depending on the iptables modles. Mean,(this i what i think, not so sure) there is some open connection that is route via iptables. Disabling iptables would mean to interrupt this connection. 
"""[1]

Related: bug#442335, bug@313051, bug#212839.

This behavior was not changed from our side since rhev-3.2.

[1] http://www.linux.org/threads/iptables-problem-help-wanted.3211/

Comment 5 Alon Bar-Lev 2014-02-14 18:47:23 UTC
Not sure it is a valid workaround.

"""
Setting IPTABLES_MODULES_UNLOAD="no" in /etc/sysconfig/iptables-config works for me.
"""[1]

[1] https://www.centos.org/forums/viewtopic.php?t=9045

Comment 6 Barak 2014-03-03 19:26:01 UTC
(In reply to Alon Bar-Lev from comment #5)
> Not sure it is a valid workaround.
> 
> """
> Setting IPTABLES_MODULES_UNLOAD="no" in /etc/sysconfig/iptables-config works
> for me.
> """[1]
> 
> [1] https://www.centos.org/forums/viewtopic.php?t=9045

Not sure we should do that, 
I don't see any bug on iptables.

Comment 7 Alon Bar-Lev 2014-03-03 19:32:17 UTC
(In reply to Barak from comment #6)
> (In reply to Alon Bar-Lev from comment #5)
> > Not sure it is a valid workaround.
> > 
> > """
> > Setting IPTABLES_MODULES_UNLOAD="no" in /etc/sysconfig/iptables-config works
> > for me.
> > """[1]
> > 
> > [1] https://www.centos.org/forums/viewtopic.php?t=9045
> 
> Not sure we should do that, 
> I don't see any bug on iptables.

There were few in the past (unrelated to us), and apart from this single report we have not gotten any other report, if we see more we should open a bug against iptables.

Comment 10 Jiri Belka 2014-03-10 10:40:45 UTC
I can't reproduce:

# rpm -qa vdsm\* iptables\*
iptables-ipv6-1.4.7-11.el6.x86_64
vdsm-cli-4.13.2-0.11.el6ev.noarch
vdsm-xmlrpc-4.13.2-0.11.el6ev.noarch
vdsm-4.13.2-0.11.el6ev.x86_64
iptables-1.4.7-11.el6.x86_64
vdsm-python-4.13.2-0.11.el6ev.x86_64

...
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       2014-03-10 11:34:48 DEBUG otopi.plugins.otopi.services.rhel plugin.executeRaw:347 execute
: ('/sbin/service', 'iptables', 'stop'), executable='None', cwd='None', env=None
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       2014-03-10 11:34:48 DEBUG otopi.plugins.otopi.services.rhel plugin.executeRaw:364 execute
-result: ('/sbin/service', 'iptables', 'stop'), rc=0
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       2014-03-10 11:34:48 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:412 execute-ou
tput: ('/sbin/service', 'iptables', 'stop') stdout:
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       iptables: Setting chains to policy ACCEPT: nat mangle filter [  OK  ]
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       iptables: Flushing firewall rules: [  OK  ]
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       iptables: Unloading modules: [  OK  ]
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       2014-03-10 11:34:48 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:417 execute-ou
tput: ('/sbin/service', 'iptables', 'stop') stderr:
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       
2014-03-10 11:34:49 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND       
...