Bug 1529233
Summary: | "Add iptables allow rules" task failed when running system container install with cri-o enabled. | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Johnny Liu <jialiu> |
Component: | Installer | Assignee: | Giuseppe Scrivano <gscrivan> |
Status: | CLOSED ERRATA | QA Contact: | Johnny Liu <jialiu> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 3.7.0 | CC: | aos-bugs, jialiu, jokerman, mmccomas, xtian |
Target Milestone: | --- | Keywords: | Regression, TestBlocker |
Target Release: | 3.7.z | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | No Doc Update | |
Doc Text: |
undefined
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2018-04-05 09:34:33 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Johnny Liu
2017-12-27 08:32:01 UTC
Created attachment 1372708 [details]
installation log with inventory embeded
Giuseppe, do you mind looking at this? It's likely missing the iptables-services package installation which gets triggered by including the os_firewall role, we need to ensure that the playbook that configures crio includes that role prior to including the docker role or potentially just make it a dependency of the docker role. Do you know if the same problem happens on 3.9? There's been significant refactoring there which would have addressed this problem. Unfortunately the fix on 3.7, which i agree with, is running into some integration test issues. (In reply to Scott Dodson from comment #7) > Do you know if the same problem happens on 3.9? There's been significant > refactoring there which would have addressed this problem. > > Unfortunately the fix on 3.7, which i agree with, is running into some > integration test issues. No, when I was reporting this bug on 3.7, 3.9 does not have such problem, just like what you said, 3.9 has a significant refactor. Verified this bug with openshift-ansible-3.7.26-1.git.0.f87f1af.el7.noarch, and PASS. iptable service is installed in the beginning of installation. <--snip--> PLAY [Ensure that all non-node hosts are accessible] *************************** TASK [Gathering Facts] ********************************************************* Friday 26 January 2018 05:18:23 +0000 (0:00:00.040) 0:00:00.738 ******** ok: [ec2-52-90-147-28.compute-1.amazonaws.com] PLAY [Initialize host facts] *************************************************** TASK [Gathering Facts] ********************************************************* Friday 26 January 2018 05:18:24 +0000 (0:00:01.026) 0:00:01.765 ******** ok: [ec2-52-90-147-28.compute-1.amazonaws.com] ok: [ec2-52-202-142-111.compute-1.amazonaws.com] TASK [os_firewall : Detecting Atomic Host Operating System] ******************** Friday 26 January 2018 05:18:25 +0000 (0:00:00.688) 0:00:02.453 ******** ok: [ec2-52-202-142-111.compute-1.amazonaws.com] => {"changed": false, "stat": {"exists": false}} ok: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"changed": false, "stat": {"exists": false}} TASK [os_firewall : Set fact r_os_firewall_is_atomic] ************************** Friday 26 January 2018 05:18:25 +0000 (0:00:00.511) 0:00:02.965 ******** ok: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"ansible_facts": {"r_os_firewall_is_atomic": false}, "changed": false} ok: [ec2-52-202-142-111.compute-1.amazonaws.com] => {"ansible_facts": {"r_os_firewall_is_atomic": false}, "changed": false} <--snip--> TASK [os_firewall : Ensure firewalld service is not enabled] ******************* Friday 26 January 2018 05:18:26 +0000 (0:00:00.039) 0:00:03.306 ******** ok: [ec2-52-202-142-111.compute-1.amazonaws.com] => {"changed": false, "failed_when_result": false, "msg": "Could not find the requested service firewalld: host"} ok: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"changed": false, "failed_when_result": false, "msg": "Could not find the requested service firewalld: host"} TASK [os_firewall : Wait 10 seconds after disabling firewalld] ***************** Friday 26 January 2018 05:18:26 +0000 (0:00:00.710) 0:00:04.016 ******** skipping: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [os_firewall : Install iptables packages] ********************************* Friday 26 January 2018 05:18:26 +0000 (0:00:00.034) 0:00:04.051 ******** ok: [ec2-52-90-147-28.compute-1.amazonaws.com] => (item=iptables) => {"attempts": 1, "changed": false, "item": "iptables", "msg": "", "rc": 0, "results": ["iptables-1.4.21-18.el7.x86_64 providing iptables is already installed"]} ok: [ec2-52-202-142-111.compute-1.amazonaws.com] => (item=iptables) => {"attempts": 1, "changed": false, "item": "iptables", "msg": "", "rc": 0, "results": ["iptables-1.4.21-18.el7.x86_64 providing iptables is already installed"]} changed: [ec2-52-90-147-28.compute-1.amazonaws.com] => (item=iptables-services) => {"attempts": 1, "changed": true, "item": "iptables-services", "msg": "", "rc": 0, "results": ["Loaded plugins: amazon-id, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package iptables-services.x86_64 0:1.4.21-18.2.el7_4 will be installed\n--> Processing Dependency: iptables = 1.4.21-18.2.el7_4 for package: iptables-services-1.4.21-18.2.el7_4.x86_64\n--> Running transaction check\n---> Package iptables.x86_64 0:1.4.21-18.el7 will be updated\n---> Package iptables.x86_64 0:1.4.21-18.2.el7_4 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iptables-services x86_64 1.4.21-18.2.el7_4 oso-rhui-rhel-server-releases 51 k\nUpdating for dependencies:\n iptables x86_64 1.4.21-18.2.el7_4 oso-rhui-rhel-server-releases 428 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\nUpgrade ( 1 Dependent package)\n\nTotal download size: 479 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 1.0 MB/s | 479 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : iptables-1.4.21-18.2.el7_4.x86_64 1/3 \n Installing : iptables-services-1.4.21-18.2.el7_4.x86_64 2/3 \n Cleanup : iptables-1.4.21-18.el7.x86_64 3/3 \n Verifying : iptables-services-1.4.21-18.2.el7_4.x86_64 1/3 \n Verifying : iptables-1.4.21-18.2.el7_4.x86_64 2/3 \n Verifying : iptables-1.4.21-18.el7.x86_64 3/3 \n\nInstalled:\n iptables-services.x86_64 0:1.4.21-18.2.el7_4 \n\nDependency Updated:\n iptables.x86_64 0:1.4.21-18.2.el7_4 \n\nComplete!\n"]} changed: [ec2-52-202-142-111.compute-1.amazonaws.com] => (item=iptables-services) => {"attempts": 1, "changed": true, "item": "iptables-services", "msg": "", "rc": 0, "results": ["Loaded plugins: amazon-id, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package iptables-services.x86_64 0:1.4.21-18.2.el7_4 will be installed\n--> Processing Dependency: iptables = 1.4.21-18.2.el7_4 for package: iptables-services-1.4.21-18.2.el7_4.x86_64\n--> Running transaction check\n---> Package iptables.x86_64 0:1.4.21-18.el7 will be updated\n---> Package iptables.x86_64 0:1.4.21-18.2.el7_4 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iptables-services x86_64 1.4.21-18.2.el7_4 oso-rhui-rhel-server-releases 51 k\nUpdating for dependencies:\n iptables x86_64 1.4.21-18.2.el7_4 oso-rhui-rhel-server-releases 428 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\nUpgrade ( 1 Dependent package)\n\nTotal download size: 479 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 1.0 MB/s | 479 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : iptables-1.4.21-18.2.el7_4.x86_64 1/3 \n Installing : iptables-services-1.4.21-18.2.el7_4.x86_64 2/3 \n Cleanup : iptables-1.4.21-18.el7.x86_64 3/3 \n Verifying : iptables-services-1.4.21-18.2.el7_4.x86_64 1/3 \n Verifying : iptables-1.4.21-18.2.el7_4.x86_64 2/3 \n Verifying : iptables-1.4.21-18.el7.x86_64 3/3 \n\nInstalled:\n iptables-services.x86_64 0:1.4.21-18.2.el7_4 \n\nDependency Updated:\n iptables.x86_64 0:1.4.21-18.2.el7_4 \n\nComplete!\n"]} TASK [os_firewall : Start and enable iptables service] ************************* Friday 26 January 2018 05:18:35 +0000 (0:00:09.014) 0:00:13.066 ******** changed: [ec2-52-90-147-28.compute-1.amazonaws.com -> ec2-52-90-147-28.compute-1.amazonaws.com] => (item=ec2-52-90-147-28.compute-1.amazonaws.com) => {"changed": true, "enabled": true, "item": "ec2-52-90-147-28.compute-1.amazonaws.com", "name": "iptables", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket basic.target syslog.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network.service shutdown.target ip6tables.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "IPv4 firewall with iptables", "DevicePolicy": "auto", "Environment": "BOOTUP=serial CONSOLETYPE=serial", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/libexec/iptables/iptables.init ; argv[]=/usr/libexec/iptables/iptables.init reload ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/libexec/iptables/iptables.init ; argv[]=/usr/libexec/iptables/iptables.init start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/libexec/iptables/iptables.init ; argv[]=/usr/libexec/iptables/iptables.init stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/iptables.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "iptables.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "14236", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "14236", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "iptables.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "syslog", "StandardInput": "null", "StandardOutput": "syslog", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "disabled", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} changed: [ec2-52-90-147-28.compute-1.amazonaws.com -> ec2-52-202-142-111.compute-1.amazonaws.com] => (item=ec2-52-202-142-111.compute-1.amazonaws.com) => {"changed": true, "enabled": true, "item": "ec2-52-202-142-111.compute-1.amazonaws.com", "name": "iptables", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice syslog.target systemd-journald.socket basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network.service ip6tables.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "IPv4 firewall with iptables", "DevicePolicy": "auto", "Environment": "BOOTUP=serial CONSOLETYPE=serial", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/libexec/iptables/iptables.init ; argv[]=/usr/libexec/iptables/iptables.init reload ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/libexec/iptables/iptables.init ; argv[]=/usr/libexec/iptables/iptables.init start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/libexec/iptables/iptables.init ; argv[]=/usr/libexec/iptables/iptables.init stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/iptables.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "iptables.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "14236", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "14236", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "iptables.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "syslog", "StandardInput": "null", "StandardOutput": "syslog", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "disabled", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} TASK [os_firewall : need to pause here, otherwise the iptables service starting can sometimes cause ssh to fail] *** Friday 26 January 2018 05:18:37 +0000 (0:00:01.475) 0:00:14.542 ******** Pausing for 10 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"changed": false, "delta": 10, "rc": 0, "start": "2018-01-26 05:18:37.403396", "stderr": "", "stdout": "Paused for 10.0 seconds", "stop": "2018-01-26 05:18:47.403617", "user_input": ""} TASK [openshift_sanitize_inventory : Check for usage of deprecated variables] *** <--snip--> TASK [docker : Create the CRI-O configuration] ********************************* Friday 26 January 2018 05:21:00 +0000 (0:00:00.668) 0:02:37.358 ******** changed: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"backup_file": "/etc/crio/crio.conf.4408.2018-01-26@00:21:01~", "changed": true, "checksum": "782ce165fac14cbd300bb8c415a81d91bea3b250", "dest": "/etc/crio/crio.conf", "gid": 0, "group": "root", "md5sum": "e0c701d5ac969d27fa5bf8a50a6c8640", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:container_config_t:s0", "size": 5781, "src": "/root/.ansible/tmp/ansible-tmp-1516944060.33-127364211185816/source", "state": "file", "uid": 0} TASK [docker : Ensure CNI configuration directory exists] ********************** Friday 26 January 2018 05:21:01 +0000 (0:00:01.042) 0:02:38.401 ******** changed: [ec2-52-90-147-28.compute-1.amazonaws.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cni/net.d/", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 6, "state": "directory", "uid": 0} TASK [docker : Add iptables allow rules] *************************************** Friday 26 January 2018 05:21:01 +0000 (0:00:00.332) 0:02:38.733 ******** changed: [ec2-52-90-147-28.compute-1.amazonaws.com] => (item={u'port': u'10010/tcp', u'service': u'crio'}) => {"changed": true, "item": {"port": "10010/tcp", "service": "crio"}, "output": ["", "Successfully created chain OS_FIREWALL_ALLOW", "iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]\r\n", "", "iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]\r\n", "", "iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]\r\n"]} Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0636 |