Bug 1505897 - OCP 3.7.0-0.176.0 install failed: SDN node startup failed: failed to set up iptables: failed to ensure rule [-s 172.20.0.0/14 -m comment --comment masque
Summary: OCP 3.7.0-0.176.0 install failed: SDN node startup failed: failed to set up...
Keywords:
Status: CLOSED DUPLICATE of bug 1501855
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.7.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Ben Bennett
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-24 14:09 UTC by Mike Fiedler
Modified: 2017-10-25 05:54 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-10-25 05:54:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Mike Fiedler 2017-10-24 14:09:55 UTC
Description of problem:

Error response from daemon: No such container: atomic-openshift-node
ft-node.service: main process exited, code=exited, status=255/n/a
6]: Try `iptables -h' or 'iptables --help' for more information.
6]: F1024 14:04:17.254476   48257 network.go:45] SDN node startup failed: failed to set up iptables: failed to ensure rule [-s 172.20.0.0/14 -m comment --comment masque
6]: I1024 14:04:17.252388   48257 manager.go:211] Machine: {NumCores:2 CpuFrequency:2400096 MemoryCapacity:8201400320 MachineID:29978383fb6d4aca98ac4bdaa28d967a SystemU
6]: I1024 14:04:17.248556   48257 fs.go:124] Filesystem partitions: map[/dev/mapper/atomicos-root:{mountpoint:/rootfs major:253 minor:0 fsType:xfs blockSize:0} /dev/map
6]: W1024 14:04:17.236928   48257 manager.go:161] unable to connect to CRI-O api service: Get http://%2Fvar%2Frun%2Fcrio.sock/info: dial unix /var/run/crio.sock: connec
6]: W1024 14:04:17.236812   48257 manager.go:152] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connecti
6]: I1024 14:04:17.226501   48257 manager.go:144] cAdvisor running in container: "/system.slice/docker-d7b20b4e06958233f9470e93e67b4806730938ff9125ba4e51f0832811212125.
6]: I1024 14:04:17.221363   48257 server.go:652] cloud provider determined current node name to be ip-172-31-0-23.us-west-2.compute.internal
6]: I1024 14:04:17.220600   48257 feature_gate.go:144] feature gates: map[]
6]: I1024 14:04:17.220030   48257 node.go:310] Starting openshift-sdn network plugin
6]: I1024 14:04:17.207615   48257 node.go:109] Connecting to Docker at unix:///var/run/docker.sock
6]: I1024 14:04:17.205077   48257 client.go:92] Start docker client with request timeout=2m0s
6]: I1024 14:04:17.204767   48257 client.go:72] Connecting to docker on unix:///var/run/docker.sock
6]: I1024 14:04:17.204687   48257 start_node.go:469] Starting node ip-172-31-0-23.us-west-2.compute.internal (v3.7.0-0.176.0)
6]: I1024 14:04:17.203950   48257 node_config.go:138] Successfully initialized cloud provider: "aws" from the config file: "/etc/origin/cloudprovider/aws.conf"
6]: I1024 14:04:17.203883   48257 tags.go:76] AWS cloud filtering on ClusterID: mffiedler
6]: I1024 14:04:17.100996   48257 aws.go:806] Building AWS cloudprovider
6]: W1024 14:04:17.088796   48257 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
6]: I1024 14:04:17.086437   48257 client.go:92] Start docker client with request timeout=2m0s
6]: I1024 14:04:17.086415   48257 client.go:72] Connecting to docker on unix:///var/run/docker.sock
6]: I1024 14:04:17.086346   48257 server.go:143] Running kubelet in containerized mode (experimental)
6]: I1024 14:04:17.086322   48257 mount_linux.go:192] Detected OS with systemd
6]: I1024 14:04:17.073100   48257 node.go:151] Initializing SDN node of type "redhat/openshift-ovs-networkpolicy" with configured hostname "ip-172-31-0-23.us-west-2.com
6]: W1024 14:04:17.070074   48257 server.go:190] WARNING: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. Please begin using a 
6]: I1024 14:04:17.068363   48257 start_node.go:288] Reading node configuration from /etc/origin/node/node-config.yaml
3]: Error response from daemon: No such container: atomic-openshift-node


Version-Release number of the following components: 3.7.0-0.176.0

How reproducible: Alwats

Steps to Reproduce:
1.  Install OCP with the inventory below (AWS creds redacted)


Actual results:

Install fails - Node fails to start with errors above

[OSEv3:children]
masters
nodes

etcd





[OSEv3:vars]

#The following parameters is used by post-actions
iaas_name=AWS
use_rpm_playbook=true
openshift_playbook_rpm_repos=[{'id': 'aos-playbook-rpm', 'name': 'aos-playbook-rpm', 'baseurl': 'http://download.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.7/latest/x86_64/os', 'enabled': 1, 'gpgcheck': 0}]




update_is_images_url=registry.ops.openshift.com











#The following parameters is used by openshift-ansible
ansible_ssh_user=root




openshift_cloudprovider_kind=aws

openshift_cloudprovider_aws_access_key=<redacted>


openshift_cloudprovider_aws_secret_key=<redacted>










openshift_master_default_subdomain_enable=true
openshift_master_default_subdomain=apps.1024-0zs.qe.rhcloud.com




openshift_auth_type=allowall

openshift_master_identity_providers=[{'name': 'allow_all', 'login': 'true', 'challenge': 'true', 'kind': 'AllowAllPasswordIdentityProvider'}]



deployment_type=openshift-enterprise
openshift_cockpit_deployer_prefix=registry.ops.openshift.com/openshift3/
osm_cockpit_plugins=['cockpit-kubernetes']
osm_use_cockpit=false
oreg_url=registry.ops.openshift.com/openshift3/ose-${component}:${version}
openshift_docker_additional_registries=registry.ops.openshift.com
openshift_docker_insecure_registries=registry.ops.openshift.com
openshift_docker_options=--log-opt max-size=10M --log-opt max-file=3 --signature-verification=false
use_cluster_metrics=true
openshift_master_cluster_method=native
openshift_master_dynamic_provisioning_enabled=true
osm_default_node_selector=region=primary
openshift_disable_check=disk_availability,memory_availability
openshift_master_portal_net=172.24.0.0/14
openshift_portal_net=172.24.0.0/14
osm_cluster_network_cidr=172.20.0.0/14
osm_host_subnet_length=9
openshift_node_kubelet_args={"pods-per-core": ["0"], "max-pods": ["510"],"minimum-container-ttl-duration": ["10s"], "maximum-dead-containers-per-container": ["1"], "maximum-dead-containers": ["20"], "image-gc-high-threshold": ["80"], "image-gc-low-threshold": ["70"]}
openshift_registry_selector="region=infra,zone=default"
openshift_hosted_router_selector="region=infra,zone=default"
openshift_hosted_router_registryurl=registry.ops.openshift.com/openshift3/ose-${component}:${version}
debug_level=2
openshift_set_hostname=true
openshift_override_hostname_check=true
os_sdn_network_plugin_name=redhat/openshift-ovs-networkpolicy
openshift_hosted_router_replicas=1
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_storage_provider=s3
openshift_hosted_registry_storage_s3_accesskey=<redacted>
openshift_hosted_registry_storage_s3_secretkey=<redacted>
openshift_hosted_registry_storage_s3_bucket=aoe-svt-test
openshift_hosted_registry_storage_s3_region=us-west-2
openshift_hosted_registry_replicas=1
openshift_metrics_install_metrics=false
openshift_metrics_image_prefix=registry.ops.openshift.com/openshift3/
openshift_metrics_image_version=v3.7.0
openshift_metrics_cassandra_storage_type=dynamic
openshift_metrics_cassandra_pvc_size=25Gi
openshift_logging_install_logging=false
openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/
openshift_logging_image_version=v3.7.0
openshift_logging_storage_kind=dynamic
openshift_logging_es_pvc_size=50Gi
openshift_logging_es_pvc_dynamic=true
openshift_clusterid=mffiedler
system_images_registry=registry.ops.openshift.com
openshift_image_tag=v3.7.0




[lb]


[etcd]
ec2-54-186-29-38.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-29-38.us-west-2.compute.amazonaws.com


[masters]
ec2-54-186-29-38.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-29-38.us-west-2.compute.amazonaws.com



[nodes]
ec2-54-186-29-38.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-29-38.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_scheduleable=false

ec2-54-186-90-79.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-90-79.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

ec2-54-186-90-79.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-90-79.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

ec2-54-186-196-188.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-196-188.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
ec2-54-212-223-7.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-212-223-7.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

Comment 1 Mike Fiedler 2017-10-24 14:16:43 UTC
Log entries are journalctl -r   ...  most recent first

Comment 2 Meng Bo 2017-10-25 05:54:47 UTC
It's regression which introduced by fixing this bz 1501855

And a new fix has been merged in latest OCP build 3.7.0-0.177.0

Close this bug.

*** This bug has been marked as a duplicate of bug 1501855 ***


Note You need to log in before you can comment on or make changes to this bug.