Bug 1577955

Summary: Router pod fails to deploy during installation
Product: OpenShift Container Platform Reporter: Preetesh Sharma <prsharma>
Component: InstallerAssignee: Stephen Cuppett <scuppett>
Status: CLOSED ERRATA QA Contact: Johnny Liu <jialiu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, fshaikh, jokerman, mgugino, mmccomas, rbost, scuppett, sdodson
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-11 16:39:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Preetesh Sharma 2018-05-14 13:55:09 UTC
Description of problem:

when installation task "[openshift_hosted : Poll for OpenShift pod deployment success]" is executed it tries to deploy router pods in all infra region nodes and during this time it also tries to deploy router pods in master nodes, in which by default "infra" region is selected, but as master servers are set to unschedulable these task fails.

Error message in verbose -
~~~
"_raw_params": "oc get replicationcontroller router-20 --namespace default --config /etc/origin/master/admin.kubeconfig -o jsonpath='{ .metadata.annotations.openshift\\.io/deployment\\.phase }'

                "images": "openshift3/ose-${component}:${version}", 
                "name": "router", 
                "namespace": "default", 
                "ports": [
                    "80:80", 
                    "443:443"
                ], 
                "replicas": "5", 
                "selector": "region=infra", 
                "serviceaccount": "router", 
                "stats_port": 1936
~~~

How reproducible: 
Steps to Reproduce:

1. Setup multi master installation with three master and three nodes, assign "infra" region to two nodes in inventory file.
2. Installer will try to deploy router pods on 5 nodes, and will finish with below message.
~~~
Failure summary:


  1. Hosts:    redhat.example.com
     Play:     Create Hosted Resources - router
     Task:     Poll for OpenShift pod deployment success
     Message:  All items completed
~~~

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Get below error message for task - "[openshift_hosted : Poll for OpenShift pod deployment success]".
~~~ 
"_raw_params": "oc get replicationcontroller router-20 --namespace default --config /etc/origin/master/admin.kubeconfig -o jsonpath='{ .metadata.annotations.openshift\\.io/deployment\\.phase }'

                "images": "openshift3/ose-${component}:${version}", 
                "name": "router", 
                "namespace": "default", 
                "ports": [
                    "80:80", 
                    "443:443"
                ], 
                "replicas": "5", 
                "selector": "region=infra", 
                "serviceaccount": "router", 
                "stats_port": 1936
~~~

Expected results: Installation should not try to deploy router pods on master servers, as they are by default set to "unschedulable" due to which router pod deployment fails.

Additional info: We can use variable "openshift_router_selector='region=primary'" as workaround in this case, router pod will only get deployed on node with primary region and installer will not try to deploy them on master nodes.


Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2018-05-14 14:00:01 UTC
Can you please provide the inventory used to build this cluster? Also, please clarify the version of openshift-ansible in use as 3.11 does not exist yet.

Comment 2 Scott Dodson 2018-05-16 14:22:57 UTC
*** Bug 1577959 has been marked as a duplicate of this bug. ***

Comment 8 Scott Dodson 2018-08-02 12:40:17 UTC
Fatima,

Can you clarify the specifics of the situation you're facing with the customer.

In 3.9 masters should be schedulable. Additionally, it's up to you to label the masters as infra so if you prefer not to run these pods on those hosts simply don't mark them with region=infra.

Comment 23 errata-xmlrpc 2018-11-11 16:39:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2709