Bug 1588707
| Summary: | Upgrade playbook fails on play Upgrade Service Catalog | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Vikas Laad <vlaad> | ||||
| Component: | Cluster Version Operator | Assignee: | Russell Teague <rteague> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | liujia <jiajliu> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.10.0 | CC: | aos-bugs, jokerman, mark.vinkx, mifiedle, mmccomas, rteague, xtian | ||||
| Target Milestone: | --- | Keywords: | Regression, TestBlocker | ||||
| Target Release: | 3.10.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
If dnsmasq was restarted, it was not being properly reconfigured by the SDN pod for local cluster name resolution. The unnecessary dnsmasq restart was removed from the upgrade playbook. The underlying problem of why dnsmasq was not getting reconfigured was updated in the linked related bug for container-selinux.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-12-20 21:37:16 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1585511 | ||||||
| Attachments: |
|
||||||
Created attachment 1448811 [details]
ansible log with -vvv
Reproduced the same on 3.10 to 3.10 upgrade. $ curl -k https://apiserver.kube-service-catalog.svc/healthz curl: (6) Could not resolve host: apiserver.kube-service-catalog.svc; Unknown error Testing [1] to see if it fixes the name resolution issue. [1] https://github.com/openshift/openshift-ansible/pull/8692 I am afraid the adding of "server=/cluster.local/127.0.0.1" would cause other critical issue, such as: BZ#1578720. This is happening because dnsmasq is restarted. We're going to test eliminating the dnsmasq restart during the upgrade however it's not reasonable to require that dnsmasq is never restarted. In 3.9 the node service was responsible for periodically ensuring that dnsmasq is configured to route requests for 'cluster.local' to the node for lookup. This doesn't appear to happen in 3.10. When we encounter this problem we can delete the sdn pod on the affected node and when it's re-created the dnsmasq configuration changes are applied. Moving this to networking team. dnsmasq restart has been removed in [1] however SDN should still properly configure dnsmasq if it is restarted. [1] https://github.com/openshift/openshift-ansible/pull/8716 We are refreshing and watching dnsmasq from the sdn pod -- https://github.com/openshift/origin/blob/master/pkg/dns/dnsmasq.go#L66 I could not reproduce it with: systemctl restart dnsmasq.service; dig kubernetes.default.svc.cluster.local. (while the sdn pod was running) We can no longer reproduce this, either with or without restarting dnsmasq. Moving back to Upgrade and to ON_QA I completed 2 upgrades on HA cluster both completed fine. I verified with latest git hash a1634c352a0ebc4476c9d961a74f2c3817ad35e8 from openshift-ansible It's an selinux problem. With setenforce 0 the problem goes away. When in enforcing mode I see: # dbus-monitor --monitor --address unix:path=/var/run/dbus/system_bus_socket ... signal time=1528915583.677059 sender=:1.310 -> destination=(null destination) serial=3 path=/uk/org/thekelleys/dnsmasq; interface=uk.org.thekelleys.dnsmasq; member=Up error time=1528915583.677070 sender=org.freedesktop.DBus -> destination=:1.310 error_name=org.freedesktop.DBus.Error.AccessDenied reply_serial=3 string "An SELinux policy prevents this sender from sending this message to this recipient, 0 matched rules; type="signal", sender=":1.310" (uid=0 pid=16705 comm="/usr/sbin/dnsmasq -k ") interface="uk.org.thekelleys.dnsmasq" member="Up" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (uid=0 pid=22466 comm="openshift start network --config=/etc/origin/node/")" When I run: systemctl restart dnsmasq.service Related container-selinux bug: https://bugzilla.redhat.com/show_bug.cgi?id=1591281 |
Description of problem: Upgrade from 3.9 to 3.10 fails with following nodes in the cluster 1 lb 3 masters 3 etcd (separate) 2 infra 2 computes 2018-06-07 18:03:36,898 p=23091 u=root | FAILED - RETRYING: Verify that the catalog api server is running (60 retries left).Result was: { "attempts": 1, "changed": false, "cmd": [ "curl", "-k", "https://apiserver.kube-service-catalog.svc/healthz" ], "delta": "0:00:00.260400", "end": "2018-06-07 18:03:36.872203", "failed": true, "invocation": { "module_args": { "_raw_params": "curl -k https://apiserver.kube-service-catalog.svc/healthz", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "warn": false } }, "msg": "non-zero return code", "rc": 6, "retries": 61, "start": "2018-06-07 18:03:36.611803", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: apiserver.kube-service-catalog.svc; Unknown error", "stderr_lines": [ " % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: apiserver.kube-service-catalog.svc; Unknown error" ], "stdout": "", "stdout_lines": [] } Version-Release number of the following components: rpm -q openshift-ansible openshift-ansible-3.10.0-0.63.0.git.0.961c60d.el7.noarch rpm -q ansible ansible-2.4.4.0-1.el7ae.noarch ansible --version ansible 2.4.4.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 4 2018, 09:38:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-34)] Steps to Reproduce: 1. create 3.9 cluster with above nodes 2. run upgrade playbook to 3.10 Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Upgrade fails Expected results: Upgrade should complete. Additional info: Please attach logs from ansible-playbook with the -vvv flag