Bug 1570539
| Summary: | Fail to upgrade ocp with htpasswd auth at task [openshift_control_plane : verify API server] | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | liujia <jiajliu> |
| Component: | Cluster Version Operator | Assignee: | Michael Gugino <mgugino> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | liujia <jiajliu> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.10.0 | CC: | aos-bugs, jiajliu, jkaur, jokerman, mmccomas, sdodson, vlaad, wmeng |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | 3.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-10-08 14:00:53 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
We're going to have to enforce that the htpasswd file exist in /etc/origin/master *** This bug has been marked as a duplicate of bug 1565447 *** Re-open the bug because upgrade against ocp with htpasswd auth still failed at task [Run variable sanity checks] **********************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/sanity_checks.yml:13
Thursday 07 June 2018 10:14:16 +0000 (0:00:00.044) 0:02:55.937 *********
fatal: [x]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jliu-r39p-master-etcd-nfs-1.0607-wxn.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set. Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."}
But for htpasswd, original resolution should be that htpasswd file was moved to mounted path /etc/origin/master/ by installer during upgrade, which was fixed in https://bugzilla.redhat.com/show_bug.cgi?id=1570935#c7(Scenario2). Seems this check should skip oauthConfig.identityProviders?
Re-open to have a confirm about above issue.
liujia, Can you please provide your inventory file? (In reply to Scott Dodson from comment #3) > liujia, > > Can you please provide your inventory file? In attachment now. Hi Scott Could u give a confirmed result about the question in comment2 before code freeze? Because the default action for htpasswd seems not clear according to the two bugs. *** Bug 1607039 has been marked as a duplicate of this bug. *** PR Created in 3.10 (only applicable branch) https://github.com/openshift/openshift-ansible/pull/9444 Should be in openshift-ansible-3.10.28-1 Verified on openshift-ansible-3.10.45-1.git.0.5aef941.el7.noarch
Before upgrade:
[root@ip-172-18-5-98 master]# pwd
/etc/origin/master
[root@ip-172-18-5-98 master]# ls -la|grep htp
[root@ip-172-18-5-98 master]# cat /etc/origin/master/master-config.yaml|grep htpasswd
name: htpasswd_auth
file: /etc/origin/htpasswd
Upgrade succeed.
[root@ip-172-18-5-98 master]# pwd
/etc/origin/master
[root@ip-172-18-5-98 master]# ls -la|grep htp
-rw-------. 1 root root 14 Sep 12 04:45 htpasswd
[root@ip-172-18-5-98 master]# cat /etc/origin/master/master-config.yaml|grep htpasswd
name: htpasswd_auth
file: /etc/origin/master/htpasswd
|
Description of problem: Upgrade ocp with htpasswd auth. Upgrade failed at task [openshift_control_plane : verify API server]. TASK [openshift_control_plane : verify API server] ************************************************************************************************************************** FAILED - RETRYING: verify API server (120 retries left). ... FAILED - RETRYING: verify API server (1 retries left). fatal: [x.x.x.x]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "--silent", "--tlsv1.2", "--cacert", "/etc/origin/master/ca-bundle.crt", "https://qe-jliu-rp39-master-etcd-1:8443/healthz/ready"], "delta": "0:00:00.012390", "end": "2018-04-22 23:15:12.521547", "msg": "non-zero return code", "rc": 7, "start": "2018-04-22 23:15:12.509157", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} # docker ps |grep master c34c274f240c registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10 "/usr/bin/pod" 26 minutes ago Up 26 minutes k8s_POD_master-controllers-qe-jliu-rp39-master-etcd-1_kube-system_c931705001eb0c6a7f44e6409a695270_0 ff9a44e9e89f registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10 "/usr/bin/pod" 26 minutes ago Up 26 minutes k8s_POD_master-api-qe-jliu-rp39-master-etcd-1_kube-system_69fd8cf417ec055a66ce2ec660ab3dcc_0 # /usr/local/bin/master-logs api api W0423 03:14:58.031291 1 start_master.go:270] Warning: kubernetesMasterConfig.apiServerArguments[runtime-config][0]: Invalid value: "apis/settings.k8s.io/v1alpha1=true": remove the apis/ prefix, master start will continue. Invalid MasterConfig /etc/origin/master/master-config.yaml oauthConfig.identityProvider[0].provider.file: Invalid value: "/etc/origin/htpasswd": could not read file: stat /etc/origin/htpasswd: no such file or directory Version-Release number of the following components: openshift-ansible-3.10.0-0.27.0.git.0.abed3b7.el7.noarch How reproducible: always Steps to Reproduce: 1. Install ocp v3.9 with htpasswd auth openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}] openshift_master_htpasswd_users={'xxx' : 'xxx'} 2. Upgrade above ocp 3. Actual results: Upgrade failed when verify master api server. Expected results: Upgrade should succeed when use htpasswd auth. Additional info: Please attach logs from ansible-playbook with the -vvv flag