Description of problem: Upgrade ocp with htpasswd auth. Upgrade failed at task [openshift_control_plane : verify API server]. TASK [openshift_control_plane : verify API server] ************************************************************************************************************************** FAILED - RETRYING: verify API server (120 retries left). ... FAILED - RETRYING: verify API server (1 retries left). fatal: [x.x.x.x]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "--silent", "--tlsv1.2", "--cacert", "/etc/origin/master/ca-bundle.crt", "https://qe-jliu-rp39-master-etcd-1:8443/healthz/ready"], "delta": "0:00:00.012390", "end": "2018-04-22 23:15:12.521547", "msg": "non-zero return code", "rc": 7, "start": "2018-04-22 23:15:12.509157", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} # docker ps |grep master c34c274f240c registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10 "/usr/bin/pod" 26 minutes ago Up 26 minutes k8s_POD_master-controllers-qe-jliu-rp39-master-etcd-1_kube-system_c931705001eb0c6a7f44e6409a695270_0 ff9a44e9e89f registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10 "/usr/bin/pod" 26 minutes ago Up 26 minutes k8s_POD_master-api-qe-jliu-rp39-master-etcd-1_kube-system_69fd8cf417ec055a66ce2ec660ab3dcc_0 # /usr/local/bin/master-logs api api W0423 03:14:58.031291 1 start_master.go:270] Warning: kubernetesMasterConfig.apiServerArguments[runtime-config][0]: Invalid value: "apis/settings.k8s.io/v1alpha1=true": remove the apis/ prefix, master start will continue. Invalid MasterConfig /etc/origin/master/master-config.yaml oauthConfig.identityProvider[0].provider.file: Invalid value: "/etc/origin/htpasswd": could not read file: stat /etc/origin/htpasswd: no such file or directory Version-Release number of the following components: openshift-ansible-3.10.0-0.27.0.git.0.abed3b7.el7.noarch How reproducible: always Steps to Reproduce: 1. Install ocp v3.9 with htpasswd auth openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}] openshift_master_htpasswd_users={'xxx' : 'xxx'} 2. Upgrade above ocp 3. Actual results: Upgrade failed when verify master api server. Expected results: Upgrade should succeed when use htpasswd auth. Additional info: Please attach logs from ansible-playbook with the -vvv flag
We're going to have to enforce that the htpasswd file exist in /etc/origin/master *** This bug has been marked as a duplicate of bug 1565447 ***
Re-open the bug because upgrade against ocp with htpasswd auth still failed at task [Run variable sanity checks] ********************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/init/sanity_checks.yml:13 Thursday 07 June 2018 10:14:16 +0000 (0:00:00.044) 0:02:55.937 ********* fatal: [x]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jliu-r39p-master-etcd-nfs-1.0607-wxn.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set. Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."} But for htpasswd, original resolution should be that htpasswd file was moved to mounted path /etc/origin/master/ by installer during upgrade, which was fixed in https://bugzilla.redhat.com/show_bug.cgi?id=1570935#c7(Scenario2). Seems this check should skip oauthConfig.identityProviders? Re-open to have a confirm about above issue.
liujia, Can you please provide your inventory file?
(In reply to Scott Dodson from comment #3) > liujia, > > Can you please provide your inventory file? In attachment now.
Hi Scott Could u give a confirmed result about the question in comment2 before code freeze? Because the default action for htpasswd seems not clear according to the two bugs.
*** Bug 1607039 has been marked as a duplicate of this bug. ***
PR Created in 3.10 (only applicable branch) https://github.com/openshift/openshift-ansible/pull/9444
Should be in openshift-ansible-3.10.28-1
Verified on openshift-ansible-3.10.45-1.git.0.5aef941.el7.noarch Before upgrade: [root@ip-172-18-5-98 master]# pwd /etc/origin/master [root@ip-172-18-5-98 master]# ls -la|grep htp [root@ip-172-18-5-98 master]# cat /etc/origin/master/master-config.yaml|grep htpasswd name: htpasswd_auth file: /etc/origin/htpasswd Upgrade succeed. [root@ip-172-18-5-98 master]# pwd /etc/origin/master [root@ip-172-18-5-98 master]# ls -la|grep htp -rw-------. 1 root root 14 Sep 12 04:45 htpasswd [root@ip-172-18-5-98 master]# cat /etc/origin/master/master-config.yaml|grep htpasswd name: htpasswd_auth file: /etc/origin/master/htpasswd