Bug 1570539 - Fail to upgrade ocp with htpasswd auth at task [openshift_control_plane : verify API server]
Summary: Fail to upgrade ocp with htpasswd auth at task [openshift_control_plane : ver...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.10.z
Assignee: Michael Gugino
QA Contact: liujia
: 1607039 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2018-04-23 07:25 UTC by liujia
Modified: 2018-10-08 14:00 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-10-08 14:00:53 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description liujia 2018-04-23 07:25:34 UTC
Description of problem:
Upgrade ocp with htpasswd auth. Upgrade failed at task [openshift_control_plane : verify API server].
TASK [openshift_control_plane : verify API server] **************************************************************************************************************************
FAILED - RETRYING: verify API server (120 retries left).
FAILED - RETRYING: verify API server (1 retries left).
fatal: [x.x.x.x]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "--silent", "--tlsv1.2", "--cacert", "/etc/origin/master/ca-bundle.crt", "https://qe-jliu-rp39-master-etcd-1:8443/healthz/ready"], "delta": "0:00:00.012390", "end": "2018-04-22 23:15:12.521547", "msg": "non-zero return code", "rc": 7, "start": "2018-04-22 23:15:12.509157", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

# docker ps |grep master
c34c274f240c        registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10                                                                      "/usr/bin/pod"           26 minutes ago      Up 26 minutes                           k8s_POD_master-controllers-qe-jliu-rp39-master-etcd-1_kube-system_c931705001eb0c6a7f44e6409a695270_0
ff9a44e9e89f        registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10                                                                      "/usr/bin/pod"           26 minutes ago      Up 26 minutes                           k8s_POD_master-api-qe-jliu-rp39-master-etcd-1_kube-system_69fd8cf417ec055a66ce2ec660ab3dcc_0

# /usr/local/bin/master-logs api api
W0423 03:14:58.031291       1 start_master.go:270] Warning: kubernetesMasterConfig.apiServerArguments[runtime-config][0]: Invalid value: "apis/settings.k8s.io/v1alpha1=true": remove the apis/ prefix, master start will continue.
Invalid MasterConfig /etc/origin/master/master-config.yaml
  oauthConfig.identityProvider[0].provider.file: Invalid value: "/etc/origin/htpasswd": could not read file: stat /etc/origin/htpasswd: no such file or directory

Version-Release number of the following components:

How reproducible:

Steps to Reproduce:
1. Install ocp v3.9 with htpasswd auth
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}]
openshift_master_htpasswd_users={'xxx' : 'xxx'}

2. Upgrade above ocp

Actual results:
Upgrade failed when verify master api server.

Expected results:
Upgrade should succeed when use htpasswd auth.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2018-04-23 20:52:29 UTC
We're going to have to enforce that the htpasswd file exist in /etc/origin/master

*** This bug has been marked as a duplicate of bug 1565447 ***

Comment 2 liujia 2018-06-07 10:21:45 UTC
Re-open the bug because upgrade against ocp with htpasswd auth still failed at task [Run variable sanity checks] **********************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/sanity_checks.yml:13
Thursday 07 June 2018  10:14:16 +0000 (0:00:00.044)       0:02:55.937 ********* 
fatal: [x]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jliu-r39p-master-etcd-nfs-1.0607-wxn.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set.  Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."}

But for htpasswd, original resolution should be that htpasswd file was moved to mounted path /etc/origin/master/ by installer during upgrade, which was fixed in https://bugzilla.redhat.com/show_bug.cgi?id=1570935#c7(Scenario2). Seems this check should skip oauthConfig.identityProviders?

Re-open to have a confirm about above issue.

Comment 3 Scott Dodson 2018-06-07 14:00:43 UTC

Can you please provide your inventory file?

Comment 5 liujia 2018-06-08 00:52:36 UTC
(In reply to Scott Dodson from comment #3)
> liujia,
> Can you please provide your inventory file?

In attachment now.

Comment 6 liujia 2018-06-27 09:35:18 UTC
Hi Scott

Could u give a confirmed result about the question in comment2 before code freeze?  Because the default action for htpasswd seems not clear according to the two bugs.

Comment 7 Scott Dodson 2018-07-23 14:50:24 UTC
*** Bug 1607039 has been marked as a duplicate of this bug. ***

Comment 8 Michael Gugino 2018-08-06 17:10:53 UTC
PR Created in 3.10 (only applicable branch) https://github.com/openshift/openshift-ansible/pull/9444

Comment 9 Scott Dodson 2018-08-14 21:40:11 UTC
Should be in openshift-ansible-3.10.28-1

Comment 12 liujia 2018-09-12 09:33:10 UTC
Verified on openshift-ansible-3.10.45-1.git.0.5aef941.el7.noarch

Before upgrade:
[root@ip-172-18-5-98 master]# pwd
[root@ip-172-18-5-98 master]# ls -la|grep htp
[root@ip-172-18-5-98 master]# cat /etc/origin/master/master-config.yaml|grep htpasswd
    name: htpasswd_auth
      file: /etc/origin/htpasswd

Upgrade succeed.
[root@ip-172-18-5-98 master]# pwd
[root@ip-172-18-5-98 master]# ls -la|grep htp
-rw-------. 1 root root     14 Sep 12 04:45 htpasswd
[root@ip-172-18-5-98 master]# cat /etc/origin/master/master-config.yaml|grep htpasswd
    name: htpasswd_auth
      file: /etc/origin/master/htpasswd

Note You need to log in before you can comment on or make changes to this bug.