Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1565447 - installation fail due to HTPasswdPassword file is not mounted into master static pod
Summary: installation fail due to HTPasswdPassword file is not mounted into master sta...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 3.10.0
Assignee: Michael Gugino
QA Contact: Johnny Liu
URL:
Whiteboard:
: 1624031 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-10 03:32 UTC by Johnny Liu
Modified: 2018-10-24 13:49 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-24 13:49:59 UTC
Target Upstream Version:


Attachments (Terms of Use)
installation log with inventory file embedded (921.90 KB, text/plain)
2018-04-23 08:10 UTC, Johnny Liu
no flags Details

Description Johnny Liu 2018-04-10 03:32:05 UTC
Description of problem:
When user use htpaswd auth for master, and locate the htpasswd file on some host path, but this path is never mounted into master pod, then static master pod is never stated.


Version-Release number of the following components:
openshift-ansible-3.10.0-0.16.0.git.0.8925606.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Define the following parameters in inventory file
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}]
openshift_master_htpasswd_users={'jialiu' : '$xxx$ccc$xxx'}
2. Trigger a rpm install.
3.

Actual results:
TASK [openshift_control_plane : Verify that the control plane is running] ******
Monday 09 April 2018  22:38:45 -0400 (0:00:00.025)       0:07:12.519 ********** 
FAILED - RETRYING: Verify that the control plane is running (60 retries left).
<--snip-->
FAILED - RETRYING: Verify that the control plane is running (1 retries left).

fatal: [qe-jialiu310-master-etcd-1.0409-fx9.qe.rhcloud.com]: FAILED! => {"attempts": 60, "changed": false, "cmd": ["curl", "-k", "https://qe-jialiu310-master-etcd-1:8443/healthz/ready"], "delta": "0:00:00.011817", "end": "2018-04-09 22:44:57.660087", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2018-04-09 22:44:57.648270", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to qe-jialiu310-master-etcd-1:8443; Connection refused", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to qe-jialiu310-master-etcd-1:8443; Connection refused"], "stdout": "", "stdout_lines": []}


Go to master, the htpasswd file is not mounted into master container.
# docker ps -a
CONTAINER ID        IMAGE                                                                                                           COMMAND                  CREATED             STATUS                        PORTS               NAMES
2f7f147a61c9        a5bb4dd94b69                                                                                                    "/bin/bash -c '#!/..."   34 seconds ago      Exited (255) 30 seconds ago                       k8s_api_master-api-qe-jialiu310-master-etcd-1_kube-system_00848990aa0cb639cb3c3eb6de95bf9b_11
6ea20b9caf9f        a5bb4dd94b69                                                                                                    "/bin/bash -c '#!/..."   47 seconds ago      Exited (255) 41 seconds ago                       k8s_controllers_master-controllers-qe-jialiu310-master-etcd-1_kube-system_cdffaf37286b04f878217a337815cd3f_11

# docker logs 7002d74f052b
<--snip-->
I0410 03:02:57.089955       1 plugins.go:83] Registered admission plugin "PVCProtection"
Invalid MasterConfig /etc/origin/master/master-config.yaml
  oauthConfig.identityProvider[0].provider.file: Invalid value: "/etc/origin/htpasswd": could not read file: stat /etc/origin/htpasswd: no such file or directory


# ll /etc/origin/htpasswd
-rw-------. 1 root root 45 Apr  9 22:39 /etc/origin/htpasswd

# docker inspect 2f7f147a61c9
<--snip-->
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/origin",
                "Destination": "/var/lib/origin",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/origin/openshift.local.volumes/pods/00848990aa0cb639cb3c3eb6de95bf9b/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/origin/openshift.local.volumes/pods/00848990aa0cb639cb3c3eb6de95bf9b/containers/api/7388b049",
                "Destination": "/dev/termination-log",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/origin/master",
                "Destination": "/etc/origin/master",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/origin/cloudprovider",
                "Destination": "/etc/origin/cloudprovider",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
<--snip-->

Expected results:
Master pod is started and installation should move on.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2018-04-19 13:19:42 UTC
I think that we should mount /etc/origin rather than /etc/origin/master, /etc/origin/cloudprovider and then we need to document and/or remove the ability to place these files outside of /etc/origin.

Need to make sure that this makes into release notes, as folks upgrading may need to relocate files into /etc/origin

Comment 2 Michael Gugino 2018-04-19 18:53:50 UTC
PR Created: https://github.com/openshift/openshift-ansible/pull/8047

Comment 3 Johnny Liu 2018-04-23 08:09:44 UTC
Seem like the PR is already merged into openshift-ansible-3.10.0-0.27.0.git.0.abed3b7.el7.noarch, re-run the test, still failed.


openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_users={'jialiu' : 'xxxx'}

Comment 4 Johnny Liu 2018-04-23 08:10:24 UTC
Created attachment 1425573 [details]
installation log with inventory file embedded

Comment 5 Michael Gugino 2018-04-23 16:56:00 UTC
Fix for new error: https://github.com/openshift/openshift-ansible/pull/8087

Comment 6 Scott Dodson 2018-04-23 20:52:29 UTC
*** Bug 1570539 has been marked as a duplicate of this bug. ***

Comment 7 Johnny Liu 2018-04-26 10:44:33 UTC
Seem like the above PR still not fix the issue, will hit another error.

Re-test this with openshift-ansible-3.10.0-0.29.0.git.0.8973ee4.el7.noarch.

openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_users={'jialiu' : 'xxxx'}

master api pod is failed to be started.

# docker logs d0ad95987983
<--snip-->
I0426 10:36:23.469228       1 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection"
Invalid MasterConfig /etc/origin/master/master-config.yaml
  oauthConfig.identityProvider[0].provider.file: Required value

# cat /etc/origin/master/master-config.yaml
<--snip-->
  assetPublicURL: https://ec2-174-129-81-79.compute-1.amazonaws.com:8443/console/
  grantConfig:
    method: auto
  identityProviders:
  - challenge: true
    login: true
    mappingMethod: claim
    name: htpasswd_auth
    provider:
      apiVersion: v1
      filename: /etc/origin/master/htpasswd
      kind: HTPasswdPasswordIdentityProvider

Comment 8 Scott Dodson 2018-04-26 13:14:45 UTC
Just need to change th mutation from 'filename' to 'file'

Comment 9 Michael Gugino 2018-04-26 17:48:28 UTC
PR Created to change to 'file' instead of 'filename': https://github.com/openshift/openshift-ansible/pull/8168

Comment 10 Scott Dodson 2018-05-02 15:52:40 UTC
in openshift-ansible-3.10.0-0.32.0

Comment 11 Johnny Liu 2018-05-03 08:37:55 UTC
Verified this bug with openshift-ansible-3.10.0-0.32.0.git.0.bb50d68.el7.noarch, and PASS.

Set the following parameters in inventory file:
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_users={'jialiu' : 'xxxx'}

After installation:
# cat /etc/origin/master/htpasswd 
jialiu:xxxx

# cat /etc/origin/master/master-config.yaml
<--snip-->
oauthConfig:
  assetPublicURL: https://35.232.68.70:8443/console/
  grantConfig:
    method: auto
  identityProviders:
  - challenge: true
    login: true
    mappingMethod: claim
    name: htpasswd_auth
    provider:
      apiVersion: v1
      file: /etc/origin/master/htpasswd
      kind: HTPasswdPasswordIdentityProvider
<--snip-->

master pod is running well, access web console, it is using htpaswd auth.

Comment 12 Johnny Liu 2018-05-03 08:41:36 UTC
# docker inspect 8201cd4e652d
<--snip-->
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/origin/openshift.local.volumes/pods/092ddb4b3b3b965611387ef19c4e2377/containers/api/529d074e",
                "Destination": "/dev/termination-log",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/origin/master",
                "Destination": "/etc/origin/master",
                "Mode": "rslave",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/etc/origin/cloudprovider",
                "Destination": "/etc/origin/cloudprovider",
                "Mode": "rslave",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/origin",
                "Destination": "/var/lib/origin",
                "Mode": "rslave",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/origin/openshift.local.volumes/pods/092ddb4b3b3b965611387ef19c4e2377/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
<--snip-->

Comment 13 Johnny Liu 2018-05-03 09:33:31 UTC
Seem like the PR is not fixing this issue in the way which is mentioned in comment 1.

scenario 1:
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}]
openshift_master_htpasswd_users={'jialiu' : '$xxx$ccc$xxx'}


Installation log:
TASK [Run variable sanity checks] **********************************************
Thursday 03 May 2018  04:59:25 -0400 (0:00:00.753)       0:00:26.137 ********** 
fatal: [qe-jialiu1-master-etcd-1.0503-wk7.qe.rhcloud.com]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jialiu1-master-etcd-1.0503-wk7.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set.  Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."}

scenario 2:
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/aa'}]

Installation log:
TASK [Run variable sanity checks] **********************************************
Thursday 03 May 2018  05:09:56 -0400 (0:00:00.731)       0:00:26.265 ********** 
fatal: [qe-jialiu111-master-etcd-1.0503-uhx.qe.rhcloud.com]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jialiu111-master-etcd-1.0503-uhx.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set.  Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."}

scenario 3:
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]


Installation log:
TASK [Run variable sanity checks] **********************************************
Thursday 03 May 2018  05:06:25 -0400 (0:00:00.834)       0:00:26.616 ********** 
fatal: [qe-jialiu2-master-etcd-1.0503-mq5.qe.rhcloud.com]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jialiu2-master-etcd-1.0503-mq5.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set.  Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."}


According to the above test result, installer does not allow user specify any "filename" whatever the htpasswd file is located in /etc/origin/htpasswd, or /etc/origin/master/htpasswd. Is that by design?

Re-assign to wait for confirm.

Comment 14 Michael Gugino 2018-05-03 12:49:02 UTC
(In reply to Johnny Liu from comment #13)
> 
> According to the above test result, installer does not allow user specify
> any "filename" whatever the htpasswd file is located in
> /etc/origin/htpasswd, or /etc/origin/master/htpasswd. Is that by design?
> 
> Re-assign to wait for confirm.

Yes, we decided to remove the option of where to place the file.  It will now always be in the same place.

Comment 15 Johnny Liu 2018-05-04 02:46:33 UTC
Per comment 11, move to verified.

Comment 18 Jeremy Whiting 2018-08-31 08:47:37 UTC
*** Bug 1624031 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.