Description of problem: When user use htpaswd auth for master, and locate the htpasswd file on some host path, but this path is never mounted into master pod, then static master pod is never stated. Version-Release number of the following components: openshift-ansible-3.10.0-0.16.0.git.0.8925606.el7.noarch How reproducible: Always Steps to Reproduce: 1. Define the following parameters in inventory file openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}] openshift_master_htpasswd_users={'jialiu' : '$xxx$ccc$xxx'} 2. Trigger a rpm install. 3. Actual results: TASK [openshift_control_plane : Verify that the control plane is running] ****** Monday 09 April 2018 22:38:45 -0400 (0:00:00.025) 0:07:12.519 ********** FAILED - RETRYING: Verify that the control plane is running (60 retries left). <--snip--> FAILED - RETRYING: Verify that the control plane is running (1 retries left). fatal: [qe-jialiu310-master-etcd-1.0409-fx9.qe.rhcloud.com]: FAILED! => {"attempts": 60, "changed": false, "cmd": ["curl", "-k", "https://qe-jialiu310-master-etcd-1:8443/healthz/ready"], "delta": "0:00:00.011817", "end": "2018-04-09 22:44:57.660087", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2018-04-09 22:44:57.648270", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to qe-jialiu310-master-etcd-1:8443; Connection refused", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to qe-jialiu310-master-etcd-1:8443; Connection refused"], "stdout": "", "stdout_lines": []} Go to master, the htpasswd file is not mounted into master container. # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2f7f147a61c9 a5bb4dd94b69 "/bin/bash -c '#!/..." 34 seconds ago Exited (255) 30 seconds ago k8s_api_master-api-qe-jialiu310-master-etcd-1_kube-system_00848990aa0cb639cb3c3eb6de95bf9b_11 6ea20b9caf9f a5bb4dd94b69 "/bin/bash -c '#!/..." 47 seconds ago Exited (255) 41 seconds ago k8s_controllers_master-controllers-qe-jialiu310-master-etcd-1_kube-system_cdffaf37286b04f878217a337815cd3f_11 # docker logs 7002d74f052b <--snip--> I0410 03:02:57.089955 1 plugins.go:83] Registered admission plugin "PVCProtection" Invalid MasterConfig /etc/origin/master/master-config.yaml oauthConfig.identityProvider[0].provider.file: Invalid value: "/etc/origin/htpasswd": could not read file: stat /etc/origin/htpasswd: no such file or directory # ll /etc/origin/htpasswd -rw-------. 1 root root 45 Apr 9 22:39 /etc/origin/htpasswd # docker inspect 2f7f147a61c9 <--snip--> "Mounts": [ { "Type": "bind", "Source": "/var/lib/origin", "Destination": "/var/lib/origin", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/origin/openshift.local.volumes/pods/00848990aa0cb639cb3c3eb6de95bf9b/etc-hosts", "Destination": "/etc/hosts", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/origin/openshift.local.volumes/pods/00848990aa0cb639cb3c3eb6de95bf9b/containers/api/7388b049", "Destination": "/dev/termination-log", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/origin/master", "Destination": "/etc/origin/master", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/origin/cloudprovider", "Destination": "/etc/origin/cloudprovider", "Mode": "", "RW": true, "Propagation": "rprivate" } ], <--snip--> Expected results: Master pod is started and installation should move on. Additional info: Please attach logs from ansible-playbook with the -vvv flag
I think that we should mount /etc/origin rather than /etc/origin/master, /etc/origin/cloudprovider and then we need to document and/or remove the ability to place these files outside of /etc/origin. Need to make sure that this makes into release notes, as folks upgrading may need to relocate files into /etc/origin
PR Created: https://github.com/openshift/openshift-ansible/pull/8047
Seem like the PR is already merged into openshift-ansible-3.10.0-0.27.0.git.0.abed3b7.el7.noarch, re-run the test, still failed. openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}] openshift_master_htpasswd_users={'jialiu' : 'xxxx'}
Created attachment 1425573 [details] installation log with inventory file embedded
Fix for new error: https://github.com/openshift/openshift-ansible/pull/8087
*** Bug 1570539 has been marked as a duplicate of this bug. ***
Seem like the above PR still not fix the issue, will hit another error. Re-test this with openshift-ansible-3.10.0-0.29.0.git.0.8973ee4.el7.noarch. openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}] openshift_master_htpasswd_users={'jialiu' : 'xxxx'} master api pod is failed to be started. # docker logs d0ad95987983 <--snip--> I0426 10:36:23.469228 1 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection" Invalid MasterConfig /etc/origin/master/master-config.yaml oauthConfig.identityProvider[0].provider.file: Required value # cat /etc/origin/master/master-config.yaml <--snip--> assetPublicURL: https://ec2-174-129-81-79.compute-1.amazonaws.com:8443/console/ grantConfig: method: auto identityProviders: - challenge: true login: true mappingMethod: claim name: htpasswd_auth provider: apiVersion: v1 filename: /etc/origin/master/htpasswd kind: HTPasswdPasswordIdentityProvider
Just need to change th mutation from 'filename' to 'file'
PR Created to change to 'file' instead of 'filename': https://github.com/openshift/openshift-ansible/pull/8168
in openshift-ansible-3.10.0-0.32.0
Verified this bug with openshift-ansible-3.10.0-0.32.0.git.0.bb50d68.el7.noarch, and PASS. Set the following parameters in inventory file: openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}] openshift_master_htpasswd_users={'jialiu' : 'xxxx'} After installation: # cat /etc/origin/master/htpasswd jialiu:xxxx # cat /etc/origin/master/master-config.yaml <--snip--> oauthConfig: assetPublicURL: https://35.232.68.70:8443/console/ grantConfig: method: auto identityProviders: - challenge: true login: true mappingMethod: claim name: htpasswd_auth provider: apiVersion: v1 file: /etc/origin/master/htpasswd kind: HTPasswdPasswordIdentityProvider <--snip--> master pod is running well, access web console, it is using htpaswd auth.
# docker inspect 8201cd4e652d <--snip--> "Mounts": [ { "Type": "bind", "Source": "/var/lib/origin/openshift.local.volumes/pods/092ddb4b3b3b965611387ef19c4e2377/containers/api/529d074e", "Destination": "/dev/termination-log", "Mode": "Z", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/origin/master", "Destination": "/etc/origin/master", "Mode": "rslave", "RW": true, "Propagation": "rslave" }, { "Type": "bind", "Source": "/etc/origin/cloudprovider", "Destination": "/etc/origin/cloudprovider", "Mode": "rslave", "RW": true, "Propagation": "rslave" }, { "Type": "bind", "Source": "/var/lib/origin", "Destination": "/var/lib/origin", "Mode": "rslave", "RW": true, "Propagation": "rslave" }, { "Type": "bind", "Source": "/var/lib/origin/openshift.local.volumes/pods/092ddb4b3b3b965611387ef19c4e2377/etc-hosts", "Destination": "/etc/hosts", "Mode": "Z", "RW": true, "Propagation": "rprivate" } ], <--snip-->
Seem like the PR is not fixing this issue in the way which is mentioned in comment 1. scenario 1: openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}] openshift_master_htpasswd_users={'jialiu' : '$xxx$ccc$xxx'} Installation log: TASK [Run variable sanity checks] ********************************************** Thursday 03 May 2018 04:59:25 -0400 (0:00:00.753) 0:00:26.137 ********** fatal: [qe-jialiu1-master-etcd-1.0503-wk7.qe.rhcloud.com]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jialiu1-master-etcd-1.0503-wk7.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set. Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."} scenario 2: openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/aa'}] Installation log: TASK [Run variable sanity checks] ********************************************** Thursday 03 May 2018 05:09:56 -0400 (0:00:00.731) 0:00:26.265 ********** fatal: [qe-jialiu111-master-etcd-1.0503-uhx.qe.rhcloud.com]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jialiu111-master-etcd-1.0503-uhx.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set. Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."} scenario 3: openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}] Installation log: TASK [Run variable sanity checks] ********************************************** Thursday 03 May 2018 05:06:25 -0400 (0:00:00.834) 0:00:26.616 ********** fatal: [qe-jialiu2-master-etcd-1.0503-mq5.qe.rhcloud.com]: FAILED! => {"failed": true, "msg": "last_checked_host: qe-jialiu2-master-etcd-1.0503-mq5.qe.rhcloud.com, last_checked_var: openshift_master_manage_htpasswd;openshift_master_identity_providers contains a provider of kind==HTPasswdPasswordIdentityProvider and filename is set. Please migrate your htpasswd files to /etc/origin/master/htpasswd and update your existing master configs, and remove the filename keybefore proceeding."} According to the above test result, installer does not allow user specify any "filename" whatever the htpasswd file is located in /etc/origin/htpasswd, or /etc/origin/master/htpasswd. Is that by design? Re-assign to wait for confirm.
(In reply to Johnny Liu from comment #13) > > According to the above test result, installer does not allow user specify > any "filename" whatever the htpasswd file is located in > /etc/origin/htpasswd, or /etc/origin/master/htpasswd. Is that by design? > > Re-assign to wait for confirm. Yes, we decided to remove the option of where to place the file. It will now always be in the same place.
Per comment 11, move to verified.
*** Bug 1624031 has been marked as a duplicate of this bug. ***