Description of problem: ipa auth doesn't work when I put configmap with ipa data and re-deploy httpd pod. When I log into httpd pod, I see initialize-httpd-auth service failed along with network service. sssd service isn't up as well. If I start initialize-httpd-auth service and sssd service and restart apache, ipa ext auth works well Version-Release number of selected component (if applicable): 5.9.0.18 How reproducible: 100% Steps to Reproduce: 1. deploy cfme 2. generate config map with ipa auth data using httpd-configmap-generator 3. replace existing httpd-auth-configs configmap 4. stop httpd pod and rollout it again 5. check whether ipa auth work 6. log into httpd pod and check apache logs and services state Actual results: ipa auth doesn't work, there are some failed services in httpd pod Expected results: no such issue Additional info: I'll attach logs and more details in a bit.
Constantly failing network service: [root@httpd-1-kf1zb httpd]# systemctl status -l network ● network.service - LSB: Bring up/down networking Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2018-01-31 22:06:25 UTC; 14h ago Docs: man:systemd-sysv-generator(8) Process: 20 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE) Jan 31 22:06:25 httpd-1-kf1zb network[20]: RTNETLINK answers: Operation not permitted Jan 31 22:06:25 httpd-1-kf1zb network[20]: RTNETLINK answers: Operation not permitted Jan 31 22:06:25 httpd-1-kf1zb network[20]: RTNETLINK answers: Operation not permitted Jan 31 22:06:25 httpd-1-kf1zb network[20]: RTNETLINK answers: Operation not permitted Jan 31 22:06:25 httpd-1-kf1zb network[20]: RTNETLINK answers: Operation not permitted Jan 31 22:06:25 httpd-1-kf1zb network[20]: RTNETLINK answers: Operation not permitted Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: network.service: control process exited, code=exited status=1 Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: Failed to start LSB: Bring up/down networking. Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: Unit network.service entered failed state. Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: network.service failed. When I deploy or scale down/up dc/httpd, I always see initialize-httpd-auth service failed. [root@httpd-1-kf1zb httpd]# systemctl status -l initialize-httpd-auth ● initialize-httpd-auth.service - Initializes Httpd External Authentication Loaded: loaded (/usr/lib/systemd/system/initialize-httpd-auth.service; enabled; vendor preset: disabled) Active: failed (Result: resources) Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: Failed to load environment files: No such file or directory Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: initialize-httpd-auth.service failed to run 'start-pre' task: No such file or directory Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: Failed to start Initializes Httpd External Authentication. Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: Unit initialize-httpd-auth.service entered failed state. Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: initialize-httpd-auth.service failed. Jan 31 22:06:25 httpd-1-kf1zb systemd[1]: Starting Initializes Httpd External Authentication... I tried to run start-pre script from that service manually. It certainly works sh-4.2# /bin/bash -c "until [ -f /etc/container-environment ]; do sleep 1; > done" sh-4.2# echo $? 0 When I kick off that service manually, it starts w/o issues. [root@httpd-1-kbcz8 httpd]# systemctl start initialize-httpd-auth [root@httpd-1-kbcz8 httpd]# systemctl status -l initialize-httpd-auth ● initialize-httpd-auth.service - Initializes Httpd External Authentication Loaded: loaded (/usr/lib/systemd/system/initialize-httpd-auth.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2018-02-01 13:22:20 UTC; 2s ago Process: 357 ExecStart=/usr/bin/initialize-httpd-auth.sh (code=exited, status=0/SUCCESS) Process: 356 ExecStartPre=/bin/bash -c until [ -f /etc/container-environment ]; do sleep 1; done (code=exited, status=0/SUCCESS) Main PID: 357 (code=exited, status=0/SUCCESS) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: nsswitch.conf => /etc/nsswitch.conf (644:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: password-auth-ac => /etc/pam.d/password-auth-ac (644:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: postlogin-ac => /etc/pam.d/postlogin-ac (644:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: pwdfile.txt => /etc/ipa/nssdb/pwdfile.txt (600:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: secmod.db.base64 => /etc/ipa/nssdb/secmod.db (644:root:root) BINARY Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: smartcard-auth-ac => /etc/pam.d/smartcard-auth-ac (644:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: sssd.conf => /etc/sssd/sssd.conf (600:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Copying: system-auth-ac => /etc/pam.d/system-auth-ac (644:root:root) Feb 01 13:22:20 httpd-1-kbcz8 initialize-httpd-auth.sh[357]: Finished copying Authentication Files. Feb 01 13:22:20 httpd-1-kbcz8 systemd[1]: Started Initializes Httpd External Authentication.
This is working fine for me (minishift). I know for openshift, the oci-systemd-hook must be at least oci-systemd-hook-0.1.8-4.1.gite533efa.el7.x86_64 Could you double check that you have the latest ? If not, you could run "yum update oci-systemd-hook" on any schedulable nodes. Thanks.
[root@cmqe-smicro-628-b09 ~]# yum list oci-systemd-hook Installed oci-systemd-hook.x86_64 1:0.1.14-1.git1ba44c6.el7 @rhel-7-server-extras-rpms Available oci-systemd-hook.x86_64 1:0.1.14-2.git9b1e622.el7 rhel-7-server-extras-rpms
Alberto, btw, I can deploy cfme and give you access to it if you wish. please find me in irc or gitter if so.
Ievgen, The fact that you can manually restart things and they work makes me think you are likely hitting a network / timing / startup issue unrelated to CFME. I will investigate if you could please PM me the creds to the setup. Thank you, JoeV
creds and project with this issue are provided.
To work around this failure log into the httpd_pod and run the following: systemctl restart initialize-httpd-auth.service systemctl status initialize-httpd-auth.service systemctl restart sssd systemctl status sssd systemctl restart httpd systemctl status httpd (Thank you Ievgen!)
verified in 5.9.2.3