Created attachment 1176281 [details] Some screenshots with the tests we made Description of problem: while configuring the appliance to use an external IPA we get the following error: Configure External Authentication (httpd) IPA Server Parameters: Enter the IPA Server Hostname: ipa.hailstorm3.coe.muc.redhat.com Enter the IPA Server Domain: |hailstorm3.coe.muc.redhat.com| Enter the IPA Server Realm: |HAILSTORM3.COE.MUC.REDHAT.COM| Enter the IPA Server Principal: |admin| Enter the IPA Server Principal Password: ******** External Authentication (httpd) Configuration: IPA Server Details: Hostname: ipa.hailstorm3.coe.muc.redhat.com Domain: hailstorm3.coe.muc.redhat.com Realm: HAILSTORM3.COE.MUC.REDHAT.COM Naming Context: dc=hailstorm3,dc=coe,dc=muc,dc=redhat,dc=com Principal: admin Proceed? (Y/N): y Checking connectivity to ipa.hailstorm3.coe.muc.redhat.com ... Failed. Could not connect to ipa.hailstorm3.coe.muc.redhat.com, the IPA Server must be reachable by name. External Authentication configuration failed! Press any key to continue. When ipa-client-install is configured manually it works providing exactly the same parameters, and the IPA server is also used successfully in a RHEV environment Version-Release number of selected component (if applicable): 5.6.0.13 How reproducible: Deploy a IPA server and configure the appliance to use the external authentication Steps to Reproduce: 1. Install an IPA server 2. Configure CFME to use external authentication 3. Actual results: Failure Expected results: Appliance configured with the IPA server Additional info:
Victor, #1 On the appliance can you please capture the output from the following commands: % nslookup ipa.hailstorm3.coe.muc.redhat.com and % traceroute ipa.hailstorm3.coe.muc.redhat.com and % dig ipa.hailstorm3.coe.muc.redhat.com and % ssh admin.coe.muc.redhat.com using the correct admin password It can't hurt to try but this one may be less interesting as it is possible ssh port is blocked. #2 Please review the documentation and confirm the version requirements for the IPA and Windows Server https://github.com/ManageIQ/manageiq_docs/blob/master/auth/ipa_ad_trust.adoc #3 Please provide the following logs: /var/www/miq/vmdb/log/evm.log /var/www/miq/vmdb/log/audit.log /var/www/miq/vmdb/log/appliance_console.log /var/log/secure* /var/log/sssd/*.log #4 Please provide config file /etc/sssd/sssd.conf from both the system where ipa-client-install was run successfully and on the system where the appliance_console failed Thank you! JoeV
Victor, Can I log into this machine? Can you PM me the creds? JoeV
Victor, I logged in to your system and was able to configure IPA without making any modifications. I simply ran the appliance_console with the credentials you provided me. I will post a private transcript of my session. JoeV
Victor, I'm going to mark this as "CLOSED / WORKSFORME" I'm guessing you may have corrected something when gathering the data I had requested... ??? Please reopen if you think this is still a bug and provide what you see as the issue. Thank you, JoeV
Hi Joe, thank's for taking a look. We are rolling out these environments via Ansible scripts and can therefore more or less reliably recreate the environments. I'll be posting a transcript on the same attempt on the sister environment hailstorm3.
Wolfram, For any failing machine please post a private message providing all the information I requested in https://bugzilla.redhat.com/show_bug.cgi?id=1352822#c3 Including the credentials so I can log in. Thank you, JoeV
Wolfram, I just logged into cloudforms.hailstorm3.coe.muc.redhat.com and successfully configure IPA. --- abridged transcript on hailstorm3 Configure External Authentication (httpd) IPA Server Parameters: Enter the IPA Server Hostname: ipa.hailstorm3.coe.muc.redhat.com Enter the IPA Server Domain: |hailstorm3.coe.muc.redhat.com| Enter the IPA Server Realm: |HAILSTORM3.COE.MUC.REDHAT.COM| Enter the IPA Server Principal: |admin| Enter the IPA Server Principal Password: ******** External Authentication (httpd) Configuration: IPA Server Details: Hostname: ipa.hailstorm3.coe.muc.redhat.com Domain: hailstorm3.coe.muc.redhat.com Realm: HAILSTORM3.COE.MUC.REDHAT.COM Naming Context: dc=hailstorm3,dc=coe,dc=muc,dc=redhat,dc=com Principal: admin Proceed? (Y/N): y Checking connectivity to ipa.hailstorm3.coe.muc.redhat.com ... Succeeded. Configuring IPA (may take a minute) ... Configuring the IPA Client ... Configuring pam ... Configuring sssd ... Configuring IPA HTTP Service and Keytab ... Configuring httpd ... Configuring SELinux ... Restarting sssd and httpd ... Configuring sssd to start upon reboots ... External Authentication configured successfully. Press any key to continue. --- Let's plan to get on a BlueJeans call ASAP. JoeV
In the a BlueJeans call we've determined that the problem occurs when the appliance console tries to ping the IPA server via Net::Ping::External, which uses the Linux ping command. This command in turn fails when I invoke it, but succeeds when overriding the locale settings: [root@cloudforms ~]# ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com ping: bad timing interval [root@cloudforms ~]# LC_CTYPE= ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com -bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory PING ipa.hailstorm3.coe.muc.redhat.com (192.168.104.11) 56(84) bytes of data. 64 bytes from ipa.hailstorm3.coe.muc.redhat.com (192.168.104.11): icmp_seq=1 ttl=64 time=0.615 ms --- ipa.hailstorm3.coe.muc.redhat.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.615/0.615/0.615/0.000 ms [root@cloudforms ~]# [root@cloudforms ~]# locale locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory LANG=en_US.UTF-8 LC_CTYPE=UTF-8 LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= [root@cloudforms ~]# I can confirm that the appliance console configures IPA correctly when I invoke it overriding the locale setting: [root@cloudforms ~]# LC_CTYPE= ap
It also now seems to work to configure it via the appliance_console_cli (when overriding the locale): [root@cloudforms ~]# LC_CTYPE= appliance_console_cli --ipaserver ipa.hailstorm3.coe.muc.redhat.com --ipapassword XXXXXXX -bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory Configuring IPA (may take a minute) ... Configuring the IPA Client ... Configuring pam ... Configuring sssd ... Configuring IPA HTTP Service and Keytab ... Configuring httpd ... Configuring SELinux ... Restarting sssd and httpd ... Configuring sssd to start upon reboots ... [root@cloudforms ~]#
One of the intermediate causes is the ping command failing due to the locale settings. The ping command is invoked as part of the IPA configuration to test if the IPA server is available. The locale settings are probably inherited somehow from my OSX 10.11.5 client: Wolframs-MacBook-Pro-6:ansible wolfram$ locale LANG= LC_COLLATE="C" LC_CTYPE="UTF-8" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL= Wolframs-MacBook-Pro-6:ansible wolfram$ ssh cloudforms.hailstorm3.coe.muc.redhat.com -o User=root root.coe.muc.redhat.com's password: Last login: Tue Aug 2 15:02:12 2016 from 10.10.49.33 Welcome to the Appliance Console For a menu, please type: appliance_console [root@cloudforms ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.2 (Maipo) [root@cloudforms ~]# locale locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory LANG=en_US.UTF-8 LC_CTYPE=UTF-8 LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= [root@cloudforms ~]# ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com ping: bad timing interval [root@cloudforms ~]# LC_CTYPE= ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com -bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory PING ipa.hailstorm3.coe.muc.redhat.com (192.168.103.11) 56(84) bytes of data. 64 bytes from ipa.hailstorm3.coe.muc.redhat.com (192.168.103.11): icmp_seq=1 ttl=64 time=0.614 ms --- ipa.hailstorm3.coe.muc.redhat.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.614/0.614/0.614/0.000 ms [root@cloudforms ~]#
Thank you for the help with this Dan
Wolfram, Dan Berger and I tried to reproduce this and can not, which leads me to wonder if there might be something we have not yet identified that is unique to your environment. When you return from PTO let's reassess and if we can not fine a way to force the ping failure outside of your environment let's consider closing this. Thank you for your help. JoeV
This could be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1283277
Some other things to check: - Does /etc/ssh/ssh_config on local OSX machine have this line in it? SendEnv LANG LC_* - Which shell is being used on OSX? Regular bash shell, iterm2, or something else? - What is output of: yum list installed | grep langtable
Wolfram, This BZ seems to be an environmental issue for you. Can you please close it or provide more information including what Daniel Berger asked for in Comment 24? Thank you! JoeV
I'm sorry, this got lost while I was on PTO. Q: Does /etc/ssh/ssh_config on local OSX machine have this line in it? SendEnv LANG LC_* A: yes: Wolframs-MBP-6:~ wolfram$ cat /etc/ssh/ssh_config | grep SendEnv SendEnv LANG LC_* # SendEnv # (no default) Wolframs-MBP-6:~ wolfram$ Q: Which shell is being used on OSX? Regular bash shell, iterm2, or something else? A: I'm using bash in iterm2 Wolframs-MBP-6:~ wolfram$ sh -version GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15) Copyright (C) 2007 Free Software Foundation, Inc. Wolframs-MBP-6:~ wolfram$ Q: What is output of: yum list installed | grep langtable A: (assuming this to be run on the CFME server): Wolframs-MBP-6:~ wolfram$ ssh root.coe.muc.redhat.com root.coe.muc.redhat.com's password: Last login: Mon Sep 12 21:43:20 2016 from 10.36.6.28 Welcome to the Appliance Console For a menu, please type: appliance_console -bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory [root@cloudforms ~]# yum list installed | grep langtable Failed to set locale, defaulting to C [root@cloudforms ~]#
Wolfram, I think I have a solution for this. Would you please try a quick test for me? # Login to your cloudforms test machine: mac> ssh cloudforms.hailstorm3.coe.muc.redhat.com -o User=root # try the ping, expecting it to fail cmfe> ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com ping: bad timing interval # Apply the following updates to /etc/default/evm cfme> cat <<EOT >> /etc/default/evm # Force locale export LANGUAGE=en_US.UTF-8 export LANG=en_US.UTF-8 export LC_CTYPE=en_US.UTF-8 EOT # log off the CFME machine and log back in: cmfe> exit mac> ssh cloudforms.hailstorm3.coe.muc.redhat.com -o User=root # try the ping. It should now succeed cmfe> ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com
Hi Joe, this seems to work exactly as you envisioned: Wolframs-MBP-6:ansible wolfram$ ssh cloudforms.hailstorm3.coe.muc.redhat.com -o User=root root.coe.muc.redhat.com's password: Last login: Wed Sep 21 10:09:10 2016 from 10.36.4.41 Welcome to the Appliance Console For a menu, please type: appliance_console -bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory [root@cloudforms ~]# ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com ping: bad timing interval [root@cloudforms ~]# cat <<EOT >> /etc/default/evm > # Force locale > export LANGUAGE=en_US.UTF-8 > export LANG=en_US.UTF-8 > export LC_CTYPE=en_US.UTF-8 > EOT [root@cloudforms ~]# exit logout Connection to cloudforms.hailstorm3.coe.muc.redhat.com closed. Wolframs-MBP-6:ansible wolfram$ ssh cloudforms.hailstorm3.coe.muc.redhat.com -o User=root root.coe.muc.redhat.com's password: Last login: Thu Sep 22 20:59:38 2016 from 10.36.6.233 Welcome to the Appliance Console For a menu, please type: appliance_console [root@cloudforms ~]# ping -c 1 -W 1 -i 1 ipa.hailstorm3.coe.muc.redhat.com PING ipa.hailstorm3.coe.muc.redhat.com (192.168.101.11) 56(84) bytes of data. 64 bytes from ipa.hailstorm3.coe.muc.redhat.com (192.168.101.11): icmp_seq=1 ttl=64 time=0.343 ms --- ipa.hailstorm3.coe.muc.redhat.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.343/0.343/0.343/0.000 ms [root@cloudforms ~]#
https://github.com/ManageIQ/manageiq-appliance/pull/90
New commit detected on ManageIQ/manageiq-appliance/master: https://github.com/ManageIQ/manageiq-appliance/commit/976e9dfdac0db47d7b41f4497ad7db41a87eaa9c commit 976e9dfdac0db47d7b41f4497ad7db41a87eaa9c Author: Joe VLcek <jvlcek> AuthorDate: Fri Sep 23 14:25:29 2016 -0400 Commit: Joe VLcek <jvlcek> CommitDate: Fri Sep 23 14:25:29 2016 -0400 Force locale to avoid failures due to erroneous locale configurations ssh will set the locale to match the source host, which if not correctly configured can result in invalid locale setting. This change will ensure the locales are valid. https://bugzilla.redhat.com/show_bug.cgi?id=1352822 LINK/etc/default/evm | 6 ++++++ 1 file changed, 6 insertions(+)
Workstation: locale LANG=en_GB.UTF-8 LC_CTYPE=en_GB.UTF-8 LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_PAPER="en_GB.UTF-8" LC_NAME="en_GB.UTF-8" LC_ADDRESS="en_GB.UTF-8" LC_TELEPHONE="en_GB.UTF-8" LC_MEASUREMENT="en_GB.UTF-8" LC_IDENTIFICATION="en_GB.UTF-8" LC_ALL= Appliance: locale LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= Configured External Auth - FreeIPA on 5.8.0.7 and validated LDAP users can login as expected.