Tuan, I have written a blog post [1] describing how to diagnose and what to report regarding authentication issues. There is a section titled Reporting Authentication Issues [2] Please have the customer provide the information requested in the section titled: "Capturing Authentication Data From ManageIQ DB" [3] for both the succeeding and failing appliance. That being the output from the following commands: /var/www/miq/vmdb/bin/rails runner 'puts Settings.authentication' /var/www/miq/vmdb/bin/rails runner 'puts MiqGroup.pluck(:description)' /var/www/miq/vmdb/bin/rails runner 'puts User.pluck(:userid)' Also please ask the customer to go through the section titled: "Confirming LDAP Configuration" [4] and provide the output from the example ldapsearch commands for both the failing and succeeding appliances. That being the output from the following commands: ldapsearch -x -H ldap://ldaphost.example.com:389 -b "dc=example,dc=com" -D "cn=Manager,dc=example,dc=com" -w password ldapsearch -x -H ldap://ldaphost.example.com:389 -b "dc=example,dc=com" ldapsearch -x -H ldaps://ldaphost.example.com:636 -b "dc=example,dc=com" This information is critical in allowing me to help diagnose the issue. Thank you. JoeV [1] http://manageiq.org/blog/2018/01/troubleshooting-auth/ [2] http://manageiq.org/blog/2018/01/troubleshooting-auth/#reporting-authentication-issues [3] http://manageiq.org/blog/2018/01/troubleshooting-auth/#capturing-authentication-data-from-manageiq-db [4] http://manageiq.org/blog/2018/01/troubleshooting-auth/#confirming-ldap-configuration
Tuan, Please extend my thanks to the customer for all the help and patience with this issue. After reviewing the provided data I am fairly sure I realize what is happening. In the 4.2 appliance there was a bug that was fixed in the newer versions. Groups should be manually created in every region but the bug was that groups would match across regions. So groups only had to be manually created in one region. The fix plugged this issue by confirming that the user attempting to log in had matching groups manually created in the region they are logging into. The new functionality checks that groups were manually created in every region you want those users to log into. Please test by manually adding the groups for one user to one region that is failing. If this does not resolve the problem, let's try to setup a conference call to diagnose this directly some time Tuesday or later next week.
Taun, Please also ask them to time how long the ldapsearch command takes on both of the system. They should take about the same time on both. e.g.: time ldapsearch -x -H ldaps://ldaphost.example.com:636 -b "dc=example,dc=com" Thank you. Joe
Taun, I noticed the customer has not updated the case with the output from "time" on the ldapsearch commands. Since the last update the customer states that the failures are observed on both their 4.5 and and 4.2 environments a likely issues could be LDAP performance related. Upon reexamining the data they have already provided I see they have "group_memberships_max_depth" set to "10". This deep of a group membership search could produce performance issues. I also noticed they have configured two LDAP servers. It would be best if they could do some experimenting with using only one at a time to eliminate the possibility that one server is having performance issues. Please ask them to provide the output from a command similar to the one below run on both regions and to both of their LDAP servers: time ldapsearch -x -LLL -H ldaps://<my ldap server>:636 -b "<my searchbase>" -D "<my binddn?" -w <my password> -s sub "(userPrincipalName=<my user>@<my user suffix>)" '*' memberof Please pass this comment on "as is" directly to the customers. I am also available for a video conf call if needed. JoeV
Tuan, Please also ask Ben to provide the debugging data as described in comment 17 https://bugzilla.redhat.com/show_bug.cgi?id=1579986#c17 To enable debugging for Authentication please set `:level:` under section `:log:` in the `Configuration/Settings/Advanced` to `debug` the tail the evm.log file while attempting to log in with the failing user ID. You should see line with: MiqLdap#... and Authenticator::Ldap#... Thank you, JoeV
The customer reports things work fine for some users but users with many more groups are seeing various failures. This indicates to me that the issue is very likely routed in the customers environment. I have offered many times to join a conf call to help diagnose and root cause the issue but the customers seems reluctant to do that. II propose this case be downgraded or closed. JoeV
Tuan, This BZ has been in "Waiting on Customer" status for many weeks now. The description of the problem has also changed during the course of time this bug has been open, making it hard to understand what is wrong. I'm going to mark it closed/insufficient_data. If the customer wishes to reopen please ask them if we can get on a video conference to see what is failing and diagnose the issue. Thank you. JoeV
Closed. Canceling NEEDINFO