Bug 994604

Description of problem:
Seeing and issue where users are not able to log in. Also for some reason the engine is seemingly forgeting about AD users. Removing the AD domain via engine-manage-domains and re-adding it works for enumerating the users, however the first attempt to login as a user results in the engine no longer enumerating the users nor allowing logins.


Version-Release number of selected component (if applicable):
ovirt-engine master

How reproducible:
100%

Steps to Reproduce:

Start the engine bound to an AD for authentication
log in to the user portal as an AD user which has been granted a Role (I used PowerUserRole)

Result: Login will succeed
Data from engine.log:
2013-08-06 15:54:10,088 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-10) Running command: LoginUserCommand internal: false.
2013-08-06 15:54:10,139 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-10) Correlation ID: 23c4709, Call Stack: null, Custom Event ID: -1, Message: User ovirttest logged in.

log out of the user portal
Result: log out succeeds
Data from engine.log:
2013-08-06 15:54:12,448 INFO  [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-2) Running command: LogoutUserCommand internal: false.
2013-08-06 15:54:12,474 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: 52a89e7d, Call Stack: null, Custom Event ID: -1, Message: User ovirttest logged out.

As the same user log in to the user portal again but this purposely input the wrong password.
Result: log in will fail
Data from engine.log:
2013-08-06 15:54:20,830 ERROR [org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy] (ajp--127.0.0.1-8702-7) Kerberos error: Pre-authentication information was invalid (24)
2013-08-06 15:54:20,832 ERROR [org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy] (ajp--127.0.0.1-8702-7) Authentication Failed. Please verify the username and password.
2013-08-06 15:54:20,843 ERROR [org.ovirt.engine.core.bll.adbroker.DirectorySearcher] (ajp--127.0.0.1-8702-7) Failed ldap search server LDAP://foodc02.foo.test.com:389 using user ovirttest.COM due to Authentication Failed. Please verify the username and password.. We should not try the next server
2013-08-06 15:54:20,850 ERROR [org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy] (ajp--127.0.0.1-8702-7) Kerberos error: Pre-authentication information was invalid (24)
2013-08-06 15:54:20,851 ERROR [org.ovirt.engine.core.bll.adbroker.GSSAPIDirContextAuthenticationStrategy] (ajp--127.0.0.1-8702-7) Authentication Failed. Please verify the username and password.
2013-08-06 15:54:20,852 ERROR [org.ovirt.engine.core.bll.adbroker.DirectorySearcher] (ajp--127.0.0.1-8702-7) Failed ldap search server LDAP://foodc01.foo.test.com:389 using user ovirttest.COM due to Authentication Failed. Please verify the username and password.. We should not try the next server
2013-08-06 15:54:20,853 ERROR [org.ovirt.engine.core.bll.adbroker.LdapAuthenticateUserCommand] (ajp--127.0.0.1-8702-7) Failed authenticating user: ovirttest to domain gso.med.ge.com. Ldap Query Type is getUserByName
2013-08-06 15:54:20,854 ERROR [org.ovirt.engine.core.bll.adbroker.LdapAuthenticateUserCommand] (ajp--127.0.0.1-8702-7) Authentication Failed. Please verify the username and password.
2013-08-06 15:54:20,855 ERROR [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) USER_FAILED_TO_AUTHENTICATE_WRONG_USERNAME_OR_PASSWORD : ovirttest
2013-08-06 15:54:20,856 WARN  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) CanDoAction of action LoginUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE_WRONG_USERNAME_OR_PASSWORD

Try again to log in as the same user this time typing the correct password.
Result: Login fails!
Data from engine.log:
2013-08-06 15:54:25,186 ERROR [org.ovirt.engine.core.bll.adbroker.LdapAuthenticateUserCommand] (ajp--127.0.0.1-8702-7) Failed authenticating user: ovirttest to domain gso.med.ge.com. Ldap Query Type is getUserByName
2013-08-06 15:54:25,187 ERROR [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) USER_FAILED_TO_AUTHENTICATE : ovirttest
2013-08-06 15:54:25,187 WARN  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) CanDoAction of action LoginUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE

Try again with another AD user.
Result: Login fails!
Data from engine.log:
2013-08-06 15:54:38,056 ERROR [org.ovirt.engine.core.bll.adbroker.LdapAuthenticateUserCommand] (ajp--127.0.0.1-8702-5) Failed authenticating user: ovirtadmin to domain gso.med.ge.com. Ldap Query Type is getUserByName
2013-08-06 15:54:38,057 ERROR [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) USER_FAILED_TO_AUTHENTICATE : ovirtadmin
2013-08-06 15:54:38,058 WARN  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-5) CanDoAction of action LoginUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE

Logging into the admin portal as the admin@internal user will yield that engine seems to have forgotten about and can no longer enumerate AD users and groups.
engine stays in this state until it has been restarted.

I also note the two following errors in the engine log file as well:
2013-08-06 15:53:41,098 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (MSC service thread 1-9) Could not parse option AutoRecoveryAllowedTypes value.
2013-08-06 15:53:41,161 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (MSC service thread 1-9) Failed to decrypt value for property AttestationTruststorePass will be used encrypted value: javax.crypto.BadPaddingException: Data must start with zero