Bug 1156577

Summary: [AAA] Adding an LDAP domain against ldap installed on rhel 6.6 fails
Product: Red Hat Enterprise Virtualization Manager Reporter: James W. Mills <jamills>
Component: ovirt-engineAssignee: Yair Zaslavsky <yzaslavs>
Status: CLOSED ERRATA QA Contact: Ondra Machacek <omachace>
Severity: urgent Docs Contact:
Priority: medium    
Version: 3.4.0CC: adahms, alonbl, bazulay, bkorren, bmcclain, ecohen, iheim, lpeer, lsurette, mkalinin, mtessun, mwest, ngupta, nyechiel, oourfali, prmarino1, pstehlik, rbalakri, Rhev-m-bugs, tdosek, vvasilev, yeylon, yzaslavs
Target Milestone: ---Keywords: ZStream
Target Release: 3.5.0   
Hardware: All   
OS: Linux   
Whiteboard: infra
Fixed In Version: org.ovirt.engine-root-3.5.0-18 Doc Type: Bug Fix
Doc Text:
Previously, upgrading LDAP to rhel-6.6 would cause the Manager to fail to communicate with IPA due to a NegativeArraySizeException exception under certain circumstances. This would prevent LDAP authentication from functioning correctly. As a work around, you can explicitly set "minssf=1" on the IPA side to enable Java to communicate correctly, or run the following command on the machine where the Manager is installed to protect only the authentication sequence: # engine-config -s SASL_QOP auth
Story Points: ---
Clone Of:
: 1157888 1158042 (view as bug list) Environment:
Last Closed: 2015-02-11 18:09:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1157888, 1158042    
Attachments:
Description Flags
tcpdump from 6.4 IPA server, specific to port 389
none
tcpdump from 6.6 IPA server port 389 when adding a domain from RHEV none

Description James W. Mills 2014-10-24 17:07:30 UTC
Created attachment 950447 [details]
tcpdump from 6.4 IPA server, specific to port 389

Description of problem:

After upgrading from IPA 6.4 to IPA 6.6, unable to add domains via engine-manage-domains


Version-Release number of selected component (if applicable):

RHEV 3.x

IPA:
#  rpm -qa ipa-{server,client} 389-ds-*
ipa-server-3.0.0-42.el6.x86_64
389-ds-base-libs-1.2.11.15-47.el6.x86_64
389-ds-base-1.2.11.15-47.el6.x86_64
ipa-client-3.0.0-42.el6.x86_64

How reproducible:

100%


Steps to Reproduce:

* Install IPA 6.6
* Use engine-manage-domain to add an IPA domain


Actual results:

# rhevm-manage-domains add --domain=example.com --provider=IPA --user=admin
Enter password:
General error has occurednull
java.lang.NegativeArraySizeException
	at sun.security.jgss.krb5.CipherHelper.aes256Encrypt(CipherHelper.java:1367)
	at sun.security.jgss.krb5.CipherHelper.encryptData(CipherHelper.java:722)
	at sun.security.jgss.krb5.WrapToken_v2.<init>(WrapToken_v2.java:200)
...
Failure while testing domain %1$s. Details: %2$s: One of the parameters for this error is null and no default message to show



Expected results:

Domain is added.


Additional info:


This has been brought up before, and the response was to change minssf from 0 to 1 to avoid a JVM error.  However, minssf for IPA in RHEL 6.4 was 0 as well, and worked fine, so there is something else happening.

In addition, setting minssf to anything greater than 0 has a negative effect on OpenStack installations configured to use the same IPA server.  Bug forthcoming for that as well.

In essence, customers using RHEV and OpenStack, and using a central IPA server to manage users in these environments could find themselves in a situation where all three products will no longer work together.  This is the first bug of three to raise awareness to this.

I have attached:
* capture file from 6.4 IPA server for LDAP "conversation" when adding the domain via engine-manage-domains

Comment 1 James W. Mills 2014-10-24 17:08:54 UTC
Created attachment 950448 [details]
tcpdump from 6.6 IPA server port 389 when adding a domain from RHEV

Comment 2 Alon Bar-Lev 2014-10-24 17:12:55 UTC
please try to use the ovirt-engine-extension-aaa-ldap in this configuration and not legacy. moving bug.

Comment 3 James W. Mills 2014-10-24 17:47:42 UTC
Additional bugs around using RHEL 6.6 IPA for RHEV + RHOS LDAP authentication in the same env:

IPA: https://bugzilla.redhat.com/show_bug.cgi?id=1156595
RHOS: https://bugzilla.redhat.com/show_bug.cgi?id=1156585

Comment 4 Alon Bar-Lev 2014-10-24 19:41:18 UTC
We should try:

GSSAPIDirContextAuthenticationStrategy.java
---
    @Override
    public void setupEnvironment(Hashtable env, String userDn, String password) throws NamingException {
        env.put(Context.SECURITY_AUTHENTICATION, GSS_API_AUTHENTICATION);
        String qopValue = configuration.getProperty("config.SASL_QOP");
        env.put("javax.security.sasl.qop", qopValue);
+       env.put("javax.security.sasl.strength", "high");
    }
---

Comment 5 Alon Bar-Lev 2014-10-24 20:46:46 UTC
(In reply to James W. Mills from comment #3)
> Additional bugs around using RHEL 6.6 IPA for RHEV + RHOS LDAP
> authentication in the same env:
> 
> IPA: https://bugzilla.redhat.com/show_bug.cgi?id=1156595
> RHOS: https://bugzilla.redhat.com/show_bug.cgi?id=1156585

there is no need to open so many bugs for single issue, it is also possible to specify bug references as bug#1156595 or bug 1156595 for example.

Comment 6 Alon Bar-Lev 2014-10-24 20:48:28 UTC
The actual issue is on the sasl server side[1]

"""
Having done a bit more debugging, it looks like an oddity within Cyrus-SASL's
GSSAPI plugin. If no minimum security strength factor is specified for the
acceptor, it will allow integrity and privacy protection to be negotiated,
but it will just send 0 for the maximum recv size. If I add "minssf=1" to the
sasl-secprops setting on the OpenLDAP server, the server will send a max recv
size of 65536, the Java code can successfully communicate with the server,
and all is well.
"""

Java behaves correctly, but let's see if we can workaround it at client side, or we move this to other products to handle.

[1] http://mail.openjdk.java.net/pipermail/security-dev/2013-February/006665.html

Comment 7 Alon Bar-Lev 2014-10-25 16:52:30 UTC
Thanks for Ondra Machacek for quick setup of environment!

Workaround is to use sasl qop auth instead of auth-conf:

# engine-config -s SASL_QOP auth

Unfortunately, the engine-manage-domains utility does not respect this setting, hence the need to push a fix.

Comment 8 Itamar Heim 2014-10-25 21:27:00 UTC
iiuc, this means RHEL 6.6 will break RHEV 3.4.z (and earlier versions) for manage domains. don't this needs a 3.4.z as well?

Comment 9 Alon Bar-Lev 2014-10-25 21:31:12 UTC
(In reply to Itamar Heim from comment #8)
> iiuc, this means RHEL 6.6 will break RHEV 3.4.z (and earlier versions) for
> manage domains. don't this needs a 3.4.z as well?

yes, this is exactly the plan.

Comment 10 Alon Bar-Lev 2014-10-25 21:38:49 UTC
 Itamar Heim 2014-10-25 17:33:53 EDT
Target Release: 3.5.1 → 3.4.3-1
Flags: rhevm-3.4.z? rhevm-3.5.0?

this only shows how faulty the system is...

this bug should have moved to upstream component and sched to 3.5.1.

the downstream 3.5.0 (which is invalid anyway...) should have inherited this by rebase.

the 3.4.z should have cloned if upstream 3.4.z is eof, otherwise the 3.4.z downstream would have inherited this by rebase.

we do not need pm nor qa acks to fix this in upstream.

Comment 15 Ondra Machacek 2014-10-30 16:42:43 UTC
Ok in vt8.

[root@om-rh35 ~]# engine-config -s SASL_QOP=auth
[root@om-rh35 ~]# service ovirt-engine restart
Stopping oVirt Engine:                                     [  OK  ]
Starting oVirt Engine:                                     [  OK  ]
[root@om-rh35 ~]# rhevm-manage-domains add --domain=brq-ipa-rh66.rhev.lab.eng.brq.redhat.com --user=vdcadmin --provider=ipa
Enter password:
The domain brq-ipa-rh66.rhev.lab.eng.brq.redhat.com has been added to the engine as an authentication source but no users from that domain have been granted permissions within the oVirt Manager.
Users from this domain can be granted permissions by editing the domain using action edit and specifying --add-permissions or from the Web administration interface logging in as admin@internal user.
oVirt Engine restart is required in order for the changes to take place (service ovirt-engine restart).
Manage Domains completed successfully

Then I am able to work without any problem with users from IPA.

Comment 16 Paul Robert Marino 2014-11-11 14:23:10 UTC
Hate to say this but I think I ran into the same issue in RHEV.
this started right around the time when the openssl and NNS patches for POODLE came out.
my LDAP servers in RHDS mode word then suddenly it couldn't query them even though it logged into the LDAP servers successfully.

workaround method 2 worked for me
I didn't try method 1 yet because its a far more intrusive for change for my production environment.

Comment 17 Alon Bar-Lev 2014-11-19 12:34:27 UTC
effects all cyrus-sasl consumers.

Comment 18 Vladimir Vasilev 2014-11-19 13:22:00 UTC
Workaround method 2 don't work for me and I can't change the LDAP server.

[root@cheburator ~]# engine-config -s SASL_QOP=auth
[root@cheburator ~]# service ovirt-engine stop
Stopping oVirt Engine:                                     [  OK  ]
[root@cheburator ~]# service ovirt-engine start
Starting oVirt Engine:                                     [  OK  ]
[root@cheburator ~]# engine-config -g SASL_QOP
SASL_QOP: auth version: general
[root@cheburator ~]# rhevm-manage-domains add --domain=redhat.com --user=vvasilev --provider=RHDS
Enter password:
General error has occurednull
java.lang.NegativeArraySizeException
        at sun.security.jgss.krb5.CipherHelper.aes256Encrypt(CipherHelper.java:1367)
        at sun.security.jgss.krb5.CipherHelper.encryptData(CipherHelper.java:722)
        at sun.security.jgss.krb5.WrapToken_v2.<init>(WrapToken_v2.java:200)
        at sun.security.jgss.krb5.Krb5Context.wrap(Krb5Context.java:861)
        at sun.security.jgss.GSSContextImpl.wrap(GSSContextImpl.java:385)
        at com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(GssKrb5Base.java:104)
        at com.sun.jndi.ldap.sasl.SaslOutputStream.write(SaslOutputStream.java:89)
        at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:430)
        at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:555)
        at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
        at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)
        at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)
        at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:386)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:356)
        at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:339)
        at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:267)
        at org.ovirt.engine.core.ldap.RootDSEData.<init>(RootDSEData.java:52)
        at org.ovirt.engine.core.utils.kerberos.JndiAction.getDomainDN(JndiAction.java:254)
        at org.ovirt.engine.core.utils.kerberos.JndiAction.run(JndiAction.java:87)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:356)
        at org.ovirt.engine.core.utils.kerberos.KerberosConfigCheck.promptSuccessfulAuthentication(KerberosConfigCheck.java:129)
        at org.ovirt.engine.core.utils.kerberos.KerberosConfigCheck.validateKerberosInstallation(KerberosConfigCheck.java:105)
        at org.ovirt.engine.core.utils.kerberos.KerberosConfigCheck.checkInstallation(KerberosConfigCheck.java:90)
        at org.ovirt.engine.core.domains.ManageDomains.checkKerberosConfiguration(ManageDomains.java:751)
        at org.ovirt.engine.core.domains.ManageDomains.testConfiguration(ManageDomains.java:919)
        at org.ovirt.engine.core.domains.ManageDomains.addDomain(ManageDomains.java:507)
        at org.ovirt.engine.core.domains.ManageDomains.runCommand(ManageDomains.java:251)
        at org.ovirt.engine.core.domains.ManageDomainsExecutor.main(ManageDomainsExecutor.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.modules.Module.run(Module.java:312)
        at org.jboss.modules.Main.main(Main.java:460)
Failure while testing domain %1$s. Details: %2$s: One of the parameters for this error is null and no default message to show
[root@cheburator ~]# rhevm-manage-domains list
Manage Domains completed successfully

Comment 19 Alon Bar-Lev 2014-11-19 13:26:05 UTC
(In reply to Vladimir Vasilev from comment #18)
> Workaround method 2 don't work for me and I can't change the LDAP server.

what version of rhevm do you use? not sure this z-stream reached your channel.

Comment 21 errata-xmlrpc 2015-02-11 18:09:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0158.html