Bug 1201974

Nothing has changes in this are of SSSD in 7.1, but I suspect we're receiving this object through dereference after the permission system in IPA has been reworked in 7.1

About DN validating - ldb's documentation on ldb_dn_explode() which is called by ldb_dn_validate explicitly states:

 282   explode a DN string into a ldb_dn structure
 283   based on RFC4514 except that we don't support multiple valued RDNs
 284
 285   TODO: according to MS-ADTS:3.1.1.5.2 Naming Constraints
 286   DN must be compliant with RFC2253
 287 */
 288 static bool ldb_dn_explode(struct ldb_dn *dn)

So the multi-valued DNs IPA started are not supported by SSSD's cache.

Ignoring malformed DNs would OK as long as the ipa_hbac_treat_deny_as option is set to DENY_ALL (which is the default). Otherwise, we can't ignore any malformed DN because we can't be sure if it should deny access. This would be OK for 99.99% of deployments as DENY rules are not supported on the server side since ipa-2.1.

So I propose we:
1) Add a new error code that signals that the DN is malformed. Either it failed validation or it doesn't contain the expected fields.
2) If the ipa_hbac_treat_deny_as option is set to DENY_ALL, then skip malformed DNs, but log them
3) After dereference, only store objects of the type we're interested in.
4) In master (1.13), deprecate the ipa_hbac_treat_deny_as option and only allow DENY_ALL.

btw thank you Paul for precisely pin-pointing the error in the code!

(In reply to Paul Wayper from comment #0)
> Description of problem:
> 
> During the HBAC rule processing, if an error is encountered while parsing
> any of the results found by the search for HBAC rules, the entire rule
> parsing is aborted and the default rule (usually Deny) is used.
> 
> The relevant piece of code is in hbac_eval_user_element():
> 
> $ grep -n -B 6 -A 7 'Parse error' src/providers/ipa/ipa_hbac_common.c 
> 565-    for (i = 0; i < el->num_values; i++) {
> 566-        member_dn = (const char *)el->values[i].data;
> 567-
> 568-        ret = get_ipa_groupname(users->groups, sysdb, member_dn,
> 569-                                &users->groups[num_groups]);
> 570-        if (ret != EOK && ret != ENOENT) {
> 571:            DEBUG(SSSDBG_MINOR_FAILURE, "Parse error on [%s]\n",
> member_dn);
> 572-            goto done;
> 573-        } else if (ret == EOK) {
> 574-            DEBUG(SSSDBG_TRACE_LIBS, "Added group [%s] for user [%s]\n",
> 575-                      users->groups[num_groups], users->name);
> 576-            num_groups++;
> 577-            continue;
> 578-        }
> 
> If get_ipa_groupname returns an ENOMEM (unlikely) or EINVAL (which it does
> if, for example, ldb_dn_validate() fails), then we exit the loop and return
> whatever get_ipa_groupname returned.  In logs this looks like:
> 
> (Fri Mar 13 09:50:45 2015) [sssd[be[example.com]]] [hbac_eval_user_element]
> (0x0080): Parse error on [cn=Modify PassSync Managers
> Configuration+nsuniqueid=c726ac13-c92911e4-8010ef6f-d3c93081,cn=permissions,
> cn=pbac,dc=example,dc=com]
This is a replication conflict entry.

I would suggest to read 389-ds documentation[1] how to fix conflicts.
This might be bug in FreeIPA 

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html

(In reply to Lukas Slebodnik from comment #4)
> > (Fri Mar 13 09:50:45 2015) [sssd[be[example.com]]] [hbac_eval_user_element]
> > (0x0080): Parse error on [cn=Modify PassSync Managers
> > Configuration+nsuniqueid=c726ac13-c92911e4-8010ef6f-d3c93081,cn=permissions,
> > cn=pbac,dc=example,dc=com]
> This is a replication conflict entry.
> 
> I would suggest to read 389-ds documentation[1] how to fix conflicts.
> This might be bug in FreeIPA 
> 
> [1]
> https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/
> html/Administration_Guide/Managing_Replication-
> Solving_Common_Replication_Conflicts.html

OK, let's use this bug to track the replication problem and I'll split another one that would track making SSSD more robust.

Some comments on IRC/mailing list.

A possibility to see this replication conflict is that during the upgrade, the entry 'cn=Modify PassSync Managers Configuration,cn=permissions,cn=pbac,dc=example,dc=com' was added on several DS instance almost at the same time.

It would be useful to get the DS config/logs.
Also could you run the following commands:
ldapsearch -LLL -D "cn=directory manager" -w xxx -b "cn=Modify PassSync Managers Configuration, cn=permissions,cn=pbac,dc=example,dc=com" -s base nscpentrywsi

and

ldapsearch -LLL -D "cn=directory manager" -w xxx -b "cn=Modify PassSync Managers Configuration+nsuniqueid=c726ac13-c92911e4-8010ef6f-d3c93081, cn=permissions,cn=pbac,dc=example,dc=com" -s base nscpentrywsi

thanks

Dan was able to reproduce this BZ.
I'm changing needinfo flag to him.

Hello,

I was helping a consultant who was running through the upgrade from 3.3. to 4.1 and we ran into this issue. Since then we have fixed the problem and no longer have the broken environment.

I'll see if he has an old snapshot or if I'm able to reproduce the problem.

But to fix the issue, we resolved the conflicts, re-initialized and SSSD was then able to parse the entry correctly again.

Created attachment 1004259 [details]
sssd logs

I may be seeing a similar issue with hbac ssh test with IPA upgrades.  

[root@intel-chiefriver-02 ~]# kdestroy -A
[root@intel-chiefriver-02 ~]# kinit hbacreplica_u
Password for hbacreplica_u: 
[root@intel-chiefriver-02 ~]# ipa hbacrule-find
--------------------
4 HBAC rules matched
--------------------
  Rule name: allow_all
  User category: all
  Host category: all
  Service category: all
  Description: Allow all users to access any host from any host
  Enabled: FALSE

  Rule name: hbacclient
  Service category: all
  Enabled: FALSE
  User Groups: hbacclient_g
  Hosts: qe-blade-11.testrelm.test

  Rule name: hbacmaster
  Service category: all
  Enabled: FALSE
  User Groups: hbacmaster_g
  Hosts: qe-blade-04.testrelm.test

  Rule name: hbacreplica
  Service category: all
  Enabled: TRUE
  User Groups: hbacreplica_g
  Hosts: intel-chiefriver-02.testrelm.test
----------------------------
Number of entries returned 4
----------------------------
[root@intel-chiefriver-02 ~]# ipa group-show hbacreplica_g
  Group name: hbacreplica_g
  Description: 0
  GID: 1802400014
  Member users: hbacreplica_u
  Member of HBAC rule: hbacreplica

[root@intel-chiefriver-02 ~]# ssh hbacreplica_u@$(hostname)
Connection closed by UNKNOWN
[root@intel-chiefriver-02 ~]# tail -3 /var/log/secure
Mar 19 20:57:50 intel-chiefriver-02 sshd[24782]: Authorized to hbacreplica_u, krb5 principal hbacreplica_u (ssh_gssapi_krb5_cmdok)
Mar 19 20:57:50 intel-chiefriver-02 sshd[24782]: pam_sss(sshd:account): Access denied for user hbacreplica_u: 4 (System error)
Mar 19 20:57:50 intel-chiefriver-02 sshd[24782]: fatal: Access denied for user hbacreplica_u by PAM account configuration [preauth]
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_parse_deref] (0x1000): Dereferenced DN: ipauniqueid=2158a844-ce76-11e4-959b-00215e20
2e2e,cn=hbac,dc=testrelm,dc=test
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_parse_deref] (0x0080): Dereferenced entry [ipauniqueid=2158a844-ce76-11e4-959b-00215
e202e2e,cn=hbac,dc=testrelm,dc=test] has no attributes
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_x_deref_parse_entry] (0x0040): sdap_parse_deref failed [22]: Invalid argument
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_get_generic_ext_done] (0x0020): reply parsing callback failed.
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_x_deref_search_done] (0x0100): sdap_get_generic_ext_recv failed [22]: Invalid argume
nt
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_deref_search_done] (0x0040): dereference processing failed [22]: Invalid argument
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [ipa_hostgroup_info_done] (0x0040): Error [22][Invalid argument]
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [sdap_id_op_done] (0x4000): releasing operation connection
(Thu Mar 19 20:38:53 2015) [sssd[be[testrelm.test]]] [be_pam_handler_callback] (0x0100): Backend returned: (3, 4, <NULL>) [Internal Error (Syst
em error)]

So, is this the same or different issue? I'm seeing this pretty consistently in my tests after adding an hbac to test the upgrade testing.

No, this sounds like a different issue, probably something with ACIs. What is the SSSD version in the test?

It appears to be happening after IPA master is upgraded and before replica is.  So Master at RHEL7.1 running sssd:

# rpm -q ipa-server sssd
ipa-server-4.1.0-18.el7_1.2.x86_64
sssd-1.12.2-58.el7_1.4.x86_64

And on Replica where I'm seeing the failure:

# rpm -q ipa-server sssd
ipa-server-3.3.3-28.el7.x86_64
sssd-1.11.2-65.el7.x86_64

That version of sssd doesn't look right though.  Shouldn't RHEL7.0 have -68?

(In reply to Scott Poore from comment #11)
> It appears to be happening after IPA master is upgraded and before replica
> is.  So Master at RHEL7.1 running sssd:
> 
> # rpm -q ipa-server sssd
> ipa-server-4.1.0-18.el7_1.2.x86_64
> sssd-1.12.2-58.el7_1.4.x86_64
> 
> And on Replica where I'm seeing the failure:
> 
> # rpm -q ipa-server sssd
> ipa-server-3.3.3-28.el7.x86_64
> sssd-1.11.2-65.el7.x86_64
> 
> That version of sssd doesn't look right though.  Shouldn't RHEL7.0 have -68?

Does it happen also on the client or only on the client?

Ludwig, do you remember, which version of 389-DS had the fix where at least the dereference control was returned even if the client couldn't read the dereferenced entry?

The problem is occurring on the replica and client after master is upgrade.  Once replica is upgraded, problem doesn't occur.

fyi, this is what I have:

master:

389-ds-base-1.3.3.1-15.el7_1.x86_64

replica:

389-ds-base-1.3.1.6-25.el7.x86_64

(In reply to Scott Poore from comment #13)
> The problem is occurring on the replica and client after master is upgrade. 
> Once replica is upgraded, problem doesn't occur.
> 

Ah, then I think it's kind of expected, because ACIs are updated on the master first, replicated and until replica is updated to a version of 389-DS that handles the dereference "better" (or at least in a way that SSSD expects), then SSSD error out.

> fyi, this is what I have:
> 
> master:
> 
> 389-ds-base-1.3.3.1-15.el7_1.x86_64
> 
> replica:
> 
> 389-ds-base-1.3.1.6-25.el7.x86_64

Thank you, it would still be nice if Ludwig could confirm or deny my theory.

So no hbac restrictions when in a RHEL 7.0 and 7.1 mixed environment?  Do you know if we have a bug logged for that already?

I would really like to hear Ludwig's opinion on that, but I think we require a certain DS version with ACI dereference fixes.

(In reply to Jakub Hrozek from comment #12)
> (In reply to Scott Poore from comment #11)
> > It appears to be happening after IPA master is upgraded and before replica
> > is.  So Master at RHEL7.1 running sssd:
> > 
> > # rpm -q ipa-server sssd
> > ipa-server-4.1.0-18.el7_1.2.x86_64
> > sssd-1.12.2-58.el7_1.4.x86_64
> > 
> > And on Replica where I'm seeing the failure:
> > 
> > # rpm -q ipa-server sssd
> > ipa-server-3.3.3-28.el7.x86_64
> > sssd-1.11.2-65.el7.x86_64
> > 
> > That version of sssd doesn't look right though.  Shouldn't RHEL7.0 have -68?
> 
> Does it happen also on the client or only on the client?
> 
> Ludwig, do you remember, which version of 389-DS had the fix where at least
> the dereference control was returned even if the client couldn't read the
> dereferenced entry?

Hello Jakub,

The ticket https://fedorahosted.org/389/ticket/47885 is fixed since 389-ds 1.3.3.2.

I agree with your analyse (https://bugzilla.redhat.com/show_bug.cgi?id=1201974#c14)  a earlier version of the deref plugin will allow access to a dereferenced entry even if the access is not granted to the client.

Additional info, the deref bug was fixed with two separated bugs/tickets.

https://fedorahosted.org/389/ticket/47821: that is fixed in 1.2.11.30
https://fedorahosted.org/389/ticket/47885: that fixed in 1.2.11 branch but not yet released

My understanding is that sssd needs those two tickets, so in that case the deref fix is not available in 1.2.11.
If sssd only needs that the ACI is correctly evaluated on the deref entry, then the version 1.2.11.30 is enough.

Thierry, Ludwig,

Ok, looking at those 389 tickets, it looks like the issues are fixed in 1.3.2 version (not 1.3.1)?

So in my case I have the following:

RHEL 7.1 IPA master - 389-ds-base-1.3.3.1-15.el7_1.x86_64
RHEL 7.0 IPA replica - 389-ds-base-1.3.1.6-25.el7.x86_64

So, I should expect to see failures until upgrade to RHEL7.1 and I get the 1.3.3 version that handles dereferences better?

Thanks,
Scott

Scott,

Both tickets 47821 and 47821 are fixed in 1.3.3.1 (RHEL 7.1). Is it a problem to upgrade the replica to this version ?

thanks
thierry

No, it's not a problem to upgrade.  I started seeing this failure when I added an hbac test to the upgrade test suite.  I just needed to confirm this is an expected failure when replica/clients at RHEL7.0 version (1.3.1) so I can adjust my test or at least waive the failure in the report as expected.

Thanks,
Scott

FYI, I think we have resolved my issue down to bug #1140888.  Sorry for hijacking this bug.  I think this one can be resumed from comment #8 from Dan.

Thanks for the help.

Ok then. Dan, based on your last Comment 8, are we able to reproduce and fix the issue or would we need to close the bug with insufficient data?

Martin,

I do not have a good reproducer at the moment and errors have been fixed in the environment I had access to. I will see if I can reproduce it in the next couple of days.

This is what Ludwig advised to me, but I haven't tested the scenario myself:
- prepare two replicas
- bring one down. On the live server, add an entry. Bring the live one down
- bring the replica up. Add the same entry as on the other server.
- Add the first server up as well.

You should end up with replication conflicts.

The reproduction scenario works for fixing SSSD but it won't tell us why the replication conflict happened during upgrade from 3.3 to 4.1 and if there is anything to fix on the IPA side.

I'm inclined to close it as insufficient data. Will wait for Dan - if he manages to reproduce it.

I have seen some users complaining on IRC with replication conflicts.
They have two replicas. One was on rhel 7.0 (3.3) and second one was already upgraded to rhel7.1(4.1).

(In reply to Lukas Slebodnik from comment #27)
> I have seen some users complaining on IRC with replication conflicts.
> They have two replicas. One was on rhel 7.0 (3.3) and second one was already
> upgraded to rhel7.1(4.1).

I suspect that as well, because all the conflicting entries were permissions which were IIRC reworked in 4.1

Closing the bug according to comment 31 - no clear bug or reproduction - reported replication conflicts are intermittent and related to different parts of IPA which suggests that this bug is too general.