Bug 1669012

Summary: host_del and host_disable fails, ra.find() search for every certificates instead of the host's certificate by subject
Product: Red Hat Enterprise Linux 7 Reporter: Marc Sauton <msauton>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.6CC: cheimes, frenaud, gparente, jwooten, ksiddiqu, myusuf, ndehadra, pvoborni, rcritten, tscherf
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ipa-4.6.5-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1672176 (view as bug list) Environment:
Last Closed: 2019-08-06 13:09:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1672176    

Description Marc Sauton 2019-01-24 06:39:22 UTC 
Description of problem:

host_del and host_disable can fail as cert_find calls ra.find() which search for every certificates instead of the host's certificate by subject DN or serial number.

it should do like for example "ipa cert-find --subject=test0.example.com" , use an LDAP filter in the form of (subjectName=*CN=*test0.example.com*) instead of "(certStatus=*)".

this bug was opened because there is a situation with 560K certificates, where the LDAP search does not complete (it should not, is a separate issue), and as a result, an ipa host_disable command fails with a time out and HTTP 500, ra.find() is searching for every certificate:

ipa: DEBUG: request POST https://ipaserverx:443/ca/rest/certs/search?size=2147483647

and this is a paged search.
and LDAP filters with certStatus=* will use significant cpu resources when there are 100K's of certs.


we have:
/usr/lib/python2.7/site-packages/ipaserver/plugins/host.py
class host_disable(LDAPQuery):
...
            certs = self.api.Command.cert_find(host=keys)['result']

and in
/usr/lib/python2.7/site-packages/ipaserver/plugins/cert.py


why are the host_del and host_disable 's cert_find / ra.find() not doing the cert find by subject that provide the serial number(s), and not use the REST API to revoke by serial number  /ca/rest/agent/certs/{id}/revoke ?


Version-Release number of selected component (if applicable):
RHEL-7.6
ipa-server-4.6.4-10.el7.x86_64
redhat-release-server-7.6-4.el7.x86_64


How reproducible:
N/A for failing host_disable or host__del ipa command


Steps to Reproduce:
1.
2.
3.

Actual results:

in the customer case with 560K certs, a host_disable fails:
[Wed Jan 23 11:38:39.213868 2019] [:error] [pid 30733] ipa: ERROR: ra.find(): Unable to communicate with CMS (500)

host_disable calls cert_find by host, but then call ra.find() which looks for every single certificates in the database:

[23/Jan/2019:11:37:39.161829132 -0500] conn=1337 op=67028 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL
[23/Jan/2019:11:37:39.162170031 -0500] conn=1337 op=67028 SORT serialno
[23/Jan/2019:11:37:39.162179894 -0500] conn=1337 op=67028 VLV 0:2147483647:A 559174:559174 (0)
[23/Jan/2019:11:37:39.162320010 -0500] conn=1337 op=67028 RESULT err=0 tag=101 nentries=1 etime=0.0000770013
[23/Jan/2019:11:37:39.163614312 -0500] conn=1337 op=67029 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL
...
[23/Jan/2019:11:37:39.688976496 -0500] conn=1337 op=67029 SORT serialno
[23/Jan/2019:11:37:39.689011899 -0500] conn=1337 op=67029 VLV 0:2147483647:0:559174 1:559174 (0)
...
[23/Jan/2019:11:37:40.499161920 -0500] conn=1337 op=67029 RESULT err=4 tag=101 nentries=4000 etime=1.0335840456
[23/Jan/2019:11:37:40.551004169 -0500] conn=1337 op=67030 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL
...
[23/Jan/2019:11:37:40.939437144 -0500] conn=1337 op=67030 SORT serialno
[23/Jan/2019:11:37:40.939462040 -0500] conn=1337 op=67030 VLV 0:2147483647:3999:559174 4000:559174 (0)
...
[23/Jan/2019:11:37:41.792340014 -0500] conn=1337 op=67030 RESULT err=4 tag=101 nentries=4000 etime=1.0241568254
...



Expected results:
HTTP 200


Additional info:

will add private attachments.
Comment 4 joel 2019-01-24 23:09:01 UTC 
cu considers this a production down situation, and is requesting a work around.

Is there a method to get around this issue?
Comment 6 Christian Heimes 2019-01-29 11:59:00 UTC 
Fixed upstream
master:
https://pagure.io/freeipa/c/ae74d348c3da580264c56441c136af3fc6ae58df
https://pagure.io/freeipa/c/6cd37542e6c563ddd8595d488712935a4d6b17bf
Comment 8 Alexander Bokovoy 2019-01-29 14:02:32 UTC 
Upstream ticket:
https://pagure.io/freeipa/issue/7835
Comment 9 Christian Heimes 2019-01-30 07:09:36 UTC 
Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/1e842f0fea290cbf8935eea79ea16e17650b7328
https://pagure.io/freeipa/c/9a0783f5ac8a2c7426b4db3a818fa6583bab16e1
ipa-4-7:
https://pagure.io/freeipa/c/a243bd56a65e47b03ee3ba772413b580aeb35ba9
https://pagure.io/freeipa/c/2ccd4da44b5d66298bf1d4938fa3dc6ce2dbfa84
Comment 19 Mohammad Rizwan 2019-05-02 11:52:24 UTC 
Version:
ipa-server-4.6.5-6.el7.x86_64

Steps:
1. add a host to ipa
   # ipa host-add test.testrelm.test --force

2. Add a service 
   # ipa service-add testservice/test.testrelm.test --force

3. Add service to the host
   # ipa service-add-host testservice/test.testrelm.test --hosts=$(hostname)

4. Issue a certificate to use the service on host
   # ipa-getcert request -k /etc/pki/tls/private/testservice.key -f /etc/pki/tls/certs/testservice.pem -w -N CN=test.testrelm.test -D test.testrelm.test -K testservice/test.ipa.example

5. check issued certificate, status should be valid
   # ipa cert-find --subject=test.testrelm.test

6. delete the host
   # ipa host-del test.testrelm.test

7. check certificate status as revoked.
   # ipa cert-find --subject=test.testrelm.test



Actual result:
[root@master ~]# ipa host-add test.testrelm.test --force
-------------------------------
Added host "test.testrelm.test"
-------------------------------
  Host name: test.testrelm.test
  Principal name: host/test.testrelm.test
  Principal alias: host/test.testrelm.test
  Password: False
  Keytab: False
  Managed by: test.testrelm.test
[root@master ~]# 
[root@master ~]# ipa service-add testservice/test.testrelm.test --force
------------------------------------------------------------
Added service "testservice/test.testrelm.test"
------------------------------------------------------------
  Principal name: testservice/test.testrelm.test
  Principal alias: testservice/test.testrelm.test
  Managed by: test.testrelm.test

[root@master ~]# ipa service-add-host testservice/test.testrelm.test --hosts=$(hostname)
  Principal name: testservice/test.testrelm.test
  Principal alias: testservice/test.testrelm.test
  Managed by: test.testrelm.test, master.testrelm.test
-------------------------
Number of members added 1
-------------------------

[root@master ~]# ipa-getcert request -k /etc/pki/tls/private/testservice.key -f /etc/pki/tls/certs/testservice.pem -w -N CN=test.testrelm.test -D test.testrelm.test -K testservice/test.testrelm.test
New signing request "20190502112859" added.


[root@master ~]# ipa cert-find --subject=test.testrelm.test
---------------------
1 certificate matched
---------------------
  Issuing CA: ipa
  Subject: CN=test.testrelm.test,O=TESTRELM.TEST
  Issuer: CN=Certificate Authority,O=TESTRELM.TEST
  Not Before: Thu May 02 16:59:02 2019 UTC
  Not After: Sun May 02 16:59:02 2021 UTC
  Serial number: 11
  Serial number (hex): 0xB
  Status: VALID
  Revoked: False
----------------------------
Number of entries returned 1
----------------------------

[root@master ~]# ipa host-del test.testrelm.test
---------------------------------
Deleted host "test.testrelm.test"
---------------------------------

[root@master ~]# ipa cert-find --subject=test.testrelm.test
---------------------
1 certificate matched
---------------------
  Issuing CA: ipa
  Subject: CN=test.testrelm.test,O=TESTRELM.TEST
  Issuer: CN=Certificate Authority,O=TESTRELM.TEST
  Not Before: Thu May 02 16:59:02 2019 UTC
  Not After: Sun May 02 16:59:02 2021 UTC
  Serial number: 11
  Serial number (hex): 0xB
  Status: REVOKED
  Revoked: True
----------------------------
Number of entries returned 1
----------------------------
Comment 23 Mohammad Rizwan 2019-05-09 07:31:28 UTC 
Tested with around 46k certificate. Host delete properly revokes the certificate.

[root@master ~]# ipa cert-find --sizelimit=0
[..]
--------------------------------
Number of entries returned 46417
--------------------------------

[root@master ~]# ipa host-del test1525437.testrelm.test
----------------------------------------
Deleted host "test1525437.testrelm.test"
----------------------------------------
[root@master ~]# 
[root@master ~]# 
[root@master ~]# ipa cert-find --subject=test1525437.testrelm.test
ipa: WARNING: Search result has been truncated: Configured size limit exceeded
---------------------
1 certificate matched
---------------------
  Issuing CA: ipa
  Subject: CN=test1525437.testrelm.test,O=TESTRELM.TEST
  Issuer: CN=Certificate Authority,O=TESTRELM.TEST
  Not Before: Thu May 09 12:44:08 2019 UTC
  Not After: Sun May 09 12:44:08 2021 UTC
  Serial number: 46417
  Serial number (hex): 0xB551
  Status: REVOKED
  Revoked: True
----------------------------
Number of entries returned 1
----------------------------

Automation also passed in CI. Hence based on these observations, marking the bug as verified.
Comment 25 errata-xmlrpc 2019-08-06 13:09:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2241