Bug 2164349

When a user is browsing the WebUI and navigates to Authentication > Certificates, the webui performs a command equivalent to "ipa cert-find" without any size or time limit.

The IPA framework handles this operation in /ipaserver/plugins/cert.py, class cert_find, method execute()
https://github.com/freeipa/freeipa/blob/7d1d91fc86c49fcaaec05c772add13af36fc0209/ipaserver/plugins/cert.py#L1817

The execute method launches _ca_search, which in turn call ra.find()
https://github.com/freeipa/freeipa/blob/7d1d91fc86c49fcaaec05c772add13af36fc0209/ipaserver/plugins/cert.py#L1701

This method is defined in ipaserver/plugins/dogtag.py, class ra
https://github.com/freeipa/freeipa/blob/7d1d91fc86c49fcaaec05c772add13af36fc0209/ipaserver/plugins/dogtag.py#L1727

The method is doing an http request to PKI: POST /ca/rest/certs/search?size=2147483647
(the size is the default one, 0x7fffffff, used because no options.sizelimit was provided)
https://github.com/freeipa/freeipa/blob/7d1d91fc86c49fcaaec05c772add13af36fc0209/ipaserver/plugins/dogtag.py#L1828

Note: even if the user defines a sizelimit with ipa cert-find --sizelimit=xx, the limit passed to PKI is the default one 0x7fffffff. The limit is not forwarded from IPA framework to PKI.

The limit is only used when all the certs have been retrieved in order to truncate the returned entries:
https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L1917

        if (len(result) > sizelimit > 0):
            if not truncated:
                self.add_message(messages.SearchResultTruncated(
                        reason=errors.SizeLimitExceeded()))
            result = result[:sizelimit]
            truncated = True


On PKI side:
PKI server seems to properly honor the size parameter. If it receives POST /ca/rest/certs/search?size=0x00000001 it sends only the requested number of certificates, ie one.

In summary:
- IPA framework should be enhanced and make a better use of the size limit provided through ipa cert-find --sizelimit=xxx
- When no sizelimit is provided, the best would be to do a paged size when querying PKI. Not sure if PKI implements this functionality.

It is more complex.. cert-find is used internally in a number of different ways that could require loops to replace the current massive size, which I'm not defending. The PKI REST API doesn't support paging directly but does allow for a start value which could be used for non-RSNv3 installs. It will not work for random serial numbers. Additional performance could be gained by filtering on VALID certificates in some cases, or by subject. Attempts have been made in the past to improve it but were cut short by corner cases.

The search API is very limited such that complex searches are just not possible (AFAIR it is AND logic only, no OR or NOT).

This is somewhat related to https://pagure.io/freeipa/issue/7835 which tries to optimize the search a bit to include only VALID certs.

I tested the --start and --size options using the REST API and it isn't exactly paging, more like a shifting window. start is a point within the list of total certs and size is the number of certs to return. We could add a start for paged searching in the new WebUI.

While it is probably a one-liner to fix size limit I think at the same time the search backend should switch from the old XML-based API to REST. The XML API is already deprecated and will be removed soon. Best to take advantage of the time now to fix it properly.

I see no reason to pull every single cert from the CA only to filter it, agreeing with Flo's analysis.

I think the steps will be several-fold:

1. Revive issue 7835 to do smarter searching when doing revocations
2. Wrap this searching to re-run the search until no certs are found on the off-chance that some service/host has more certs than the default search size
3. Switch to the REST search API, and JSON return values

Step 1 is basically done. A corner case was found so the effort was abandoned but it is likely straightforward to fix
Step 2 should be straightforward and hopefully won't end up with too much code duplication
Step 3 is a big step but worth it in the long run

The API is partially documented at https://github.com/dogtagpki/pki/wiki/CA-Search-Certificates-REST-API . What's missing is the possible values within the POST to refine the search. It is discoverable in the code and/or using pki ca-cert-find to reverse engineer the search options.

My mistake. The REST API is used, but with XML and not JSON.

The enormous sizelimit was added in https://github.com/freeipa/freeipa/commit/d84edc43e55c2f7c30614a4a5268aeb58e33a087 to address https://pagure.io/freeipa/issue/6564

The implication is that the search limit is used by PKI to determine how deeply to search in the VLV and is not a limiter in the number of certificates returned. This would explain why the entire contents are retrieved and the size truncated by IPA.

Upstream PR https://github.com/freeipa/freeipa/pull/6698

master:
https://pagure.io/freeipa/c/2b2f10c2eb7f3b796c68771bc8cbf5dbaa646481
https://pagure.io/freeipa/c/191880bc9f77c3e8a3cecc82e6eea33ab5ad03e4

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/6e04c4959bf1ebdc0a9d1cb24969bec278191438
https://pagure.io/freeipa/c/f54a55a1b4636391b443368e318176ea79d5ffc5

Fixed upstream
ipa-4-10:
https://pagure.io/freeipa/c/e2576670e692117c11987118abd5e9381bb90b1f
https://pagure.io/freeipa/c/50dd79d1a35549034bc281fbdffea4399baed3c7

Fix for cert-find --all

Fixed upstream
master:
https://pagure.io/freeipa/c/1f30cc65276a532e7288217f216b72a2b0628c8f

Fix for cert-find --all

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/59cab232a0dcc210972dcfb43cb77df468e377ad

ipa-4-10:
https://pagure.io/freeipa/c/918b6e011795ba4854d178d18c86ad54f3cf75ab

version:
ipa-server-4.9.12-2.module+el8.9.0+18921+013c0de2.x86_64

Steps:
1. install ipa server and add no. of certificates, I added 150
2. issue ipa cert-find command with --sizelimit option

without --sizelimit option:

access_log:
xx.xx.xx.xx - - [08/Jun/2023:03:51:14 -0400] "POST /ca/rest/certs/search?size=100 HTTP/1.1" 200 114525
xx.xx.xx.xx - admin [08/Jun/2023:03:51:14 -0400] "POST /ipa/session/json HTTP/1.1" 200 1969


[root@master ~]# ipa cert-find 
------------------------
100 certificates matched
------------------------
[..]

  Issuing CA: ipa
  Subject: CN=tuser90,O=TESTRELM.TEST
  Issuer: CN=Certificate Authority,O=TESTRELM.TEST
  Not Before: Thu Jun 08 07:13:13 2023 UTC
  Not After: Sun Jun 08 07:13:13 2025 UTC
  Serial number: 100
  Serial number (hex): 0x64
  Status: VALID
  Revoked: False
------------------------------
Number of entries returned 100
------------------------------

with --sizelimit=15 : 

xx.xx.xx.xx - - [08/Jun/2023:03:53:23 -0400] "POST /ca/rest/certs/search?size=15 HTTP/1.1" 200 17461
xx.xx.xx.xx - admin [08/Jun/2023:03:53:23 -0400] "POST /ipa/session/json HTTP/1.1" 200 589

[root@master ~]# ipa cert-find --sizelimit=15
-----------------------
15 certificates matched
-----------------------
[..]

  Issuing CA: ipa
  Subject: CN=tuser5,O=TESTRELM.TEST
  Issuer: CN=Certificate Authority,O=TESTRELM.TEST
  Not Before: Thu Jun 08 07:11:05 2023 UTC
  Not After: Sun Jun 08 07:11:05 2025 UTC
  Serial number: 15
  Serial number (hex): 0xF
  Status: VALID
  Revoked: False
-----------------------------
Number of entries returned 15
-----------------------------
[root@master ~]# 


with --sizelimit=0

[root@master ~]# ipa cert-find --sizelimit=0

xx.xx.xx.xx - - [08/Jun/2023:03:55:25 -0400] "POST /ca/rest/certs/search?size=2147483647 HTTP/1.1" 200 183026
xx.xx.xx.xx - admin [08/Jun/2023:03:55:25 -0400] "POST /ipa/session/json HTTP/1.1" 200 2955

------------------------
160 certificates matched
------------------------
[..]

  Issuing CA: ipa
  Subject: CN=tuser150,O=TESTRELM.TEST
  Issuer: CN=Certificate Authority,O=TESTRELM.TEST
  Not Before: Thu Jun 08 07:14:45 2023 UTC
  Not After: Sun Jun 08 07:14:45 2025 UTC
  Serial number: 160
  Serial number (hex): 0xA0
  Status: VALID
  Revoked: False
------------------------------
Number of entries returned 160
------------------------------
[root@master ~]# 


[root@master ~]# ipa cert-find --sizelimit=-1
ipa: ERROR: invalid 'sizelimit': must be at least 0
[root@master ~]# 
[root@master ~]# 
[root@master ~]# ipa cert-find --sizelimit=sa
ipa: ERROR: invalid 'sizelimit': must be an integer
[root@master ~]#

--sizelimit properly propagated to CA calls, hence marking the bug as verified.

A replacement fix that doesn't rely on python3-pyOpenSSL (and is faster)

master:
https://pagure.io/freeipa/c/2a605c5d07906e157e79458724be098aab28cc7c
https://pagure.io/freeipa/c/8a250201494fa0864c81ba0bb2d16a485cdd2533
https://pagure.io/freeipa/c/fa3a69f91fcb4e15714f78a6eee4944bb8ca5e1b

Replacement fix for additional branches

ipa-4-10:
https://pagure.io/freeipa/c/276138087158c6b2ea76b43c754084144e543c0b
https://pagure.io/freeipa/c/d83a4b0babdc7beb124d3748b5815ce309739eb7
https://pagure.io/freeipa/c/d9aa75459d650e5282a160a3eef09ed175dc5b51

ipa-4-9:
https://pagure.io/freeipa/c/9fe30f21c987bdccf80ef5f6d645fdc59b393bdb
https://pagure.io/freeipa/c/3b1dbcdba2994bf57908f530913998e9ab888e4c
https://pagure.io/freeipa/c/d00fd3398c32beb2c3e72f4878c87f9d2c0e833d

Steps:
1. install ipa-server
2. generate 4000 certificate
3. $ time ipa cert-find


old version:
ipa-server-4.9.11-6.module+el8.8.0+19022+e8902f4b.x86_64


[root@master ~]# time ipa cert-find
ipa: WARNING: Search result has been truncated: Configured size limit exceeded
------------------------
100 certificates matched
------------------------
[..]
  Issuing CA: ipa
  Subject: CN=user86,O=TESTREALM.TEST
  Issuer: CN=Certificate Authority,O=TESTREALM.TEST
  Not Before: Tue Jul 18 08:22:42 2023 UTC
  Not After: Fri Jul 18 08:22:42 2025 UTC
  Serial number: 100
  Serial number (hex): 0x64
  Status: VALID
  Revoked: False
------------------------------
Number of entries returned 100
------------------------------

real	0m15.891s
user	0m0.464s
sys	0m0.031s
[root@master ~]#


version:
ipa-server-4.9.12-5.module+el8.9.0+19430+5c00c3bc.x86_64


2023-07-20T13:28:09 root.test# time ipa cert-find
2023-07-20T13:28:22 ------------------------
2023-07-20T13:28:22 100 certificates matched
2023-07-20T13:28:22 ------------------------
[..]
2023-07-20T13:28:22   Issuing CA: ipa
2023-07-20T13:28:22   Subject: CN=user86,O=TESTRELM.TEST
2023-07-20T13:28:22   Issuer: CN=Certificate Authority,O=TESTRELM.TEST
2023-07-20T13:28:22   Not Before: Thu Jul 20 11:51:57 2023 UTC
2023-07-20T13:28:22   Not After: Sun Jul 20 11:51:57 2025 UTC
2023-07-20T13:28:22   Serial number: 100
2023-07-20T13:28:22   Serial number (hex): 0x64
2023-07-20T13:28:22   Status: VALID
2023-07-20T13:28:22   Revoked: False
2023-07-20T13:28:22 ------------------------------
2023-07-20T13:28:22 Number of entries returned 100
2023-07-20T13:28:22 ------------------------------
2023-07-20T13:28:22 
2023-07-20T13:28:22 real	0m2.170s
2023-07-20T13:28:22 user	0m0.482s
2023-07-20T13:28:22 sys	0m0.036s

There is significant difference between time taken on old and new version (0m2.170s against 0m15.891s). Hence marking the bug verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (idm:client and idm:DL1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6977