Description of problem: When customer searches for a user via IPA WebUI, it takes really long time to display the user's details. The same user can be queries via CLI, within few seconds. The ipa cert-find command also takes time to complete, also sometime it fails with error `Unable to communicate with CMS (500)`. ---------------------------------- [root@srvlx40235 ~]# time ipa cert-find --users=p017079 --all ---------------------- 0 certificates matched ---------------------- ---------------------------- Number of entries returned 0 ---------------------------- real 1m6.283s user 0m0.529s sys 0m0.095s [root@srvlx40235 ~]# ---------------------------------- The customer got 125053 entries under `ou=certificateRepository,ou=ca,o=ipaca` when user details displayed on IPA web-UI. It looks for every single certificates in the database. ---- [29/Mar/2019:13:26:41.424312347 +0100] conn=265 op=2411 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL [29/Mar/2019:13:26:41.447810348 +0100] conn=265 op=2411 SORT serialno [29/Mar/2019:13:26:41.447824289 +0100] conn=265 op=2411 VLV 0:2147483647:99990:125049 99991:125049 (0) [29/Mar/2019:13:26:47.409452926 +0100] conn=265 op=2410 RESULT err=4 tag=101 nentries=10000 etime=15.1550448031 [29/Mar/2019:13:26:47.720526716 +0100] conn=265 op=2412 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL [29/Mar/2019:13:26:47.813523986 +0100] conn=265 op=2412 SORT serialno [29/Mar/2019:13:26:47.813541377 +0100] conn=265 op=2412 VLV 0:2147483647:49995:125049 49996:125049 (0) [29/Mar/2019:13:26:56.972922273 +0100] conn=265 op=2411 RESULT err=4 tag=101 nentries=10000 etime=15.0548812925 [29/Mar/2019:13:26:57.490204761 +0100] conn=265 op=2413 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL [29/Mar/2019:13:26:57.547431167 +0100] conn=265 op=2413 SORT serialno [29/Mar/2019:13:26:57.547445102 +0100] conn=265 op=2413 VLV 0:2147483647:109989:125049 109990:125049 (0) [29/Mar/2019:13:26:59.977039120 +0100] conn=265 op=2414 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=0 filter="(|(objectClass=*)(objectClass=ldapsubentry))" attrs="description" [29/Mar/2019:13:26:59.977385581 +0100] conn=265 op=2414 RESULT err=0 tag=101 nentries=1 etime=0.0001801085 [29/Mar/2019:13:27:03.623850364 +0100] conn=265 op=2412 RESULT err=4 tag=101 nentries=10000 etime=15.1095023500 [29/Mar/2019:13:27:03.774106766 +0100] conn=265 op=2415 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=1 filter="(certStatus=*)" attrs=ALL [29/Mar/2019:13:27:03.949381244 +0100] conn=265 op=2415 SORT serialno [29/Mar/2019:13:27:03.949395594 +0100] conn=265 op=2415 VLV 0:2147483647:59994:125049 59995:125049 (0) [29/Mar/2019:13:27:13.444654998 +0100] conn=265 op=-1 fd=81 closed error 104 (Connection reset by peer) - TCP connection reset by peer. --------- Version-Release number of selected component (if applicable): ipa-server-4.6.4-10.el7_6.3.x86_64 Additional info:
Upstream ticket: https://pagure.io/freeipa/issue/7901
I implemented an optimization for service and host certificates in RHBZ#1669012. The optimization doesn't apply to user certificates. I think it should be possible to optimize the user cert case, too. Fraser, do you remember if all user certificates have "cn=$username"?
I'm not sure it is safe to make assumptions on what the subject will contain. It could be keyed on uid, e-mail and/or cn, or some other available attribute (like principal).
The cert_request API endpoint enforces CN == username, https://github.com/freeipa/freeipa/blob/350954589774499d99bf87cb5631c664bb0707c4/ipaserver/plugins/cert.py#L747-L753 . Are there any ways to work around this restrictions?
Sorry, you're right. I was thinking more broadly about the all the searches that happen for a certificate. Given that we require cn=<uid> it is safe to query the CA on this IMHO.
Fixed upstream master: https://pagure.io/freeipa/c/8a5dc1b375db94c4e722fa725f48eb16d032f1aa
Fixed upstream ipa-4-7: https://pagure.io/freeipa/c/11907edc71f352f5b6960ed8c175099aac792e4c ipa-4-6: https://pagure.io/freeipa/c/b480a8a979682cb1613783904f9471dc18a5f207
Test data: About 30k users About 15k certs Test Machine memory [root@master ~]# free -mh total used free shared buff/cache available Mem: 3.7G 2.8G 143M 51M 756M 562M Swap: 2.0G 763M 1.3G [root@master ~]# CLI queries [root@master ~]# time ipa cert-find --subject=test103707 --all ipa: WARNING: Search result has been truncated: Configured size limit exceeded --------------------- 1 certificate matched --------------------- Issuing CA: ipa Certificate: MIIE8DCCA9igAwIBAgICL3UwDQYJKoZIhvcNAQELBQAwOTEXMBUGA1UECgwOVEVTVFJFQUxNLlRFU1QxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTEyMTcxMTUzMTdaFw0yMTEyMTcxMTUzMTdaMDwxFzAVBgNVBAoMDlRFU1RSRUFMTS5URVNUMSEwHwYDVQQDDBh0ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDME/AB1omh41fAasyDsSUGiB/5dXaOPVNI6LfL7Ui+36iDT+a/DPc9Unr+oKE+uMKG4NJilRo5EXdsUGJHZVmUOVyiVAHyLUOjNkZmTBzeiw5JFERRNCEQREqqdV4sFufucr0qYn53aY2+oE61oW0Q9bBCi2Q4Sn+BWJkMQT7Qajb57xheIzrJaqKdPLazH2P+IEDPxoyBM9PXKuXGvWqQdrpbACgGlfDUEV9bymPfFzHpOgTPVxoPKI8l0fgnH9aQEQkceHjdDxP0MYSb1nfbGUMqCqVqt/DRlIF7Fql6Bv0QdnFCkSkblwXw0cXujlpK4LyxxJUmCYqwN01Y+zTpAgMBAAGjggH9MIIB+TAfBgNVHSMEGDAWgBTNVHjlypVlKsjcWUCGUssE1G7MTjBABggrBgEFBQcBAQQ0MDIwMAYIKwYBBQUHMAGGJGh0dHA6Ly9pcGEtY2EudGVzdHJlYWxtLnRlc3QvY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMHkGA1UdHwRyMHAwbqA2oDSGMmh0dHA6Ly9pcGEtY2EudGVzdHJlYWxtLnRlc3QvaXBhL2NybC9NYXN0ZXJDUkwuYmluojSkMjAwMQ4wDAYDVQQKDAVpcGFjYTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB0GA1UdDgQWBBRCrFYJVADwkjUF9WcrV8NyFKxodjCBygYDVR0RBIHCMIG/ghh0ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3SgSQYKKwYBBAGCNxQCA6A7DDl0ZXN0c2VydmljZTEwMzcwNy90ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3RAVEVTVFJFQUxNLlRFU1SgWAYGKwYBBQICoE4wTKAQGw5URVNUUkVBTE0uVEVTVKE4MDagAwIBAaEvMC0bEXRlc3RzZXJ2aWNlMTAzNzA3Gxh0ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3QwDQYJKoZIhvcNAQELBQADggEBAGoYnC+6u3uS9B+ajkUXdMJpOT+uWw1b1e5aUzgXFXsWRyMcOhK8ULgOikJrNllVk2HojSp/hbsS+6zqVLTG5WSsaaDlWpLJeesr7AL+GkooOpQViPHzUtImSslyQOclHj27kkVAckmxWKcUtQjMNzSo431kCpIp3TggEr0fQ7Uwk9P5ZGIn7tgAwXyelzILjy36oKYed4BKFG3AuaNMYTNjWTBhUVSu1qmPMTBeLEzmZpqD3sB2sfRpBDQHB6noXCV0tSD4EyneIHwv8zAoOtZ8yVC3Ds/WbdAL2ymgQAJSes9XNSmrpUCZfMpeQBK3PQaLuQM/USLNm0QAvpIUJS4= Certificate chain: MIIE8DCCA9igAwIBAgICL3UwDQYJKoZIhvcNAQELBQAwOTEXMBUGA1UECgwOVEVTVFJFQUxNLlRFU1QxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTEyMTcxMTUzMTdaFw0yMTEyMTcxMTUzMTdaMDwxFzAVBgNVBAoMDlRFU1RSRUFMTS5URVNUMSEwHwYDVQQDDBh0ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDME/AB1omh41fAasyDsSUGiB/5dXaOPVNI6LfL7Ui+36iDT+a/DPc9Unr+oKE+uMKG4NJilRo5EXdsUGJHZVmUOVyiVAHyLUOjNkZmTBzeiw5JFERRNCEQREqqdV4sFufucr0qYn53aY2+oE61oW0Q9bBCi2Q4Sn+BWJkMQT7Qajb57xheIzrJaqKdPLazH2P+IEDPxoyBM9PXKuXGvWqQdrpbACgGlfDUEV9bymPfFzHpOgTPVxoPKI8l0fgnH9aQEQkceHjdDxP0MYSb1nfbGUMqCqVqt/DRlIF7Fql6Bv0QdnFCkSkblwXw0cXujlpK4LyxxJUmCYqwN01Y+zTpAgMBAAGjggH9MIIB+TAfBgNVHSMEGDAWgBTNVHjlypVlKsjcWUCGUssE1G7MTjBABggrBgEFBQcBAQQ0MDIwMAYIKwYBBQUHMAGGJGh0dHA6Ly9pcGEtY2EudGVzdHJlYWxtLnRlc3QvY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMHkGA1UdHwRyMHAwbqA2oDSGMmh0dHA6Ly9pcGEtY2EudGVzdHJlYWxtLnRlc3QvaXBhL2NybC9NYXN0ZXJDUkwuYmluojSkMjAwMQ4wDAYDVQQKDAVpcGFjYTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB0GA1UdDgQWBBRCrFYJVADwkjUF9WcrV8NyFKxodjCBygYDVR0RBIHCMIG/ghh0ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3SgSQYKKwYBBAGCNxQCA6A7DDl0ZXN0c2VydmljZTEwMzcwNy90ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3RAVEVTVFJFQUxNLlRFU1SgWAYGKwYBBQICoE4wTKAQGw5URVNUUkVBTE0uVEVTVKE4MDagAwIBAaEvMC0bEXRlc3RzZXJ2aWNlMTAzNzA3Gxh0ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3QwDQYJKoZIhvcNAQELBQADggEBAGoYnC+6u3uS9B+ajkUXdMJpOT+uWw1b1e5aUzgXFXsWRyMcOhK8ULgOikJrNllVk2HojSp/hbsS+6zqVLTG5WSsaaDlWpLJeesr7AL+GkooOpQViPHzUtImSslyQOclHj27kkVAckmxWKcUtQjMNzSo431kCpIp3TggEr0fQ7Uwk9P5ZGIn7tgAwXyelzILjy36oKYed4BKFG3AuaNMYTNjWTBhUVSu1qmPMTBeLEzmZpqD3sB2sfRpBDQHB6noXCV0tSD4EyneIHwv8zAoOtZ8yVC3Ds/WbdAL2ymgQAJSes9XNSmrpUCZfMpeQBK3PQaLuQM/USLNm0QAvpIUJS4=, MIIDlDCCAnygAwIBAgIBATANBgkqhkiG9w0BAQsFADA5MRcwFQYDVQQKDA5URVNUUkVBTE0uVEVTVDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTE5MTIwOTA1NTIyMloXDTM5MTIwOTA1NTIyMlowOTEXMBUGA1UECgwOVEVTVFJFQUxNLlRFU1QxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKld8aPzbfnNZ6L3uNnj6A/WPY3R8WItoIs7smKLFF6NV1RMrw7jov9WEvakJthirn88ViXDSk7zmKq5G5SEx/uCoFcQ47OcBwGzjv3oRWLGSPMOam2TfaTfLnkDdDIHu33FeKKPDk1cTnyTc2iECZXY7UjlO/aMGp1xuxR8uP/qd0083LlIeHaCjDsjwmbF7GPnl+5bD8yt09s2SHpZOsyopWgPIMzJZ8NNJ/BhlTG93RHpXF1xNd2qZ+Nou+4lDD0BeeDjYUoXkg2Wp66tRF5GrKZz2SfIGEEcZYWf8WkBI756Yd6JBUrOD+RujRHBagoQqcJiSkiUmnlFP7YRFsECAwEAAaOBpjCBozAfBgNVHSMEGDAWgBTNVHjlypVlKsjcWUCGUssE1G7MTjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUzVR45cqVZSrI3FlAhlLLBNRuzE4wQAYIKwYBBQUHAQEENDAyMDAGCCsGAQUFBzABhiRodHRwOi8vaXBhLWNhLnRlc3RyZWFsbS50ZXN0L2NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAIIg54wFzMaY8faeSSl1EgSC4dhcEvtRN650iIveBWb/+fe7wFGZL6+vx0IPvVp3z+mzzKBj+fZ2dx5gX2ik+bC0ey4l4Xv7B4cuf8NbRNkWdJZRQuqugFuKp6JL/oCTgq2N7rQ+WMkrQV1Ct9In6E8kY48sEtPzeWmlG0CU004bGSnkfqNNaoQ4b3OhnMi6RCivuTAEg5yqPnJemcyR3SPlgLgQXv3/VG1me9Y2aHarrBnGazAX7XfG5sIgy/Ctk9psBBItEmsbAF/WZptvKXPpVDLhxeni+AhQZwjD5zVhOQDqUo4XuDtcGoGU3TNdI122ANABVU+WsO4Ft2f9EAE= Subject: CN=test103707.testrelm.test,O=TESTREALM.TEST Subject DNS name: test103707.testrelm.test Subject UPN: testservice103707/test103707.testrelm.test Subject Kerberos principal name: testservice103707/test103707.testrelm.test Subject Other Name: 1.3.6.1.4.1.311.20.2.3:DDl0ZXN0c2VydmljZTEwMzcwNy90ZXN0MTAzNzA3LnRlc3RyZWxtLnRlc3RAVEVTVFJFQUxNLlRFU1Q=, 1.3.6.1.5.2.2:MEygEBsOVEVTVFJFQUxNLlRFU1ShODA2oAMCAQGhLzAtGxF0ZXN0c2VydmljZTEwMzcwNxsYdGVzdDEwMzcwNy50ZXN0cmVsbS50ZXN0 Issuer: CN=Certificate Authority,O=TESTREALM.TEST Not Before: Tue Dec 17 11:53:17 2019 UTC Not After: Fri Dec 17 11:53:17 2021 UTC Fingerprint (SHA1): b7:a5:20:51:12:1c:0e:36:99:93:39:fe:ec:1a:b4:93:d7:9b:87:3a Fingerprint (SHA256): 20:75:ba:43:8d:cb:fc:3c:95:0c:c4:9e:ad:22:db:cc:af:f1:77:fe:e9:4c:86:05:7e:18:32:ed:6b:ce:88:b6 Serial number: 12149 Serial number (hex): 0x2F75 Status: VALID Revoked: False ---------------------------- Number of entries returned 1 ---------------------------- real 0m23.673s user 0m0.500s sys 0m0.063s [root@master ~]# [root@master ~]# time ipa cert-find --users=test103707 --all ---------------------- 0 certificates matched ---------------------- ---------------------------- Number of entries returned 0 ---------------------------- real 0m0.884s user 0m0.448s sys 0m0.073s [root@master ~]# [root@master ~]# time ipa user-find --login user2121 -------------- 1 user matched -------------- User login: user2121 First name: user2121 Last name: user2121 Home directory: /other-home/user2121 Login shell: /bin/zsh Principal name: user2121 Principal alias: user2121 Email address: user2121 UID: 405002123 GID: 405002123 SSH public key fingerprint: SHA256:cStA9o5TRSARbeketEOooMUMSWRSsArIAXloBZ4vNsE public key test (ssh-rsa) Account disabled: False ---------------------------- Number of entries returned 1 ---------------------------- real 0m0.685s user 0m0.516s sys 0m0.077s [root@master ~]# WEB UI: 1. trying to find a user https://master.testrealm.test/ipa/ui/#/e/user/search//filter=user21212 almost instant 2. Accessing the Certificates tab, it takes a lot of time to load the data about ~15-20 seconds 3. Querying for a particular user https://master.testrealm.test/ipa/ui/#/e/cert/search//search_option=subject&filter=test103707 The above took 27-28 seconds Based on above observations, marking the bug in ASSIGNED state since querying is taking significant time and this needs further investigation.
Build used for testing: [root@master ~]# rpm -qa ipa-* ipa-common-4.6.6-11.el7.noarch ipa-client-common-4.6.6-11.el7.noarch ipa-client-4.6.6-11.el7.x86_64 ipa-server-4.6.6-11.el7.x86_64 ipa-server-trust-ad-4.6.6-11.el7.x86_64 ipa-server-common-4.6.6-11.el7.noarch ipa-server-dns-4.6.6-11.el7.noarch
Thank you taking your time and submitting this request for Red Hat Enterprise Linux 7. Unfortunately, this bug cannot be kept even as a stretch goal and was postponed to RHEL8.