Bug 1978038
| Summary: | pesign fails when run with nss-3.67 in the buildroot | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Brian Stinson <bstinson> |
| Component: | nss | Assignee: | nss-nspr-maint <nss-nspr-maint> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Ivan Nikolchev <inikolch> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 9.0 | CC: | bstinson, hkario, hkrzesin, inikolch, jlelli, jwboyer, rrelyea, vdronov |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | nss-3.67.0-6.el9 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-07 21:33:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1978430 | ||
|
Description
Brian Stinson
2021-07-01 02:14:29 UTC
I manually rebuilt and downgraded to nss-3.63.0-3.el9 and nspr-4.30.0-2.el9 based packages, and the problem with pesign is gone. So definitely it's a problem with the nss/nspr update, something in it broke pesign. To test it, you can just install pesign on x86_64 or aarch64 based host, and do: - Before the nss update: $ pesign --certdir /etc/pki/pesign-rh-test -c 'Red Hat Test Certificate' -s -i /boot/vmlinuz-5.13.0-0.rc7.51.el9.x86_64 -o vmlinuz.tmp $ # it worked $ rm vmlinuz.tmp - After the nss update: # rpm -qa | grep "^nss" nss-util-3.67.0-4.el9.x86_64 nss-util-devel-3.67.0-4.el9.x86_64 nss-softokn-freebl-3.67.0-4.el9.x86_64 nss-softokn-3.67.0-4.el9.x86_64 nss-3.67.0-4.el9.x86_64 nss-sysinit-3.67.0-4.el9.x86_64 nss-softokn-freebl-devel-3.67.0-4.el9.x86_64 nss-softokn-devel-3.67.0-4.el9.x86_64 nss-devel-3.67.0-4.el9.x86_64 nss-tools-3.67.0-4.el9.x86_64 # rpm -qa | grep "^nspr" nspr-4.31.0-2.el9.x86_64 nspr-devel-4.31.0-2.el9.x86_64 $ pesign --certdir /etc/pki/pesign-rh-test -c 'Red Hat Test Certificate' -s -i /boot/vmlinuz-5.13.0-0.rc7.51.el9.x86_64 -o vmlinuz.tmp error signing data: The key does not support the requested operation. Could not generate signed data: No such file or directory It seems for some reason can't read/work with data in /etc/pki/pesign-rh-test/ anymore. btw, to test you can use any vmlinuz-* file available at /boot with pesign command. OK I'll take a look. I see it works on rhel-8.4.z with nss-3.67, so we may be looking at a policy issue. OK the issue is the code to check the signatures on private key attributes. That patch hasn't been pushed upstream yet. There appears to be an issue with the code that is supposed to handle the old broken databases. It appears rhel8 doesn't have a broken database, but rhel9 does. I should have a patch to fix this shortly. hello, my apologies, i've bumped the severity/priority to urgent. ALL rhel9 brew build are failing, could you please look at it based on urgent priority? No problem, I've pushed a patch to centos now. For posterity, here is what the issue was: History: When NSS moved to using AES_CBC to store keys, it added integrity checks to the database. Unfortunately the code was broken in 2 ways: 1) The password update code messed up the integrity check indices, but that wasn't notices because, 2) The code that checked the integrity was broken, so the integrity checks weren't being done. We fixed both of these issues in rhel-8 in the last rebase, and I added code that if the integrity checks fail, to look for them in the incorrect indices. These fixes where not pushed upstream (sigh - my #1 priority this quarter is to clear the list of patches that belong upstream), so fedora did not get these patches. pesign did not move to the sqlite database in rhel-8. The old dbm database cannot store any integrity values, so integrity checks are skipped on dbm, so we saw no issues with pesign. When we dropped dbm in fedora f34, pesign needed to update their databases to sqlite, which they did using fedora's unpatched nss. This new database now has only the incorrect integrity check indices. NSS code still expects the correct indices to be there (the old broken code created the indicies correctly, they just didn't get updated on password update). On a database update (which is fundamentally implemented with the same code as a password update), the correct integrity indices were not being created. As a result, the code that worked well for databases that had their passwords changed on rhel8, was failing on pesign's database. We could fix this by creating a correctly indexed database in pesign (taking the old dbm database on rhel-8 and using the db update feature of certutil on rhel8 rather then fedora), and we probably should fix it as some point, but NSS should be able to handle this kind of database without errors anyway pesign can't be the only one that will hit this issue. I've pushed a patch that should fix this to centos stream, and it should clear the pipeline as fast as the pipeline can work (well nss builds have very long tests, so it will be a while). (In reply to Bob Relyea from comment #8) > No problem, I've pushed a patch to centos now. > For posterity, here is what the issue was: Thanks a ton, Bob! Your explanation is a great read, I've read it as a detective book)) (In reply to Vladis Dronov from comment #9) > (In reply to Bob Relyea from comment #8) > > No problem, I've pushed a patch to centos now. > > For posterity, here is what the issue was: > > Thanks a ton, Bob! Your explanation is a great read, I've read it as a > detective book)) Indeed, it was an excellent explanation, thanks. Bob, it looks like nss is still stuck in gating (https://dashboard.osci.redhat.com/#/artifact/brew-build/aid/37875621), would you be able to look at it? While still in gating it's not available yet eg. on buildroot so builds with pesign continue to fail. |