Bug 1925370

Hi Carl,

Just to make sure I understand: the applications linking against this specific library are user/external applications or do you mean others from within RHEL distro?
I'm asking it because the only package from within RHEL default distro that uses libimaevm is the RPM package itself (more precisely, rpm-sign subpackage), which was also updated accordingly in RHEL-8.4 to support the soname bump.
OTOH, can you give me some guidance on how to request the ACG level change?

Many thanks!

I understand that the only thing in the distribution that links against this is rpm.  I first noticed this when CentOS koji build roots started failing due to the soname change.  I was able to resolve it with a quick bootstrap build [0].  I'm concerned about external user/customer applications since we ship ima-evm-utils-devel and they can build against it.  I don't have specific examples of this, so it is admittedly a hypothetical concern.  But it is still breaking ACG which should be avoided.

I'm not familiar with the ACG level change process, so I'm hoping Josh can shed more light on that part.

[0] https://git.centos.org/rpms/rpm/c/a0a02b685f4b103f0b2a446aee719752b7bd8bb0?branch=c8s-imaevm-bootstrap

This change demanded several back'n'forth changes in coordination with RPM maintainers to sort out the "chicken and egg" situation until both rpm-4.13.4-10 and ima-evm-utils-1.3.2-11 landed.
To be honest I'm not really aware of any external applications linking against it. Even being part of upstream development I've never seen a public claim about the libimaevm usage other than rpm.
But, /of course/, I may be REALLY wrong/misinformed.

What made me move forward was a request from IBM based on some features enabled in the kernel side.
Unfortunately, this package went unmaintained for quite some time, until I took official responsibility for it in both kernel and userspace package, and, as expected, the userspace tool was quite deprecated and the update was a "must". And in this case I'm willing to ask for an "exception" with the guarantee that no other soname bumps are going to happen in RHEL8.

I understand the ACG limits are there for a reason and I was (and still am) not fully aware of its extension (internal vs external apps).
So yea, any feedback is welcome :-).

(In reply to Bruno Meneguele from comment #3)
> I understand the ACG limits are there for a reason and I was (and still am)
> not fully aware of its extension (internal vs external apps).
> So yea, any feedback is welcome :-).

The specific case of a -devel in CRB and the library in BaseOS is covered in the CRB FAQ of the RHEL Developer guide.

https://one.redhat.com/rhel-developer-guide/#_crb_faq

If you need to bump your SONAME then you need to provide a compat package ima-evm-utils0 that provides libimaevm.so.0.

You should not break ABI, doing so reduces the value of a RHEL subscription for our customers.

We do not have observability into everything our customers are doing with our libraries, but we can know and see when we make decisions that break ABI for them.

Yes, it's extra work to create a ima-evm-utils0, and if you can provide strong justification that nobody could possibly be using it, then we can agree not to provide it, but that should be a discussion you have with your manager, and the program because we are make a conscious business decision about what not to ship.

(In reply to Carlos O'Donell from comment #4)
> (In reply to Bruno Meneguele from comment #3)
> > I understand the ACG limits are there for a reason and I was (and still am)
> > not fully aware of its extension (internal vs external apps).
> > So yea, any feedback is welcome :-).
> 
> The specific case of a -devel in CRB and the library in BaseOS is covered in
> the CRB FAQ of the RHEL Developer guide.
> 
> https://one.redhat.com/rhel-developer-guide/#_crb_faq
> 
> If you need to bump your SONAME then you need to provide a compat package
> ima-evm-utils0 that provides libimaevm.so.0.
> 
> You should not break ABI, doing so reduces the value of a RHEL subscription
> for our customers.

Note this is a public CentOS Stream version bug.
The above seems to be a (non-public) RHEL developer guide.
Is there a public CentOS developer guide to reference for CentOS Stream versions?

(In reply to Mark Wielaard from comment #5)
> (In reply to Carlos O'Donell from comment #4)
> > (In reply to Bruno Meneguele from comment #3)
> > > I understand the ACG limits are there for a reason and I was (and still am)
> > > not fully aware of its extension (internal vs external apps).
> > > So yea, any feedback is welcome :-).
> > 
> > The specific case of a -devel in CRB and the library in BaseOS is covered in
> > the CRB FAQ of the RHEL Developer guide.
> > 
> > https://one.redhat.com/rhel-developer-guide/#_crb_faq
> > 
> > If you need to bump your SONAME then you need to provide a compat package
> > ima-evm-utils0 that provides libimaevm.so.0.
> > 
> > You should not break ABI, doing so reduces the value of a RHEL subscription
> > for our customers.
> 
> Note this is a public CentOS Stream version bug.
> The above seems to be a (non-public) RHEL developer guide.
> Is there a public CentOS developer guide to reference for CentOS Stream
> versions?

Can you elaborate a bit on the question?  What would you expect to see in a Stream specific guide?  It is worth remembering that Stream and RHEL are virtually synonymous and that only Red Hat employees can directly commit changes to Stream.

As for the ACG itself, we have the customer facing ACG documentation.  We could certainly link to it from a CentOS Stream wiki as part of the considerations that go into making decisions for Stream and RHEL if others fine that helpful.

(In reply to Josh Boyer from comment #7)
> (In reply to Mark Wielaard from comment #5)
> > Note this is a public CentOS Stream version bug.
> > The above seems to be a (non-public) RHEL developer guide.
> > Is there a public CentOS developer guide to reference for CentOS Stream
> > versions?
> 
> Can you elaborate a bit on the question?  What would you expect to see in a
> Stream specific guide?  It is worth remembering that Stream and RHEL are
> virtually synonymous and that only Red Hat employees can directly commit
> changes to Stream.

It is great that you get a Red Hat employee to commit your changes for you but it would be even better if there was a description of what would help getting patches accepted faster and/or not getting them reversed. So for example what tests need to pass? In this case I assume some rpm based libabigail checks are done. How do you as submitter know which checks need to pass? How do you run them or are they run for you on submission? What if any of them fail, but for unrelated reasons, can you contest them, flag them as benign? In this particular case, do you have to submit a new compatibility package to go together with the update to make the checks pass again?

(In reply to Mark Wielaard from comment #8)
> (In reply to Josh Boyer from comment #7)
> > (In reply to Mark Wielaard from comment #5)
> > > Note this is a public CentOS Stream version bug.
> > > The above seems to be a (non-public) RHEL developer guide.
> > > Is there a public CentOS developer guide to reference for CentOS Stream
> > > versions?
> > 
> > Can you elaborate a bit on the question?  What would you expect to see in a
> > Stream specific guide?  It is worth remembering that Stream and RHEL are
> > virtually synonymous and that only Red Hat employees can directly commit
> > changes to Stream.
> 
> It is great that you get a Red Hat employee to commit your changes for you
> but it would be even better if there was a description of what would help
> getting patches accepted faster and/or not getting them reversed. So for
> example what tests need to pass? In this case I assume some rpm based
> libabigail checks are done. How do you as submitter know which checks need
> to pass? How do you run them or are they run for you on submission? What if
> any of them fail, but for unrelated reasons, can you contest them, flag them
> as benign? In this particular case, do you have to submit a new
> compatibility package to go together with the update to make the checks pass
> again?

Ah, yes.  These are all great questions.  That kind of information is part of the roadmap for when Stream contributions from the public are enabled in a broader manner.

I just ran into this issue. We have an internal rebuild of rpm, which was linked against libimaevm.so.0. Since that point we've pulled in another snapshot of CentOS 8 Stream and upgraded all our servers to it. That snapshot only contains libimaevm.so.2 so attempts to install the older[1] package would end up removing dnf due to an unsatisfied dependency.

I see[2] that there was a new, compatibility package included. However I do not see ima-evm-utils0 in the published repos. I wonder if this new package was accidentally omitted. If it had been published, then we likely wouldn't have experience this issue.

Would you consider publishing ima-evm-utils0 to resolve in a more correct way going forward?

[1] Am working on moving our build to the Hyperscale SIG
[2] https://git.centos.org/rpms/ima-evm-utils/blob/c8s/f/SPECS/ima-evm-utils.spec#_48

(In reply to Matthew Almond from comment #39)
> I just ran into this issue. We have an internal rebuild of rpm, which was
> linked against libimaevm.so.0. Since that point we've pulled in another
> snapshot of CentOS 8 Stream and upgraded all our servers to it. That
> snapshot only contains libimaevm.so.2 so attempts to install the older[1]
> package would end up removing dnf due to an unsatisfied dependency.
> 
> I see[2] that there was a new, compatibility package included. However I do
> not see ima-evm-utils0 in the published repos. I wonder if this new package
> was accidentally omitted. If it had been published, then we likely wouldn't
> have experience this issue.
> 
> Would you consider publishing ima-evm-utils0 to resolve in a more correct
> way going forward?
> 
> [1] Am working on moving our build to the Hyperscale SIG
> [2]
> https://git.centos.org/rpms/ima-evm-utils/blob/c8s/f/SPECS/ima-evm-utils.
> spec#_48

Thanks for the report Matthew.

Josh, I expected that the changes from RHEL-8.4 compose would be propagated towards CentOS (comment#34).
Are there anything else needed or is it a timing issue (8.4 GA will solve it)?

(In reply to Bruno Meneguele from comment #40)
> (In reply to Matthew Almond from comment #39)
> > I just ran into this issue. We have an internal rebuild of rpm, which was
> > linked against libimaevm.so.0. Since that point we've pulled in another
> > snapshot of CentOS 8 Stream and upgraded all our servers to it. That
> > snapshot only contains libimaevm.so.2 so attempts to install the older[1]
> > package would end up removing dnf due to an unsatisfied dependency.
> > 
> > I see[2] that there was a new, compatibility package included. However I do
> > not see ima-evm-utils0 in the published repos. I wonder if this new package
> > was accidentally omitted. If it had been published, then we likely wouldn't
> > have experience this issue.
> > 
> > Would you consider publishing ima-evm-utils0 to resolve in a more correct
> > way going forward?
> > 
> > [1] Am working on moving our build to the Hyperscale SIG
> > [2]
> > https://git.centos.org/rpms/ima-evm-utils/blob/c8s/f/SPECS/ima-evm-utils.
> > spec#_48
> 
> Thanks for the report Matthew.
> 
> Josh, I expected that the changes from RHEL-8.4 compose would be propagated
> towards CentOS (comment#34).
> Are there anything else needed or is it a timing issue (8.4 GA will solve
> it)?

It's already in Stream

http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ima-evm-utils0-1.3.2-12.el8.x86_64.rpm

[jwboyer@vader nvml]$ rpm -qpP http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ima-evm-utils0-1.3.2-12.el8.x86_64.rpm
warning: http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ima-evm-utils0-1.3.2-12.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
ima-evm-utils0 = 1.3.2-12.el8
ima-evm-utils0(x86-64) = 1.3.2-12.el8
libimaevm.so.0()(64bit)
[jwboyer@vader nvml]$ 

Matthew, perhaps you need to update your snapshot.

It's fixed now: https://git.centos.org/centos/pungi-centos/c/97c88a95f11227dadca6ed7549388ffd16087b90?branch=centos-8-stream . I guess this doesn't happen too often, so it's not a huge deal.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ima-evm-utils bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1720