Will you be able to branch and build jemalloc in epel8?
The EPEL Packagers SIG would be happy to be a co-maintainer
if you do not wish to build it on epel8.

Hi

This should have been fixed by FEDORA-EPEL-2019-d8764760e7 which is marked as stable, ref BZ#1756993. jemalloc-5.2.1-2.el8 should be available in EPEL8 since August 2019. Is there anything I have missed here?

Ingvar

Hi Hagelund

You’re right 5.2.1 has already been available. I was referring to the newer 5.3.0, which has new features but is only on Fedora. I’m most happy to help package and maintain that newer version for epel8.

We should not update just for update's sake. From https://docs.fedoraproject.org/en-US/epel/epel-policy-updates/  :  

(...) All updates should strive to avoid situations that require manual intervention to keep the package functioning after update. Major updates with changes to user experience are to be avoided. (...)

There are incompatible changes (at least one) between jemalloc-5.2 and 5.3. As there are no known security issues nor crashing bugs reported for jemalloc-5.2.1, jemalloc should not be updated according to the policy. This is the way EL packages always have been treated.

One variant could be adding a separate package for jemalloc-5.3.0 We can't have two packages with the same name, so that would mean making special variants with versions hard coded into the name. I would like to avoid that.

Using module streams may be a way to work around this. Suggestions?

Ingvar

Many thanks for this, Ingvar!

Using module streams does indeed sound like the best solution in this case, and is also the one recommended in the guidelines.

Would you have time to work on this? I look forward to seeing it in EPEL and would be ready to help out if need be.

Please do NOT use modules in EPEL.  Building modules in EPEL has been discontinued.
Our current recommendation is to do it the non-modular way Fedora does it, which I believe is to name it <package><version>.
I realize that we missed some EPEL documenation that says to use modules.  We are now fixing those.

Troy is correct about the naming.  The preferred naming of an alternate jemalloc package would be jemalloc5.3.

https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#multiple

You would also need to resolve conflicting paths.  Normally in the case of an incompatible update, the library soname is different already so the library file doesn't need to be renamed.  In this case, both jemalloc 5.2 and 5.3 ship libjemalloc.so.2.  Are we sure it's actually incompatible?  What are the incompatible changes?  If they're not incompatible enough for upstream to bump the library soname, then perhaps it's compatible enough to allow as a regular update to the main jemalloc package.

from https://github.com/jemalloc/jemalloc/releases/tag/5.3.0 :

Incompatible changes:

    Maximum size class allowed in tcache (opt.[lg_]tcache_max) now has an upper
    bound of 8MiB.

If you Red Hat techies will tell me with conviction that this change will mean no problems for any user, then by all means, let's update the package. But again from the policy:

    All updates should strive to avoid situations that require manual intervention
    to keep the package functioning after update.

We don't know who uses jemalloc and for what purpose. If this is a real problem in someone's code somewhere, they will have to fix their code and rebuild. For all I know, this is not a problem at all anywhere. But it may also make someone's website crash, dialysis machine implode, or production line stop. EPEL packagers should strive for the same stability that RHEL has. If not, why care about policies, or even use enterprise linux and and EPEL in the first place?

Ingvar

Looks like this leaves us with jemalloc5.3 as the package name if we are to make 5.3.x available in EPEL 8 and 9.

Ingvar, would you be able to do it?

> If you Red Hat techies will tell me with conviction that this change will mean no problems for any user, then by all means, let's update the package.

It doesn't sound like a big deal to me, and again if it was then I would expect upstream to bump the library soname.  That said, I'm not that familiar with the software and would defer to your judgement on it as the package maintainer, or ask you to seek clarification from upstream if you're not sure.  I did find the exact commit that makes this change if you want to review it.

https://github.com/jemalloc/jemalloc/commit/5e41ff9b740258bddebcbd5575e1670a15f8b1ae

> We don't know who uses jemalloc and for what purpose. If this is a real problem in someone's code somewhere, they will have to fix their code and rebuild. For all I know, this is not a problem at all anywhere. But it may also make someone's website crash, dialysis machine implode, or production line stop. EPEL packagers should strive for the same stability that RHEL has. If not, why care about policies, or even use enterprise linux and and EPEL in the first place?

Obviously you are correct about the intent, per the guidelines.  That said, EPEL packages are community maintained, and there are no guarantees.  If someone wants the level of guarantees you might expect for things like dialysis machines or production lines, they should seek out a vendor and establish a commercial relationship with SLAs defined in a contract.  Packages somewhat regularly are "promoted" from EPEL to RHEL for this reason.  Even with a vendor, regressions can and do still happen.  We can't hold volunteer EPEL maintainers to a higher standard that RHEL.  As long as a maintainer feels that an update is compatible enough, it is allowed.  And if you want to be extra cautious, we do have a process for proceeding with a (potentially) incompatible update that involves additional coordination and communication.

https://docs.fedoraproject.org/en-US/epel/epel-policy-incompatible-upgrades/

> Looks like this leaves us with jemalloc5.3 as the package name if we are to make 5.3.x available in EPEL 8 and 9.

A parallel package won't be easy since jemalloc and jemalloc5.3 would conflict on /usr/lib64/libjemalloc.so.2.  You may have to perform some complex surgery on the source to properly namespace the library and headers into a different path.  It would also require that software built against the namespaced headers supports alternate paths, which isn't guaranteed.  If the latest version of jemalloc bumped the soname to 3 it would become significantly easier.  It may make sense to just defer this request until upstream decides to bump the soname.

Many thanks to George and Dawson for your input. I agree it’s up to Ingvar to make the call.

On the implementation of the jemalloc5.3 package, it seems to me that there’s no need for it to co-exist with jemalloc because any end user who’s decided it’s appropriate for them to upgrade to 5.3 can safely remove the 5.2 package.

And we can further facilitate this optional upgrade path through a Conflicts or Obsoletes tag in the spec for jemalloc5.3.

Am I correct about this being viable in EPEL?

A jemalloc5.3 package doesn't make sense because the library soname is the same as the current jemalloc package.  I would recommend just upgrading the jemalloc package from 5.2.1 to 5.3.0.  If the tcache incompatibility that upstream mentions is deemed important, then that upgrade can be done via the incompatible upgrade process I linked to provide notice to the community.

I would certainly welcome 5.3.0 going the incompatible upgrade route!

Would you mind just explaining to me why the .so file in the current jemalloc package can’t be replaced by a .so file with the same name from jemalloc5.3? I was under the impression that by removing the old package and installing the new one, only one file of that soname will end up in the /lib64 directory, thus posing no danger of a conflict.

That is, what would be the use case of having ‘parallel’ 5.2 and 5.3 on a system which you alluded to, instead of just having one replace the other at the discretion of the sysadmin?

Since 5.2 and 5.3 have the same library soname, there isn't a need to have both be installed in parallel.  When there is a different library soname, it can make sense.  Parallel packages need to resolve all file conflicts, either by the files already being different or by renaming/deleting them.  The exception is the -devel packages, which would just conflict with each other.  For example, if someone wanted to run an application on RHEL 8 but it strictly required libjemalloc.so.1, it wouldn't work with the jemalloc currently in EPEL 8.  They could create a parallel jemalloc3.6 compatibility package to solve this.

jemalloc, version 5.2.1:
/usr/bin/jemalloc.sh
/usr/lib64/libjemalloc.so.2
/usr/share/doc/jemalloc/*

jemalloc3.6, version 3.6.0:
/usr/bin/jemalloc3.6.sh
/usr/lib64/libjemalloc.so.1
/usr/share/doc/jemalloc3.6/*

Or for another example, let's say hypothetically that jemalloc released version 6.0.0 with libjemalloc.so.3.  In EPEL 8, we could update jemalloc to the new version if we provided a compatibility package for applications that still need the existing soname.

jemalloc, version 6.0.0:
/usr/bin/jemalloc.sh
/usr/lib64/libjemalloc.so.3
/usr/share/doc/jemalloc/*

jemalloc5.2, version 5.2.1:
/usr/bin/jemalloc5.2.sh
/usr/lib64/libjemalloc.so.2
/usr/share/doc/jemalloc5.2/*

Since in this case the soname hasn't changed, just updating jemalloc from 5.2.1 to 5.3.0 makes the most sense.

P.S. You don't need to set the needsinfo flag for me each time you reply.  I get one email for the update to the bug, then another email for the needsinfo flag.  Also, bugzilla will resend the needsinfo email every couple of days until the flag is cleared.  Most people use needsinfo as an extra reminder when someone hasn't replied for a while.

I believe I can now understand the thinking behind your point about just updating jemalloc from 5.2.1 to 5.3.0 making the most sense. Thank you very much for taking the time to educate me on this. (And thanks for pointing out my Bugzilla faux pas!)

Apologies for the digress.

Ingvar, could you please give an update of your decision on this? Is jemalloc 5.3 undergoing EPEL’s incompatible upgrade process?

Many thanks!

> We don't know who uses jemalloc and for what purpose. If this is a real problem in someone's code somewhere, they will have to fix their code and rebuild. For all I know, this is not a problem at all anywhere. But it may also make someone's website crash, dialysis machine implode, or production line stop. EPEL packagers should strive for the same stability that RHEL has. If not, why care about policies, or even use enterprise linux and and EPEL in the first place?

To give more context, I am going to explain what the change actually does.  Perhaps it would be better to start with that before using "dialysis machine" as an argument?

The "incompatible" change they have introduced just clamps the maximum value of `opt.tcache_max`.  The **tcache** is a thread-specific cache (tcache) (kind of implicit memory pool) and clamping the value to 8MiB just means that the allocations will skip the tcache and be allocated from the arena.  No program is going to break as the configured value is clamped and it doesn't fail.  Also no memory allocations are going to fail.

This is very easy to test, the default value is `23` (2^23):

MALLOC_CONF="abort:true,tcache:true,tcache_max:23" LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 /bin/ls

Anything that is larger than that:

MALLOC_CONF="abort:true,tcache:true,tcache_max:32" LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 /bin/ls

You can add `stats_print:true` to `MALLOC_CONF` for the program to print out statistics from the allocator - this is called from `atexit()`.

To test you need a reasonably complex program that allocates and deallocates some chunks of memory, but the gist is that the jemalloc developers are saying that if you are storing objects larger than 8MiB in the thread-specific cache, you are doing it wrong.  But even if you are trying to do that, the jemalloc 5.3.0 will continue to work just fine.

Thanks to Ondřej Surý for the explanation! Closing the bug as the Elevate project offers an upgrade path to EL 10, which has 5.3.0.