Bug 1656034

Description of problem:

Hello, Red Hat. I am Ingvar Hagelund, the maintainer of varnish in Fedora.

First: Kudos to Red Hat for including varnish in the rhel8 beta. I've been waiting for this. I observe that you have chosen the varnish-6.0 release for rhel8. That is a wise move, as it is a tagged an LTS release of varnish by the project. Please consider moving to the latest varnish-6.0.2 release (or later) before finalizing rhel8.

Now the problem: Varnish in the rhel8 beta is compiled without jemalloc. It seems that jemalloc is not available in the rhel8 beta at all.

Version-Release number of selected component (if applicable):
varnish-6.0.0-3.el8+1489+ed3c21d0.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install the rhel8 beta
2. Install varnish from the DVD
3. rpm -q --requires varnish | grep jemalloc
4. ldd /usr/sbin/varnishd | grep jemalloc

Actual results:
No output.

Expected results:
Something like
jemalloc
libjemalloc.so.2()(64bit)
libjemalloc.so.2 => /lib64/libjemalloc.so.2 (0x00007f26f8ab1000)


Additional info:

jemalloc is not even included in the distribution. I had a word with the core developer of varnish, Poul Henning Kamp. To rephrase his reaction without any stronger words, he thinks it is, well, unwise, to use varnish without jemalloc. varnish will get lower performance and throughput without jemalloc.

<rant>
Originally, jemalloc was not cut into released versions. Developers just included a checkout in their projects. A jemalloc checkout was included within the varnish code, and was updated only when the developers felt like it. Because of the Fedora Packaging guidelines ( https://docs.fedoraproject.org/en-US/packaging-guidelines/#_bundling_and_duplication_of_system_libraries ) I first talked core developer of jemalloc Jason Evans into making jemalloc a "real" project with email lists and real releases, in favor of simple unversioned checkouts. Then I wrapped a package from the released jemalloc in Fedora and EPEL. Then I talked core developer of varnish Poul-Henning Kamp into skipping the included jemalloc, and rather depend on my package. Finally, I got the varnish maintainer in Debian to duplicate my efforts, to get jemalloc into Debian (and thus Ubuntu). Later, the package also appeared in OpenSuSE.

So the question is then, why not include jemalloc in rhel8. There are other applications that could use jemalloc besides varnish, and it is well tested, and should work on all supported arches. It has also a well proven track record, being used in projects like FreeBSD (as the default malloc implementation) and Mozilla Firefox besides varnish. Adding an alternative malloc implementation may add complexity and maintenance to the distribution, but as the Fedora maintainer myself, I find the cost low.

jemalloc is Free Software under a BSD License, it is developed as Open Source, it has a live and attentive set of maintainers, and the main architect Jason Evans is reasonable, responsive and supportive. With him having no prior redhat/mock knowledge, I once got him to debug jemalloc inside an ubuntu hosted centos6 mock chroot, with me supporting him over email.

RHEL has always set the standard for what should and should not be included in an enterprise class distribution. I don't know the rationale for not including jemalloc. It might be complexity, security, policy, or just anything else at your end. but I'd rather not see my efforts wasted because of some simple compilation bug in varnish.
</rant>

There /is/ a known problem with jemalloc and varnish related to pcre jit in the later releases in the varnish-6.0.x series, but it has been identified as a simple overflow caused by a buffer mismatch between pcre and varnish, and it will be fixed by the developers. One valid workaround for now, is simply building varnish without pcre-jit. This is a better idea than just skipping jemalloc. Another variant may be to just downgrade jemalloc to a 3.x or 4.x version, that works well with varnish.

That was my two cents

Best regards,
Ingvar Hagelund


Fun fact: Poul-Henning wrote the former implementation of malloc in FreeBSD. Jason wrote the current one.