Bug 1656034 - Varnish is compiled without jemalloc. jemalloc is missing.
Summary: Varnish is compiled without jemalloc. jemalloc is missing.
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: varnish
Version: 8.0
Hardware: x86_64
OS: Linux
Target Milestone: rc
: 8.0
Assignee: Luboš Uhliarik
QA Contact: rhel-cs-infra-services-qe
Depends On:
TreeView+ depends on / blocked
Reported: 2018-12-04 13:58 UTC by Ingvar Hagelund
Modified: 2021-02-01 07:30 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-02-01 07:30:53 UTC
Type: Bug
Target Upstream Version:

Attachments (Terms of Use)
Varnish memory usage (1.39 MB, application/zip)
2018-12-06 16:09 UTC, Ingvar Hagelund
no flags Details
Varnish memory usage (pdf) (239.34 KB, application/pdf)
2018-12-06 16:10 UTC, Ingvar Hagelund
no flags Details

Description Ingvar Hagelund 2018-12-04 13:58:17 UTC
Description of problem:

Hello, Red Hat. I am Ingvar Hagelund, the maintainer of varnish in Fedora.

First: Kudos to Red Hat for including varnish in the rhel8 beta. I've been waiting for this. I observe that you have chosen the varnish-6.0 release for rhel8. That is a wise move, as it is a tagged an LTS release of varnish by the project. Please consider moving to the latest varnish-6.0.2 release (or later) before finalizing rhel8.

Now the problem: Varnish in the rhel8 beta is compiled without jemalloc. It seems that jemalloc is not available in the rhel8 beta at all.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Install the rhel8 beta
2. Install varnish from the DVD
3. rpm -q --requires varnish | grep jemalloc
4. ldd /usr/sbin/varnishd | grep jemalloc

Actual results:
No output.

Expected results:
Something like
libjemalloc.so.2 => /lib64/libjemalloc.so.2 (0x00007f26f8ab1000)

Additional info:

jemalloc is not even included in the distribution. I had a word with the core developer of varnish, Poul Henning Kamp. To rephrase his reaction without any stronger words, he thinks it is, well, unwise, to use varnish without jemalloc. varnish will get lower performance and throughput without jemalloc.

Originally, jemalloc was not cut into released versions. Developers just included a checkout in their projects. A jemalloc checkout was included within the varnish code, and was updated only when the developers felt like it. Because of the Fedora Packaging guidelines ( https://docs.fedoraproject.org/en-US/packaging-guidelines/#_bundling_and_duplication_of_system_libraries ) I first talked core developer of jemalloc Jason Evans into making jemalloc a "real" project with email lists and real releases, in favor of simple unversioned checkouts. Then I wrapped a package from the released jemalloc in Fedora and EPEL. Then I talked core developer of varnish Poul-Henning Kamp into skipping the included jemalloc, and rather depend on my package. Finally, I got the varnish maintainer in Debian to duplicate my efforts, to get jemalloc into Debian (and thus Ubuntu). Later, the package also appeared in OpenSuSE.

So the question is then, why not include jemalloc in rhel8. There are other applications that could use jemalloc besides varnish, and it is well tested, and should work on all supported arches. It has also a well proven track record, being used in projects like FreeBSD (as the default malloc implementation) and Mozilla Firefox besides varnish. Adding an alternative malloc implementation may add complexity and maintenance to the distribution, but as the Fedora maintainer myself, I find the cost low.

jemalloc is Free Software under a BSD License, it is developed as Open Source, it has a live and attentive set of maintainers, and the main architect Jason Evans is reasonable, responsive and supportive. With him having no prior redhat/mock knowledge, I once got him to debug jemalloc inside an ubuntu hosted centos6 mock chroot, with me supporting him over email.

RHEL has always set the standard for what should and should not be included in an enterprise class distribution. I don't know the rationale for not including jemalloc. It might be complexity, security, policy, or just anything else at your end. but I'd rather not see my efforts wasted because of some simple compilation bug in varnish.

There /is/ a known problem with jemalloc and varnish related to pcre jit in the later releases in the varnish-6.0.x series, but it has been identified as a simple overflow caused by a buffer mismatch between pcre and varnish, and it will be fixed by the developers. One valid workaround for now, is simply building varnish without pcre-jit. This is a better idea than just skipping jemalloc. Another variant may be to just downgrade jemalloc to a 3.x or 4.x version, that works well with varnish.

That was my two cents

Best regards,
Ingvar Hagelund

Fun fact: Poul-Henning wrote the former implementation of malloc in FreeBSD. Jason wrote the current one.

Comment 1 Joe Orton 2018-12-04 16:56:28 UTC
Hi Ingvar - thanks a lot for your feedback, and also for your ongoing work to maintain the varnish package in Fedora - I appreciate both!

1) Yes, we are looking at updating to 6.0.2.

2) We made the decision to drop jemalloc support in conjunction with the Red Hat toolchain team, so we can concentrate resources on supporting a single malloc implementation across the distribution.  Are you aware of specific varnish benchmarks for which jemalloc outperforms glibc malloc?  We're interested in doing more performance testing here.

Comment 2 Ingvar Hagelund 2018-12-06 15:49:28 UTC
Hello, Joe.

This kind of benchmarking has been done before. Here is one done by Reza Naghibi in Varnish Software for varnish-4.1.1


basically, look at the spread between the blue and red line.


Comment 3 Ingvar Hagelund 2018-12-06 16:09:48 UTC
Created attachment 1512177 [details]
Varnish memory usage

Comment 4 Ingvar Hagelund 2018-12-06 16:10:16 UTC
Created attachment 1512178 [details]
Varnish memory usage (pdf)

Comment 5 Ingvar Hagelund 2018-12-06 16:19:28 UTC

How did you solve this with other rhel8 packages internally/static or shared linked to jemalloc, like mariadb, redis, and firefox?


Comment 6 Ingvar Hagelund 2018-12-07 10:45:25 UTC
Also, while rolling varnish-6.0 downstream for rhel8, to avoid too much ducplicated work, consider my copr repos for el6 and el7, where I follow both fedora (varnish-6.1 in rawhide) and varnish-6.0 (as well as all other later releases of Varnish Cache). 



Comment 7 Ingvar Hagelund 2018-12-11 11:40:16 UTC
[ingvar@rhel8 ~]$ rpm -q firefox 

[ingvar@rhel8 ~]$ strings /usr/lib64/firefox/firefox-bin | grep jemalloc_

[ingvar@rhel8 ~]$ rpm -q redis

[ingvar@rhel8 ~]$ strings /usr/bin/redis-server | grep jemalloc_

So instead of the burden of maintaining one shared instance of jemalloc ... you maintain at least two different inline versions. I am sorry, I do not see the rationale here.

Upstream clearly advocates the use of jemalloc with Varnish. If Red Hat really does not want to support a shared library, I would advice including jemalloc inline in varnish (again), like you do with redis and firefox. While tossing the Fedora policy out, at least Varnish may run at full speed.

I agree that updated benchmarks would be great.


Comment 17 Ingvar Hagelund 2021-01-20 15:09:00 UTC
Relevant: #1917697

Comment 18 RHEL Program Management 2021-02-01 07:30:53 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Note You need to log in before you can comment on or make changes to this bug.