RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1656034 - Varnish is compiled without jemalloc. jemalloc is missing.
Summary: Varnish is compiled without jemalloc. jemalloc is missing.
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: varnish
Version: 8.0
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Luboš Uhliarik
QA Contact: rhel-cs-infra-services-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-04 13:58 UTC by Ingvar Hagelund
Modified: 2021-02-01 07:30 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-01 07:30:53 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Varnish memory usage (1.39 MB, application/zip)
2018-12-06 16:09 UTC, Ingvar Hagelund
no flags Details
Varnish memory usage (pdf) (239.34 KB, application/pdf)
2018-12-06 16:10 UTC, Ingvar Hagelund
no flags Details

Description Ingvar Hagelund 2018-12-04 13:58:17 UTC
Description of problem:

Hello, Red Hat. I am Ingvar Hagelund, the maintainer of varnish in Fedora.

First: Kudos to Red Hat for including varnish in the rhel8 beta. I've been waiting for this. I observe that you have chosen the varnish-6.0 release for rhel8. That is a wise move, as it is a tagged an LTS release of varnish by the project. Please consider moving to the latest varnish-6.0.2 release (or later) before finalizing rhel8.

Now the problem: Varnish in the rhel8 beta is compiled without jemalloc. It seems that jemalloc is not available in the rhel8 beta at all.

Version-Release number of selected component (if applicable):
varnish-6.0.0-3.el8+1489+ed3c21d0.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install the rhel8 beta
2. Install varnish from the DVD
3. rpm -q --requires varnish | grep jemalloc
4. ldd /usr/sbin/varnishd | grep jemalloc

Actual results:
No output.

Expected results:
Something like
jemalloc
libjemalloc.so.2()(64bit)
libjemalloc.so.2 => /lib64/libjemalloc.so.2 (0x00007f26f8ab1000)


Additional info:

jemalloc is not even included in the distribution. I had a word with the core developer of varnish, Poul Henning Kamp. To rephrase his reaction without any stronger words, he thinks it is, well, unwise, to use varnish without jemalloc. varnish will get lower performance and throughput without jemalloc.

<rant>
Originally, jemalloc was not cut into released versions. Developers just included a checkout in their projects. A jemalloc checkout was included within the varnish code, and was updated only when the developers felt like it. Because of the Fedora Packaging guidelines ( https://docs.fedoraproject.org/en-US/packaging-guidelines/#_bundling_and_duplication_of_system_libraries ) I first talked core developer of jemalloc Jason Evans into making jemalloc a "real" project with email lists and real releases, in favor of simple unversioned checkouts. Then I wrapped a package from the released jemalloc in Fedora and EPEL. Then I talked core developer of varnish Poul-Henning Kamp into skipping the included jemalloc, and rather depend on my package. Finally, I got the varnish maintainer in Debian to duplicate my efforts, to get jemalloc into Debian (and thus Ubuntu). Later, the package also appeared in OpenSuSE.

So the question is then, why not include jemalloc in rhel8. There are other applications that could use jemalloc besides varnish, and it is well tested, and should work on all supported arches. It has also a well proven track record, being used in projects like FreeBSD (as the default malloc implementation) and Mozilla Firefox besides varnish. Adding an alternative malloc implementation may add complexity and maintenance to the distribution, but as the Fedora maintainer myself, I find the cost low.

jemalloc is Free Software under a BSD License, it is developed as Open Source, it has a live and attentive set of maintainers, and the main architect Jason Evans is reasonable, responsive and supportive. With him having no prior redhat/mock knowledge, I once got him to debug jemalloc inside an ubuntu hosted centos6 mock chroot, with me supporting him over email.

RHEL has always set the standard for what should and should not be included in an enterprise class distribution. I don't know the rationale for not including jemalloc. It might be complexity, security, policy, or just anything else at your end. but I'd rather not see my efforts wasted because of some simple compilation bug in varnish.
</rant>

There /is/ a known problem with jemalloc and varnish related to pcre jit in the later releases in the varnish-6.0.x series, but it has been identified as a simple overflow caused by a buffer mismatch between pcre and varnish, and it will be fixed by the developers. One valid workaround for now, is simply building varnish without pcre-jit. This is a better idea than just skipping jemalloc. Another variant may be to just downgrade jemalloc to a 3.x or 4.x version, that works well with varnish.

That was my two cents

Best regards,
Ingvar Hagelund


Fun fact: Poul-Henning wrote the former implementation of malloc in FreeBSD. Jason wrote the current one.

Comment 1 Joe Orton 2018-12-04 16:56:28 UTC
Hi Ingvar - thanks a lot for your feedback, and also for your ongoing work to maintain the varnish package in Fedora - I appreciate both!

1) Yes, we are looking at updating to 6.0.2.

2) We made the decision to drop jemalloc support in conjunction with the Red Hat toolchain team, so we can concentrate resources on supporting a single malloc implementation across the distribution.  Are you aware of specific varnish benchmarks for which jemalloc outperforms glibc malloc?  We're interested in doing more performance testing here.

Comment 2 Ingvar Hagelund 2018-12-06 15:49:28 UTC
Hello, Joe.

This kind of benchmarking has been done before. Here is one done by Reza Naghibi in Varnish Software for varnish-4.1.1

https://filebin.varnish-software.com/isjzpg3tj1npiy46

basically, look at the spread between the blue and red line.

Ingvar

Comment 3 Ingvar Hagelund 2018-12-06 16:09:48 UTC
Created attachment 1512177 [details]
Varnish memory usage

Comment 4 Ingvar Hagelund 2018-12-06 16:10:16 UTC
Created attachment 1512178 [details]
Varnish memory usage (pdf)

Comment 5 Ingvar Hagelund 2018-12-06 16:19:28 UTC
Joe;

How did you solve this with other rhel8 packages internally/static or shared linked to jemalloc, like mariadb, redis, and firefox?

Ingvar

Comment 6 Ingvar Hagelund 2018-12-07 10:45:25 UTC
Also, while rolling varnish-6.0 downstream for rhel8, to avoid too much ducplicated work, consider my copr repos for el6 and el7, where I follow both fedora (varnish-6.1 in rawhide) and varnish-6.0 (as well as all other later releases of Varnish Cache). 

https://copr.fedorainfracloud.org/coprs/ingvar/

Ingvar

Comment 7 Ingvar Hagelund 2018-12-11 11:40:16 UTC
[ingvar@rhel8 ~]$ rpm -q firefox 
firefox-60.2.1-1.el8.x86_64

[ingvar@rhel8 ~]$ strings /usr/lib64/firefox/firefox-bin | grep jemalloc_
jemalloc_stats
jemalloc_purge_freed_pages
jemalloc_free_dirty_pages
jemalloc_ptr_info
jemalloc_thread_local_arena


[ingvar@rhel8 ~]$ rpm -q redis
redis-4.0.10-2.el8+1553+45154855.x86_64

[ingvar@rhel8 ~]$ strings /usr/bin/redis-server | grep jemalloc_
je_jemalloc_prefork
je_jemalloc_postfork_child
je_jemalloc_postfork_parent


So instead of the burden of maintaining one shared instance of jemalloc ... you maintain at least two different inline versions. I am sorry, I do not see the rationale here.

Upstream clearly advocates the use of jemalloc with Varnish. If Red Hat really does not want to support a shared library, I would advice including jemalloc inline in varnish (again), like you do with redis and firefox. While tossing the Fedora policy out, at least Varnish may run at full speed.

I agree that updated benchmarks would be great.

Ingvar

Comment 17 Ingvar Hagelund 2021-01-20 15:09:00 UTC
Relevant: #1917697

Comment 18 RHEL Program Management 2021-02-01 07:30:53 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.


Note You need to log in before you can comment on or make changes to this bug.