Hide Forgot
Description of problem: Hello, Red Hat. I am Ingvar Hagelund, the maintainer of varnish in Fedora. First: Kudos to Red Hat for including varnish in the rhel8 beta. I've been waiting for this. I observe that you have chosen the varnish-6.0 release for rhel8. That is a wise move, as it is a tagged an LTS release of varnish by the project. Please consider moving to the latest varnish-6.0.2 release (or later) before finalizing rhel8. Now the problem: Varnish in the rhel8 beta is compiled without jemalloc. It seems that jemalloc is not available in the rhel8 beta at all. Version-Release number of selected component (if applicable): varnish-6.0.0-3.el8+1489+ed3c21d0.x86_64 How reproducible: Always Steps to Reproduce: 1. Install the rhel8 beta 2. Install varnish from the DVD 3. rpm -q --requires varnish | grep jemalloc 4. ldd /usr/sbin/varnishd | grep jemalloc Actual results: No output. Expected results: Something like jemalloc libjemalloc.so.2()(64bit) libjemalloc.so.2 => /lib64/libjemalloc.so.2 (0x00007f26f8ab1000) Additional info: jemalloc is not even included in the distribution. I had a word with the core developer of varnish, Poul Henning Kamp. To rephrase his reaction without any stronger words, he thinks it is, well, unwise, to use varnish without jemalloc. varnish will get lower performance and throughput without jemalloc. <rant> Originally, jemalloc was not cut into released versions. Developers just included a checkout in their projects. A jemalloc checkout was included within the varnish code, and was updated only when the developers felt like it. Because of the Fedora Packaging guidelines ( https://docs.fedoraproject.org/en-US/packaging-guidelines/#_bundling_and_duplication_of_system_libraries ) I first talked core developer of jemalloc Jason Evans into making jemalloc a "real" project with email lists and real releases, in favor of simple unversioned checkouts. Then I wrapped a package from the released jemalloc in Fedora and EPEL. Then I talked core developer of varnish Poul-Henning Kamp into skipping the included jemalloc, and rather depend on my package. Finally, I got the varnish maintainer in Debian to duplicate my efforts, to get jemalloc into Debian (and thus Ubuntu). Later, the package also appeared in OpenSuSE. So the question is then, why not include jemalloc in rhel8. There are other applications that could use jemalloc besides varnish, and it is well tested, and should work on all supported arches. It has also a well proven track record, being used in projects like FreeBSD (as the default malloc implementation) and Mozilla Firefox besides varnish. Adding an alternative malloc implementation may add complexity and maintenance to the distribution, but as the Fedora maintainer myself, I find the cost low. jemalloc is Free Software under a BSD License, it is developed as Open Source, it has a live and attentive set of maintainers, and the main architect Jason Evans is reasonable, responsive and supportive. With him having no prior redhat/mock knowledge, I once got him to debug jemalloc inside an ubuntu hosted centos6 mock chroot, with me supporting him over email. RHEL has always set the standard for what should and should not be included in an enterprise class distribution. I don't know the rationale for not including jemalloc. It might be complexity, security, policy, or just anything else at your end. but I'd rather not see my efforts wasted because of some simple compilation bug in varnish. </rant> There /is/ a known problem with jemalloc and varnish related to pcre jit in the later releases in the varnish-6.0.x series, but it has been identified as a simple overflow caused by a buffer mismatch between pcre and varnish, and it will be fixed by the developers. One valid workaround for now, is simply building varnish without pcre-jit. This is a better idea than just skipping jemalloc. Another variant may be to just downgrade jemalloc to a 3.x or 4.x version, that works well with varnish. That was my two cents Best regards, Ingvar Hagelund Fun fact: Poul-Henning wrote the former implementation of malloc in FreeBSD. Jason wrote the current one.
Hi Ingvar - thanks a lot for your feedback, and also for your ongoing work to maintain the varnish package in Fedora - I appreciate both! 1) Yes, we are looking at updating to 6.0.2. 2) We made the decision to drop jemalloc support in conjunction with the Red Hat toolchain team, so we can concentrate resources on supporting a single malloc implementation across the distribution. Are you aware of specific varnish benchmarks for which jemalloc outperforms glibc malloc? We're interested in doing more performance testing here.
Hello, Joe. This kind of benchmarking has been done before. Here is one done by Reza Naghibi in Varnish Software for varnish-4.1.1 https://filebin.varnish-software.com/isjzpg3tj1npiy46 basically, look at the spread between the blue and red line. Ingvar
Created attachment 1512177 [details] Varnish memory usage
Created attachment 1512178 [details] Varnish memory usage (pdf)
Joe; How did you solve this with other rhel8 packages internally/static or shared linked to jemalloc, like mariadb, redis, and firefox? Ingvar
Also, while rolling varnish-6.0 downstream for rhel8, to avoid too much ducplicated work, consider my copr repos for el6 and el7, where I follow both fedora (varnish-6.1 in rawhide) and varnish-6.0 (as well as all other later releases of Varnish Cache). https://copr.fedorainfracloud.org/coprs/ingvar/ Ingvar
[ingvar@rhel8 ~]$ rpm -q firefox firefox-60.2.1-1.el8.x86_64 [ingvar@rhel8 ~]$ strings /usr/lib64/firefox/firefox-bin | grep jemalloc_ jemalloc_stats jemalloc_purge_freed_pages jemalloc_free_dirty_pages jemalloc_ptr_info jemalloc_thread_local_arena [ingvar@rhel8 ~]$ rpm -q redis redis-4.0.10-2.el8+1553+45154855.x86_64 [ingvar@rhel8 ~]$ strings /usr/bin/redis-server | grep jemalloc_ je_jemalloc_prefork je_jemalloc_postfork_child je_jemalloc_postfork_parent So instead of the burden of maintaining one shared instance of jemalloc ... you maintain at least two different inline versions. I am sorry, I do not see the rationale here. Upstream clearly advocates the use of jemalloc with Varnish. If Red Hat really does not want to support a shared library, I would advice including jemalloc inline in varnish (again), like you do with redis and firefox. While tossing the Fedora policy out, at least Varnish may run at full speed. I agree that updated benchmarks would be great. Ingvar
Relevant: #1917697
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.