Bug 1545539
Summary: | jemalloc(aarch64): munmap - Invalid Argument - build-time pagesize 4k != Fedora kernel (64k) | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Daniel Black <daniel> |
Component: | jemalloc | Assignee: | Ingvar Hagelund <ingvar> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 28 | CC: | daniel, ingvar, lmohanty, mschorm |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | aarch64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | jemalloc-4.5.0-5.fc26 jemalloc-4.5.0-5.fc27 jemalloc-5.0.1-5.fc28 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-03-20 17:31:36 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1548039 |
Description
Daniel Black
2018-02-15 09:16:28 UTC
note: please also fix the --with-lg-page=16 for ppc64, ppc64le as its kernel page size can be 4k or 64k too. It just happens that Fedora's ppc64 builders are on 64k currently. Probably a good idea to make build infrastructure use distro kernels too. Blocks MariaDB 10.1.31 update on F26. For testing please use: https://github.com/devexp-db/mariadb/tree/10.1.31 For questions about mariadb contact me whenever :) This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle. Changing version to '28'. Looks good to me. Seems to solve https://github.com/varnishcache/varnish-cache/issues/2521 as well :-) I'll push builds for fc28 tonight. Ingvar That's great! :) Please, take a look at F27 & F26 too. fedora 26,27 does not have jemalloc-5.0. They are still on 4.4.0 and 4.5.0. To be able to support the newer MariaDB on these fedora releases, that means I'll have to upgrade jemalloc to a new major release after a Fedora stable release. I'm not sure if I should do that. At least, I would have to make all maintainers check that their packages work as expected with jemalloc-5.0 I can fix jemalloc itself and varnish. You may check mariadb. For the rest, that is blender, bro, neovim, and redis, I'm unsure on how to proceed. f27$ dnf repoquery --whatrequires jemalloc blender-1:2.79-1.fc27.x86_64 blenderplayer-1:2.79-1.fc27.x86_64 bro-0:2.4.1-3.fc25.x86_64 jemalloc-devel-0:4.5.0-4.fc27.i686 jemalloc-devel-0:4.5.0-4.fc27.x86_64 mariadb-server-3:10.2.9-3.fc27.x86_64 neovim-0:0.2.0-1.fc27.x86_64 neovim-0:0.2.2-1.fc27.x86_64 redis-0:3.2.11-1.fc27.x86_64 redis-0:4.0.8-1.fc27.x86_64 varnish-0:5.1.3-2.fc27.x86_64 varnish-0:5.1.3-4.fc27.x86_64 Ingvar (In reply to Ingvar Hagelund from comment #7) > fedora 26,27 does not have jemalloc-5.0. They are still on 4.4.0 and 4.5.0. > To be able to support the newer MariaDB on these fedora releases, that means > I'll have to upgrade jemalloc to a new major release after a Fedora stable > release. I'm not sure if I should do that. No, please don't! MariaDB (well, TokuDB storage engine to be exact, originating from the Percona upstream) does not support Jemmaloc 5. That's another issue I'm solving on mentioned upstreams. -- For now, I want 2 things: 1) Fix F26 Jemalloc - there's mariadb 10.1, which has problems if the jemalloc is not compiled with "-with-lg-page=16" 2) Fix the Jemalloc on all other Fedoras - because the issue is IMHO general and beacuse the Fedora use 64k kernel page size, you should compile Jemalloc with "-with-lg-page=16" to work correctly *with any software* on all Fedora versions I'm not sure that would work across different pagesizes, which is the main culprit here. https://github.com/jemalloc/jemalloc/pull/769/commits/e4827d5c16f1411729c6d426a4d5792a8d8e88aa If I read that correctly, supporting configuring a pagesize larger than the actual running system's pagesize, is a feature that was added to jemalloc-5.0. Ingvar ... and I don't think Fedora uses 64k kernel pagesize on all arches. For instance, ppc/64/64le have had special pagesizes for years. > If I read that correctly, supporting configuring a pagesize larger than the > actual running system's pagesize, is a feature that was added to jemalloc-5.0. Correct. > ... and I don't think Fedora uses 64k kernel pagesize on all arches. For instance, ppc/64/64le have had special pagesizes for years. Correct. I think setting --with-lg-page=16 on arches with 64k page sizes default page sizes is the right solution which is a compile time hardening against a 4k page size kernel running. Having build infrastructure running with a default kernel is also recommended to ensure the tests of packages run when distributed (which is why ppc64{,le} doesn't currently hit this bug). While setting it to 4k explicitly on dual pages size arches like ppc64/ppc64le/aarch64 will actually work whether the kernel page size is 4k or 64k, this does incur an added performance overhead in memory management which is what jemalloc intends to solve. There is rarely a reason to lower the kernel page size on these kernels. I've seen it once as a temporary solution for code, like jemalloc, that by assumption or design, failed to retrieve the pagesize at runtime (sysconf(_SC_PAGESIZE)). I don't know if a pre-install check to see if the running size matches the configure size is something you wish to consider. As the default compile time options seem to work on the ppc arches (and s390x too), if I should change this, it should be on the arches that have this problem. Also, if --with-lg-page is not set, jemalloc should check that page size matches at build time, and this have not been an actual problem for a long time. Michal, I therefore wonder if the problem you observe is bound to an external library which handles this incorrectly, like my problem with pcre with jit in varnish. If that is the case, hardcoding the pagesize for jemalloc-4.x to 4k on x86_64, and to 64k on ppc64 et al, probably won't help, though you might be lucky Just for curiousness, please try to build jemalloc-4.5.0 with --with-lg-page=12 and see if that helps on fedora 26 / x86_64. Ingvar (In reply to Daniel Black from comment #11) > While setting it to 4k explicitly on dual pages size arches like > ppc64/ppc64le/aarch64 will actually work whether the kernel page size is 4k > or 64k, this does incur an added performance overhead in memory management > which is what jemalloc intends to solve. Daniel, As just setting 64k everywhere, and let jemalloc handle the rest itself, seem to work, as in the mentioned added support in jemalloc-5.0, I'll do that in fedora 28+. Does that sound allright? Then I need a fix for fedora 26 and 27: I could build jemalloc with pagesizes according to a fixed scheme, instead of relying on the builder's pagesize, using simple %ifarch. Then what are the correct pagesizes? Is this table correct? i686: 4k x86_64: 4k armv7hl: 4k s390x: 4k aarch64: 64k ppc64: 64k ppc64le: 64k Ingvar jemalloc won't build on RedHat's aarch64 builders, as they run 4k pagesize kernels, so for now: %ifarch %ix86 %arm x86_64 s390x aarch64 %define lg_page --with-lg-page=12 %endif %ifarch ppc64 ppc64le %define lg_page --with-lg-page=16 %endif Full f26 specfile: https://ingvar.fedorapeople.org/jemalloc/jemalloc.spec Builds: https://koji.fedoraproject.org/koji/taskinfo?taskID=25543005 Please try these builds for fedora 26, and report back. Ingvar Seems I brain farted with the below comment.
> While setting it to 4k explicitly on dual pages size arches like ppc64/ppc64le/aarch64 will actually work whether the kernel page size is 4k or 64k,
The larger jemalloc page size 64k, will work on kernel page size 4k and 64k.
As such aarch64 should be with the ppc64 and ppc64le in the spec above at --with-lg-page=16.
I think the same arch dependent configure should apply to jemalloc-5.0.
Michal's build log (no missing) was mariadb using jemalloc directly. The varnish build/test log didn't show munmap syscalls failing so it looks like a different problem.
I'm slight confused by the thp patch that seems to be applied in some cases but THP tests still run successfully. I'm hoping the build kernel/OS configuration there matches the release kernels in this respect.
(In reply to Daniel Black from comment #15) > Seems I brain farted with the below comment. > > > While setting it to 4k explicitly on dual pages size arches like ppc64/ppc64le/aarch64 will actually work whether the kernel page size is 4k or 64k, > > The larger jemalloc page size 64k, will work on kernel page size 4k and 64k. > > As such aarch64 should be with the ppc64 and ppc64le in the spec above at > --with-lg-page=16. With jemalloc-5.0, yes. With jemalloc-4.x, no. For all I know, it way work at runtime, but it won't build. jemalloc-4.x does some sanity checks at build time, and won't play along with 64k pagesize on a system running with 4k pagesize. The fedora aarch64 builders have 4k pagesize. So for now, I must go with 4k pagesize for jemalloc-4.x on aarch64. Btw, This is exactly what https://github.com/jemalloc/jemalloc/pull/769/commits/e4827d5c16f1411729c6d426a4d5792a8d8e88aa is addressing, so this problem is fixed in jemalloc-5.0. > I think the same arch dependent configure should apply to jemalloc-5.0. Okay, I'll go with that on jemalloc-5.0, even with 64k on aarch64. Ingvar jemalloc-4.5.0-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-c0dcbdd82f jemalloc-5.0.1-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-0e404615ab jemalloc-4.5.0-5.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-62ddf35772 jemalloc-4.5.0-5.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-c0dcbdd82f jemalloc-4.5.0-5.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-62ddf35772 jemalloc-5.0.1-5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-0e404615ab jemalloc-4.5.0-5.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report. jemalloc-4.5.0-5.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report. jemalloc-5.0.1-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |