Description of problem: The documentation for LMDB [1] states that the map size should be "chosen as large as possible". Consequently, the zarr developers have chosen a map size of 2**40 on 64-bit systems and 2**28 on 32-bit systems [2]. When the package is built on a 64-bit system [3], everything works fine. However, if the build is run on a 32-bit system [4], it fails with a MemoryError. I have narrowed this down to the mmap call at [5] which attempts to map the backing file into memory. AFAICT, the mapped size is far below the RAM on the build machine as well as far below the normal 32-bit VM limit. So I don't know why the call is failing. I can reproduce this in mock with the fedora-rawhide-i386 config, but what's weird is it also fails in the fedora-rawhide-s390x config. This machine has 16G of RAM, so it's enough for 2**28, but not 2**40. So it fails on 32-bit x86 with more than enough RAM, works on 64-bit x86 with nowhere near enough RAM, yet fails with 64-bit s390x. Version-Release number of selected component (if applicable): 0.9.23-2.fc30 Steps to Reproduce: 1. mock -r fedora-rawhide-x86_64 ./python-zarr-2.3.1-2.fc31.src.rpm (works) 2. mock -r fedora-rawhide-i386 ./python-zarr-2.3.1-2.fc31.src.rpm (fails) 3. mock -r fedora-rawhide-s390x --forcearch=s390x ./python-zarr-2.3.1-2.fc31.src.rpm (fails) Just to keep zarr out of it though, you can also do something like: $ mock -r fedora-rawhide-s390x --forcearch=s390x --install python3-lmdb $ mock -r fedora-rawhide-s390x --forcearch=s390x --shell # python3 >>> import lmdb >>> lmdb.open("test.db", map_size=2**40) Actual results: MemoryError in 4-6 tests Expected results: Tests all pass. Additional info: [1] http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5 [2] https://github.com/zarr-developers/zarr/blob/f6ced1e31b919065f8834d813ec081d2a85195b3/zarr/storage.py#L1585-L1587 [3] https://koji.fedoraproject.org/koji/taskinfo?taskID=33910305 [4] https://koji.fedoraproject.org/koji/taskinfo?taskID=33881973 [5] http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=libraries/liblmdb/mdb.c;h=e12af4482a0172da8d759c1da1530339d9095510;hb=2a5eaad6919ce6941dec4f0d5cce370707a00ba7#l4019
I have trouble setting up the s390x mock, so I have tested with i686 mock, and I cannot reproduce it so far: $ mock -r fedora-rawhide-i386 --forcearch=i686 --shell # python3 >>> import lmdb >>> lmdb.open("/tmp/test.db", map_size=2**40) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: Integer argument exceeds limit. # expected >>> lmdb.open("/tmp/test.db", map_size=2**28) <Environment object at 0xf7631448> # no error >>> _.close() As a side note, this seems to be rather low-level issue, and I do not have much of an idea on what might be wrong. I suggest asking the authors of lmdb directly (http://www.openldap.org/lists/) as they might have a better understanding of the issue. Alternatively, try the devel.org to reach a wider audience – perhaps someone will have an idea.
I don't see a problem when building in rawhide on s390x, within a 4GB VM ... Wrote: /home/sharkcz/python-zarr/python-zarr-2.3.2-1.fc31.src.rpm
Thanks to Tom's message on the mailing list, I figured out that tests don't explicitly close backing stores leaving it to gc. This leaves several 2**28 mmaps around so that it eventually runs out of virtual memory space. I added a patch to close things and it now builds on all arches in koji. This still fails in qemu'd mock: $ mock -r fedora-rawhide-s390x --forcearch=s390x --install python3-lmdb $ mock -r fedora-rawhide-s390x --forcearch=s390x --shell # python3 >>> import lmdb >>> lmdb.open("test.db", map_size=2**40) but I'm less concerned about that now that I can get koji working properly. I don't know if it's a qemu issue or elsewhere, but if you wish to close, that is fine by me (though figuring out the qemu mock issue might be nice.)
I suppose this is a qemu emulation issue.
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle. Changing version to '31'.
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle. Changing version to 31.
Re-assigning to qemu, as noted in comment 4.
wonder if it could be related to bug #1741935, but sure it makes sense to retry with latest qemu
This message is a reminder that Fedora 31 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '31'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 31 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.