Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
This also affects RHELAH 7.4.1 with docker 1.12.6
# sh repro.sh
+ dd if=/dev/urandom of=myfile count=500000
500000+0 records in
500000+0 records out
256000000 bytes (256 MB) copied, 1.87501 s, 137 MB/s
+ docker run --detach --name cnt registry.fedoraproject.org/fedora:26 sleep infinity
Unable to find image 'registry.fedoraproject.org/fedora:26' locally
Trying to pull repository registry.fedoraproject.org/fedora ...
26: Pulling from registry.fedoraproject.org/fedora
727707040437: Pull complete
Digest: sha256:311f2acf4b42a6c1c00cbffe902da9d34a684f127e39cd6b0558a18785390c3f
cf0583c0c58fa31b04558a957e46273c0304bac53312dd2367e3c80a3cd4af2b
++ seq 5
+ for i in '$(seq 5)'
+ docker cp myfile cnt:/var/tmp
+ docker cp cnt:/var/tmp/myfile .
+ for i in '$(seq 5)'
+ docker cp myfile cnt:/var/tmp
Error response from daemon: Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory
# rpm -q docker
docker-1.12.6-55.gitc4618fb.el7.x86_64
# rpm-ostree status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.4.1 (2017-08-30 19:29:56)
Commit: e83c16780259c5272684221e2a6007300d94bbfdc5432f9ab6025300f447145b
+++ This bug was initially created as a clone of Bug #1489505 +++
Description of problem:
On the latest Fedora 26 AH, it seems really easy to get the docker daemon to exhaust all the system memory by repeated `docker cp` commands until it fails with:
Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory
This makes it impractical in testing environments where we need to routinely spin up and provision throw-away containers.
Version-Release number of selected component (if applicable):
[root@jlebon ~]# rpm-ostree status
State: idle
Deployments:
* fedora-atomic:fedora/26/x86_64/atomic-host
Version: 26.101 (2017-08-06 21:27:14)
Commit: f6331bcd14577e0ee43db3ba5a44e0f63f74a86e3955604c20542df0b7ad8ad6
[root@jlebon ~]# rpm -q docker
docker-1.13.1-19.git27e468e.fc26.x86_64
How reproducible:
Always.
Steps to Reproduce:
[root@jlebon ~]# cat reproducer.sh
#!/bin/bash
set -xeuo pipefail
dd if=/dev/urandom of=myfile count=500000
docker run --detach --name cnt registry.fedoraproject.org/fedora:26 sleep infinity
for i in $(seq 5); do
docker cp myfile cnt:/var/tmp
docker cp cnt:/var/tmp/myfile .
done
[root@jlebon ~]# sh reproducer.sh
+ dd if=/dev/urandom of=myfile count=500000
500000+0 records in
500000+0 records out
256000000 bytes (256 MB, 244 MiB) copied, 1.81003 s, 141 MB/s
+ docker run --detach --name cnt registry.fedoraproject.org/fedora:26 sleep infinity
5c2203de1f075d2198194dc4cf2d7a3e891e2973f997ae19c9fb695e0922025f
++ seq 5
+ for i in $(seq 5)
+ docker cp myfile cnt:/var/tmp
Error response from daemon: Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory
[root@jlebon ~]#
Actual results:
Crashes
Expected results:
Doesn't crash
Additional info:
Looking at the memory usage of the docker service (just a simple `watch -n 1 systemctl status docker`), it's as if it's mapping the whole file into memory and not releasing it.
--- Additional comment from Jonathan Lebon on 2017-09-07 10:54:11 EDT ---
Also reproduced on the latest Fedora 26 AH release:
[root@jlebon ~]# rpm-ostree status
State: idle
Deployments:
* fedora-atomic:fedora/26/x86_64/atomic-host
Version: 26.110 (2017-08-20 18:10:09)
Commit: 13ed0f241c9945fd5253689ccd081b5478e5841a71909020e719437bbeb74424
fedora-atomic:fedora/26/x86_64/atomic-host
Version: 26.101 (2017-08-06 21:27:14)
Commit: f6331bcd14577e0ee43db3ba5a44e0f63f74a86e3955604c20542df0b7ad8ad6
--- Additional comment from Jonathan Lebon on 2017-09-07 10:58:48 EDT ---
I hit this specifically when trying to move PAPR[1] to Fedora 26 AH. We often spin up e.g. 8 containers at a time there where we need to `docker cp` whole git repositories simultaneously. Though as the reproducer shows, it doesn't even have to be simultaneous.
I forgot to add the docker version included in v26.110 in my previous comment:
# rpm -q docker
docker-1.13.1-21.git27e468e.fc26.x86_64
[1] https://github.com/projectatomic/papr
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2018:1071
This also affects RHELAH 7.4.1 with docker 1.12.6 # sh repro.sh + dd if=/dev/urandom of=myfile count=500000 500000+0 records in 500000+0 records out 256000000 bytes (256 MB) copied, 1.87501 s, 137 MB/s + docker run --detach --name cnt registry.fedoraproject.org/fedora:26 sleep infinity Unable to find image 'registry.fedoraproject.org/fedora:26' locally Trying to pull repository registry.fedoraproject.org/fedora ... 26: Pulling from registry.fedoraproject.org/fedora 727707040437: Pull complete Digest: sha256:311f2acf4b42a6c1c00cbffe902da9d34a684f127e39cd6b0558a18785390c3f cf0583c0c58fa31b04558a957e46273c0304bac53312dd2367e3c80a3cd4af2b ++ seq 5 + for i in '$(seq 5)' + docker cp myfile cnt:/var/tmp + docker cp cnt:/var/tmp/myfile . + for i in '$(seq 5)' + docker cp myfile cnt:/var/tmp Error response from daemon: Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory # rpm -q docker docker-1.12.6-55.gitc4618fb.el7.x86_64 # rpm-ostree status State: idle Deployments: ● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard Version: 7.4.1 (2017-08-30 19:29:56) Commit: e83c16780259c5272684221e2a6007300d94bbfdc5432f9ab6025300f447145b +++ This bug was initially created as a clone of Bug #1489505 +++ Description of problem: On the latest Fedora 26 AH, it seems really easy to get the docker daemon to exhaust all the system memory by repeated `docker cp` commands until it fails with: Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory This makes it impractical in testing environments where we need to routinely spin up and provision throw-away containers. Version-Release number of selected component (if applicable): [root@jlebon ~]# rpm-ostree status State: idle Deployments: * fedora-atomic:fedora/26/x86_64/atomic-host Version: 26.101 (2017-08-06 21:27:14) Commit: f6331bcd14577e0ee43db3ba5a44e0f63f74a86e3955604c20542df0b7ad8ad6 [root@jlebon ~]# rpm -q docker docker-1.13.1-19.git27e468e.fc26.x86_64 How reproducible: Always. Steps to Reproduce: [root@jlebon ~]# cat reproducer.sh #!/bin/bash set -xeuo pipefail dd if=/dev/urandom of=myfile count=500000 docker run --detach --name cnt registry.fedoraproject.org/fedora:26 sleep infinity for i in $(seq 5); do docker cp myfile cnt:/var/tmp docker cp cnt:/var/tmp/myfile . done [root@jlebon ~]# sh reproducer.sh + dd if=/dev/urandom of=myfile count=500000 500000+0 records in 500000+0 records out 256000000 bytes (256 MB, 244 MiB) copied, 1.81003 s, 141 MB/s + docker run --detach --name cnt registry.fedoraproject.org/fedora:26 sleep infinity 5c2203de1f075d2198194dc4cf2d7a3e891e2973f997ae19c9fb695e0922025f ++ seq 5 + for i in $(seq 5) + docker cp myfile cnt:/var/tmp Error response from daemon: Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory [root@jlebon ~]# Actual results: Crashes Expected results: Doesn't crash Additional info: Looking at the memory usage of the docker service (just a simple `watch -n 1 systemctl status docker`), it's as if it's mapping the whole file into memory and not releasing it. --- Additional comment from Jonathan Lebon on 2017-09-07 10:54:11 EDT --- Also reproduced on the latest Fedora 26 AH release: [root@jlebon ~]# rpm-ostree status State: idle Deployments: * fedora-atomic:fedora/26/x86_64/atomic-host Version: 26.110 (2017-08-20 18:10:09) Commit: 13ed0f241c9945fd5253689ccd081b5478e5841a71909020e719437bbeb74424 fedora-atomic:fedora/26/x86_64/atomic-host Version: 26.101 (2017-08-06 21:27:14) Commit: f6331bcd14577e0ee43db3ba5a44e0f63f74a86e3955604c20542df0b7ad8ad6 --- Additional comment from Jonathan Lebon on 2017-09-07 10:58:48 EDT --- I hit this specifically when trying to move PAPR[1] to Fedora 26 AH. We often spin up e.g. 8 containers at a time there where we need to `docker cp` whole git repositories simultaneously. Though as the reproducer shows, it doesn't even have to be simultaneous. I forgot to add the docker version included in v26.110 in my previous comment: # rpm -q docker docker-1.13.1-21.git27e468e.fc26.x86_64 [1] https://github.com/projectatomic/papr