Bug 1708115 - Running mysql container in moby-engine results in 100% memory usage
Summary: Running mysql container in moby-engine results in 100% memory usage
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: moby-engine
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Olivier Lemasle
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-09 07:38 UTC by philbates35
Modified: 2020-03-09 02:27 UTC (History)
4 users (show)

Fixed In Version: moby-engine-18.09.7-4.ce.git2d0083d.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-09 02:27:44 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Screenshots of memory usage (293.96 KB, application/zip)
2019-05-09 07:38 UTC, philbates35
no flags Details

Description philbates35 2019-05-09 07:38:00 UTC
Created attachment 1566001 [details]
Screenshots of memory usage

Description of problem:
Running the official mysql image using the "moby-engine" package with the simple command "docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6" results in 100% memory usage and terrible system performance. As soon as the container is killed, memory usage drops to what it was and the system performance returns.

If you then prune all images using "docker system prune -a", uninstall "moby-engine" and instead install the older "docker" package, you can run "docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6" fine, with expected low memory usage.

The fact that the same image (same digest) works well on "docker" package, but not on "moby-engine" suggests it is an issue with the "moby-engine" package.

Version-Release number of selected component (if applicable):
Name         : moby-engine
Version      : 18.06.3
Release      : 2.ce.gitd7080c1.fc30
Architecture : x86_64


How reproducible:
Every time

Steps to Reproduce:
1. sudo dnf install moby-engine
2. sudo systemctl start docker
3. sudo systemctl enable docker
4. docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6

Actual results:
100% memory usage making system lock up and unresponsive

Expected results:
Not a particularly higher jump in RAM usage, in the exact same way as when using the old "docker" package, for example, this works fine from a fresh system:
1. sudo dnf install docker
2. sudo systemctl start docker
3. sudo systemctl enable docker
4. docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6

Additional info:
I've attached screenshots demonstrating the memory usage:
* moby-engine-starting.png: The change in memory when running "docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6" using the "moby-engine" package (at about 45 seconds)
* moby-engine-started.png: You can see after a few seconds memory stablises and flatlines at 100% usage (at about 12 seconds)
* moby-engine-killed.png: You can see memory usage drop back to what it was when I kill the container (at about 6 seconds)
* docker-started.png: The change in memory when running "docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6" using the old "docker" package (at about 16 seconds)

Comment 1 David Michael 2019-05-15 03:49:37 UTC
I could not reproduce this behavior with a fresh F30 install.  Its memory usage remains stable.

Are you interacting with the database before its memory increases?

Does this happen for you on a new system with no preexisting data?

Does the behavior persist on 18.09?  (It's submitted to updates-testing at https://bodhi.fedoraproject.org/updates/FEDORA-2019-54864c8857 .)

Comment 2 philbates35 2019-05-15 07:28:16 UTC
Hi, thanks for the response.

> Are you interacting with the database before its memory increases?

Nope, the memory starts to increase as soon as the container begins to start up when I run "docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6".

> Does the behavior persist on 18.09?  (It's submitted to updates-testing at https://bodhi.fedoraproject.org/updates/FEDORA-2019-54864c8857 .)

I just saw from your link that 18.09 has been submitted as an update to F30, so I will wait for that update to become available and I will try again once it has been released and update the issue here, running "dnf upgrade moby-engine --refresh --enablerepo=updates-testing" isn't giving me any updates right now).

I always wanted to mention that I was using moby-engine package on F29 with the same mysql image and experienced no issues, the 100% memory usage only started immediately after upgrading to F30 via dnf, and the only fix I've found it to uninstall "moby-engine" package and install the "docker" package.

Comment 3 philbates35 2019-05-20 07:51:11 UTC
Hi,

I saw that moby-engine-18.09.6-1.ce.git481bc77.fc30.src.rpm has been pushed to updates-testing now, so I just tested and I sadly I still still get the same results - 100% RAM usage as soon as I run "docker run -e MYSQL_ROOT_PASSWORD=123 mysql:5.6".

Uninstalling moby-engine and reinstalling the old docker-1.13.1-67.git1185cfd.fc30.src.rpm package once again fixes it, mysql container runs just fine with low RAM usage when using "docker", not "moby-engine".

> Does this happen for you on a new system with no preexisting data?

Unfortunately I'm not in a position to be able to do a full OS re-install (this is a work laptop) - are there any additional steps that I can do to clear out all docker "cache" aside from "docker system -a prune && dnf remove docker moby-engine" (that I've already been running)?

Comment 4 Oliver Gassner 2019-05-22 07:40:00 UTC
I experience the same problem.

mysql containers are eating the ram. 
if I set a  mem_limit in my docker-compose file the container doesn't start. (ERROR: mysqld failed while attempting to check config)

mariadb containers are working fine.

package version: moby-engine-18.06.3-2.ce.gitd7080c1.fc30.x86_64
mysql-container: mysql:5.7

Comment 5 Oliver Gassner 2019-05-22 08:22:32 UTC
addition:
the docker-ce package for fedora 29 (docker-ce-18.09.6-3.fc29.x86_64) doesn't have this problem and seems to be working fine in f30.

Comment 6 Paulo Fidalgo 2019-05-30 14:00:05 UTC
I've notice the same behaviour (consuming all the RAM) but with elasticsearch instead of mysql.
If I run this: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#docker-cli-run-dev-mode I can get it up and running, but if I use the docker-compose.yml file of the same page it will use all the memory and crashing with a message saying it cannot allocate the file table because it's out of memory.

After a fallback to the current docker version (Docker version 1.13.1, build 1185cfd/1.13.1) it worked.

Comment 7 Fedora Update System 2019-07-13 12:43:26 UTC
FEDORA-2019-572b06a0f7 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-572b06a0f7

Comment 8 Fedora Update System 2019-07-14 03:07:39 UTC
moby-engine-18.09.7-4.ce.git2d0083d.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-572b06a0f7

Comment 9 philbates35 2019-07-17 07:37:29 UTC
I just tested moby-engine-18.09.7-4.ce.git2d0083d.fc30 and it seems to have fixed the issue, RAM stays stable instead of increasing to 100% like it was before.

Comment 10 Fedora Update System 2020-03-09 02:27:44 UTC
moby-engine-18.09.7-4.ce.git2d0083d.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.