RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1269395 - yum eats whole memory when running in docker container with OverlayFS
Summary: yum eats whole memory when running in docker container with OverlayFS
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: yum-utils
Version: 7.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Valentina Mukhamedzhanova
QA Contact: Karel Srot
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-07 08:13 UTC by Tomas Tomecek
Modified: 2017-01-02 12:38 UTC (History)
5 users (show)

Fixed In Version: yum-utils-1.1.31-34.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 12:10:44 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2129 0 normal SHIPPED_LIVE yum-utils bug fix and enhancement update 2015-11-19 10:49:36 UTC

Description Tomas Tomecek 2015-10-07 08:13:55 UTC 
I am using rhel7 docker image where I try to install epel rpm:

```
Step 8 : RUN yum -y install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
 ---> Running in a871925ee92f
Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Examining /var/tmp/yum-root-EXtJH_/epel-release-7-5.noarch.rpm: epel-release-7-5.noarch
Marking /var/tmp/yum-root-EXtJH_/epel-release-7-5.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-5 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch         Version   Repository                      Size
================================================================================
Installing:
 epel-release       noarch       7-5       /epel-release-7-5.noarch        24 k

Transaction Summary
================================================================================
Install  1 Package

Total size: 24 k
Installed size: 24 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-5.noarch                                      1/1 
  Verifying  : epel-release-7-5.noarch                                      1/1
```

And at this point it gets stuck. When I strace yum, I get flood of:

```
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
...
```

After a minute or two, yum eats whole memory (several gigs) and is nuked by OOM killer.

My host system is Fedora rawhide with this spec:

```
$ uname -a
Linux oat 4.2.1-300.fc23.x86_64 #1 SMP Mon Sep 21 22:13:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker info
Containers: 20
Images: 113
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.1-300.fc23.x86_64
Operating System: Fedora 24 (Rawhide) (containerized)
CPUs: 4
Total Memory: 11.44 GiB
Name: oat
ID: YC7N:MYIE:6SEL:JYLU:SRIG:PCVV:APZD:WTH4:4MGR:N4BG:CT53:ZW2O

$ docker version
Client:
 Version:      1.9.0-dev-fc24
 API version:  1.21
 Package Version: docker-1.9.0-5.git11b81f9.fc24.x86_64
 Go version:   go1.5.1
 Git commit:   af9b534-dirty
 Built:        Thu Sep 10 17:53:19 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-dev-fc24
 API version:  1.21
 Package Version: 
 Go version:   go1.5.1
 Git commit:   af9b534-dirty
 Built:        Thu Sep 10 17:53:19 UTC 2015
 OS/Arch:      linux/amd64
```

NVRs of packages in container:

```
[root@aaf7235203b8 /]# rpm -q rpm
rpm-4.11.3-17.el7.x86_64
[root@aaf7235203b8 /]# rpm -qa | grep yum
yum-utils-1.1.31-33.el7.noarch
error: rpmdbNextIterator: skipping h#     152 blob size(4812): BAD, 8 + 16 * il(543455598) + dl(1634560355)
yum-metadata-parser-1.1.4-10.el7.x86_64
yum-3.4.3-132.el7.noarch
yum-plugin-ovl-1.1.31-33.el7.noarch
```

When I try to verify integrity of rpm database, I get this:

```
[root@aaf7235203b8 rpm]# /usr/lib/rpm/rpmdb_verify Packages
rpmdb_verify: BDB0677 Page 1837: overflow page of invalid type 0
rpmdb_verify: Packages: BDB0090 DB_VERIFY_BAD: Database verification failed
BDB5105 Verification of Packages failed.
```
Comment 2 Tomas Tomecek 2015-10-07 11:01:37 UTC 
Debug output

```
Step 8 : RUN yum -d 99 -y install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
 ---> Running in fb9ce68e428b
Loading "ovl" plugin
Loading "product-id" plugin
Loading "search-disabled-repos" plugin
Loading "subscription-manager" plugin
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Config time: 0.050
Yum version: 3.4.3
rpmdb time: 0.000
Examining /var/tmp/yum-root-YNnqqU/epel-release-7-5.noarch.rpm: epel-release-7-5.noarch
Marking /var/tmp/yum-root-YNnqqU/epel-release-7-5.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-5 will be installed
Checking deps for epel-release.noarch 0:7-5 - u
looking for ('config(epel-release)', 'EQ', ('0', '7', '5')) as a requirement of epel-release.noarch 0:7-5 - u
looking for ('redhat-release', 'GE', ('0', '7', None)) as a requirement of epel-release.noarch 0:7-5 - u
--> Finished Dependency Resolution
Dependency Process ending
Depsolve time: 0.050

Dependencies Resolved

================================================================================
 Package            Arch         Version   Repository                      Size
================================================================================
Installing:
 epel-release       noarch       7-5       /epel-release-7-5.noarch        24 k

Transaction Summary
================================================================================
Install  1 Package

Total size: 24 k
Installed size: 24 k
Downloading packages:
Member: epel-release.noarch 0:7-5 - u
Adding Package epel-release-7-5.noarch in mode u
Running transaction check
Transaction check time: 0.001
Running transaction test
Transaction test succeeded
Transaction test time: 0.001
Running transaction
  Installing : epel-release-7-5.noarch                                      1/1
Setting up Package Sacks
ovl: Copying up (0) files from OverlayFS lower layer
pkgsack time: 2.421
Installed products updated.
  Verifying  : epel-release-7-5.noarch                                      1/1^C
```
Comment 3 Pavel Odvody 2015-10-07 12:56:03 UTC 
This is caused by the fact that the plugin gets invoked during pre-repository setup hook, and in this particular use-case the repository setup happens too late, because we don't use repository to resolve the package itself: 

   $ yum install -y https://.../some.rpm

Valentina is working on a fix which invokes the plugin during 'init' setup hook, which should give stronger guarantee.
Note that this still leaves a grey area since if PluginA gets executed *before* ovl, and if PluginA also happens to need to work with Rpmdb in some way, it might still be affected by the underlying copy-up issue.
Comment 4 Valentina Mukhamedzhanova 2015-10-09 13:53:33 UTC 
Upstream fix - https://github.com/rpm-software-management/yum-utils/commit/d03fce57c1fa3f9dff6fdd9867cbcaf66df9f841
Comment 9 errata-xmlrpc 2015-11-19 12:10:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2129.html
Note You need to log in before you can comment on or make changes to this bug.