Bug 1269395

Summary: yum eats whole memory when running in docker container with OverlayFS
Product: Red Hat Enterprise Linux 7 Reporter: Tomas Tomecek <ttomecek>
Component: yum-utilsAssignee: Valentina Mukhamedzhanova <vmukhame>
Status: CLOSED ERRATA QA Contact: Karel Srot <ksrot>
Severity: high Docs Contact:
Priority: high    
Version: 7.2CC: james.antill, ksrot, lmiksik, packaging-team-maint, podvody
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: yum-utils-1.1.31-34.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 12:10:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tomas Tomecek 2015-10-07 08:13:55 UTC
I am using rhel7 docker image where I try to install epel rpm:

```
Step 8 : RUN yum -y install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
 ---> Running in a871925ee92f
Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Examining /var/tmp/yum-root-EXtJH_/epel-release-7-5.noarch.rpm: epel-release-7-5.noarch
Marking /var/tmp/yum-root-EXtJH_/epel-release-7-5.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-5 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch         Version   Repository                      Size
================================================================================
Installing:
 epel-release       noarch       7-5       /epel-release-7-5.noarch        24 k

Transaction Summary
================================================================================
Install  1 Package

Total size: 24 k
Installed size: 24 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-5.noarch                                      1/1 
  Verifying  : epel-release-7-5.noarch                                      1/1
```

And at this point it gets stuck. When I strace yum, I get flood of:

```
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
write(21, "error: rpmdb: damaged header #152 retrieved -- skipping.\n", 57) = 57
...
```

After a minute or two, yum eats whole memory (several gigs) and is nuked by OOM killer.

My host system is Fedora rawhide with this spec:

```
$ uname -a
Linux oat 4.2.1-300.fc23.x86_64 #1 SMP Mon Sep 21 22:13:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker info
Containers: 20
Images: 113
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.1-300.fc23.x86_64
Operating System: Fedora 24 (Rawhide) (containerized)
CPUs: 4
Total Memory: 11.44 GiB
Name: oat
ID: YC7N:MYIE:6SEL:JYLU:SRIG:PCVV:APZD:WTH4:4MGR:N4BG:CT53:ZW2O

$ docker version
Client:
 Version:      1.9.0-dev-fc24
 API version:  1.21
 Package Version: docker-1.9.0-5.git11b81f9.fc24.x86_64
 Go version:   go1.5.1
 Git commit:   af9b534-dirty
 Built:        Thu Sep 10 17:53:19 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-dev-fc24
 API version:  1.21
 Package Version: 
 Go version:   go1.5.1
 Git commit:   af9b534-dirty
 Built:        Thu Sep 10 17:53:19 UTC 2015
 OS/Arch:      linux/amd64
```

NVRs of packages in container:

```
[root@aaf7235203b8 /]# rpm -q rpm
rpm-4.11.3-17.el7.x86_64
[root@aaf7235203b8 /]# rpm -qa | grep yum
yum-utils-1.1.31-33.el7.noarch
error: rpmdbNextIterator: skipping h#     152 blob size(4812): BAD, 8 + 16 * il(543455598) + dl(1634560355)
yum-metadata-parser-1.1.4-10.el7.x86_64
yum-3.4.3-132.el7.noarch
yum-plugin-ovl-1.1.31-33.el7.noarch
```

When I try to verify integrity of rpm database, I get this:

```
[root@aaf7235203b8 rpm]# /usr/lib/rpm/rpmdb_verify Packages
rpmdb_verify: BDB0677 Page 1837: overflow page of invalid type 0
rpmdb_verify: Packages: BDB0090 DB_VERIFY_BAD: Database verification failed
BDB5105 Verification of Packages failed.
```

Comment 2 Tomas Tomecek 2015-10-07 11:01:37 UTC
Debug output

```
Step 8 : RUN yum -d 99 -y install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
 ---> Running in fb9ce68e428b
Loading "ovl" plugin
Loading "product-id" plugin
Loading "search-disabled-repos" plugin
Loading "subscription-manager" plugin
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Config time: 0.050
Yum version: 3.4.3
rpmdb time: 0.000
Examining /var/tmp/yum-root-YNnqqU/epel-release-7-5.noarch.rpm: epel-release-7-5.noarch
Marking /var/tmp/yum-root-YNnqqU/epel-release-7-5.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-5 will be installed
Checking deps for epel-release.noarch 0:7-5 - u
looking for ('config(epel-release)', 'EQ', ('0', '7', '5')) as a requirement of epel-release.noarch 0:7-5 - u
looking for ('redhat-release', 'GE', ('0', '7', None)) as a requirement of epel-release.noarch 0:7-5 - u
--> Finished Dependency Resolution
Dependency Process ending
Depsolve time: 0.050

Dependencies Resolved

================================================================================
 Package            Arch         Version   Repository                      Size
================================================================================
Installing:
 epel-release       noarch       7-5       /epel-release-7-5.noarch        24 k

Transaction Summary
================================================================================
Install  1 Package

Total size: 24 k
Installed size: 24 k
Downloading packages:
Member: epel-release.noarch 0:7-5 - u
Adding Package epel-release-7-5.noarch in mode u
Running transaction check
Transaction check time: 0.001
Running transaction test
Transaction test succeeded
Transaction test time: 0.001
Running transaction
  Installing : epel-release-7-5.noarch                                      1/1
Setting up Package Sacks
ovl: Copying up (0) files from OverlayFS lower layer
pkgsack time: 2.421
Installed products updated.
  Verifying  : epel-release-7-5.noarch                                      1/1^C
```

Comment 3 Pavel Odvody 2015-10-07 12:56:03 UTC
This is caused by the fact that the plugin gets invoked during pre-repository setup hook, and in this particular use-case the repository setup happens too late, because we don't use repository to resolve the package itself: 

   $ yum install -y https://.../some.rpm

Valentina is working on a fix which invokes the plugin during 'init' setup hook, which should give stronger guarantee.
Note that this still leaves a grey area since if PluginA gets executed *before* ovl, and if PluginA also happens to need to work with Rpmdb in some way, it might still be affected by the underlying copy-up issue.

Comment 9 errata-xmlrpc 2015-11-19 12:10:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2129.html