Bug 1419040

Summary: The change to /var/log mounting breaks the running of services that require a folder in /var/log created at docker build time
Product: Red Hat Enterprise Linux 7 Reporter: James Hogarth <james.hogarth>
Component: oci-systemd-hookAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: ddarrah, dornelas, dwalsh, redhat-bugzilla, robert.scheck, tsweeney
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: oci-systemd-hook was running as container_t, (svirt_lxc_net_t) and should not have been. Consequence: This caused systemctl is-active to show failures when mariadb tried to start within a container and other selinux errors. Fix: This was corrected in oci-systemd-hook-0.1.6-1.gitfe22236.el7.x86_64. Result: mariadb containers now start as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-12 14:51:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1423502    

Description James Hogarth 2017-02-03 14:04:00 UTC
Description of problem:
In the previous version a build script that created a mariadb docker container worked fine. The dockerfile installs mariadb.

The present version results in the container having a minimal /var/log and the /var/log/mariadb that was created by the mariadb-server rpm doesn't exist ar runtime. As a result the mariadb service fails to respond.

Version-Release number of selected component (if applicable):
oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64 <-- broken
oci-systemd-hook-0.1.4-6.git337078c.el7.c86_64 <-- works

How reproducible:
always

Steps to Reproduce:
cat > Dockerfile.mariadb << EOF
FROM centos:latest
STOPSIGNAL SIGRTMIN+3
 
RUN yum -y install mariadb-server && yum clean all
 
RUN systemctl enable mariadb
 
VOLUME /var/lib/mysql
 
CMD ["/sbin/init"]
EOF
 
docker volume create --name localtest-mdb
docker build -f Dockerfile.mariadb -t localtest-mdb .
docker run -dt -v localtest-mdb:/var/lib/mysql --name localtest-mdb localtest-mdb
docker exec -t localtest-mdb /bin/bash -c 'for i in {1..30}; do if systemctl is-active mariadb ; then break  ; else sleep 1 ; fi done;'
docker exec -t localtest-mdb mysql -e "GRANT ALL PRIVILEGES ON *.* TO 'testuser'@'%' IDENTIFIED BY 'testpassword' WITH GRANT OPTION;"
docker stop localtest-mdb
docker rm localtest-mdb

Actual results:
systemctl is-active shows failed in the container when mariadb tries to start rather than active
checking the logs it cannot find /var/log/mariadb , which was created in the docker build as a result of installing the RPM

Expected results:
On a fedora system (or older RHEL7 system, but the docker 1.12 in RHEL extras has -8 as a minimal dependency so I can't cleanly revert to the older hook) systemctl is-active shows mariadb becoming active fine.


Additional info:
Note that using the oci-systemd-hook in fedora from here shows the regression:
https://koji.fedoraproject.org/koji/buildinfo?buildID=837992

On my fedora test system (and older rhel7 installs) where I have the older hook:
[root@7bfbfd09e947 /]# ls -l /var/log/
total 32
drwxr-xr-x. 1 root  root    186 Dec 14 14:59 anaconda
-rw-------. 1 root  utmp      0 Dec 14 14:58 btmp
drwxr-xr-x. 1 root  root     64 Feb  3 14:01 journal
-rw-r--r--. 1 root  root  23944 Feb  3 13:46 lastlog
drwxr-x---. 1 mysql mysql    22 Feb  3 13:46 mariadb
-rw-------. 1 root  root      0 Dec 14 14:58 tallylog
-rw-rw-r--. 1 root  utmp      0 Dec 14 14:58 wtmp
-rw-------. 1 root  root   4446 Feb  3 13:46 yum.log

On my fedora test system with the newer hook from koji (and this matches the current behaviour on rhel7):
[root@43f86ed6dacd /]# ls -l /var/log
total 4
-rw-------. 1 root utmp              0 Feb  3 14:02 btmp
drwxr-sr-x. 3 root systemd-journal  60 Feb  3 14:02 journal
-rw-rw-r--. 1 root utmp            384 Feb  3 14:02 wtmp

It looks like the original data isn't being copied in

Comment 1 James Hogarth 2017-02-03 14:14:37 UTC
Okay I just tested using rpm to force installing the older package and accepting the dependency issues for now ...

I can definitely confirm that the build&run works then and that this update is a regression that breaks anything that expects something in /var/log already to exist - such as mariadb

Comment 3 James Hogarth 2017-02-09 12:00:00 UTC
Confirming in my tests that the recent build in Fedora fixes this:

https://koji.fedoraproject.org/koji/buildinfo?buildID=839423

If this could be built and rolled out to RHEL Extras it'd be appreciated.

Comment 4 Robert Scheck 2017-02-17 13:24:44 UTC
Aside of this, the update of oci-systemd-hook to 0.1.4-9.git671c428.el7.x86_64
leads to bug #1423502 here, while 0.1.5-1.git16f7c8a.fc25.x86_64 (as mentioned
in comment #3) indeed fixes it, thus I also would like to see the fix in RHEL 
Extras.

Cross-filed case 01794554 on the Red Hat customer portal for both issues.

Comment 5 Robert Scheck 2017-02-20 14:04:12 UTC
Btw, let me clarify this: The SELinux errors show up with both packages,

 - oci-systemd-hook-0.1.4-9.git671c428.el7.x86_64
 - oci-systemd-hook-0.1.5-1.git16f7c8a.fc25.x86_64

but not with the old oci-systemd-hook-0.1.4-6.git337078c.el7.c86_64. However,
yes, oci-systemd-hook-0.1.5-1.git16f7c8a.fc25.x86_64, makes MariaDB instances
working again.

Comment 6 Robert Scheck 2017-02-20 21:12:29 UTC
As conclusion on bug #1423502 comment #5, 0.1.5-1.git16f7c8a.fc25.x86_64
for RHEL Extras is not the ultimate solution, oci-systemd-hook still needs
to be fixed instead (to avoid the behavior described in bug #1423502).

Comment 7 Daniel Walsh 2017-03-12 11:48:56 UTC
Fixed in oci-systemd-hook-0.1.6-1.gitfe22236.el7

Comment 8 Tom Sweeney 2017-03-25 20:27:56 UTC
(In reply to Daniel Walsh from comment #7)
> Fixed in oci-systemd-hook-0.1.6-1.gitfe22236.el7

It appears to be fixed.  Test results follow.  Dan I'll touch base with you Monday about closing.

[root@rhel73bz ~]# cat > Dockerfile.mariadb << EOF
> FROM centos:latest
> STOPSIGNAL SIGRTMIN+3
>  
> RUN yum -y install mariadb-server && yum clean all
>  
> RUN systemctl enable mariadb
>  
> VOLUME /var/lib/mysql
>  
> CMD ["/sbin/init"]
> EOF
[root@rhel73bz ~]# docker volume create --name localtest-mdb
localtest-mdb
[root@rhel73bz ~]# docker build -f Dockerfile.mariadb -t localtest-mdb .
Sending build context to Docker daemon 14.85 kB
Step 1 : FROM centos:latest
 ---> 98d35105a391
Step 2 : STOPSIGNAL SIGRTMIN+3
 ---> Using cache
 ---> d21037da37ed
Step 3 : RUN yum -y install mariadb-server && yum clean all
 ---> Using cache
 ---> 4440f237aa2b
Step 4 : RUN systemctl enable mariadb
 ---> Using cache
 ---> 007cbeb7dcff
Step 5 : VOLUME /var/lib/mysql
 ---> Using cache
 ---> dc7109037dbf
Step 6 : CMD /sbin/init
 ---> Using cache
 ---> 53e981575dd6
Successfully built 53e981575dd6
[root@rhel73bz ~]# docker run -dt -v localtest-mdb:/var/lib/mysql --name localtest-mdb localtest-mdb
e54bcedff15d675fecba00e74a302a2a0948dc99c17afd575992ee501675a114
[root@rhel73bz ~]# docker exec -t localtest-mdb /bin/bash -c 'for i in {1..30}; do if systemctl is-active mariadb ; then break  ; else sleep 1 ; fi done;'
activating
activating
activating
active
[root@rhel73bz ~]# docker exec -t localtest-mdb mysql -e "GRANT ALL PRIVILEGES ON *.* TO 'testuser'@'%' IDENTIFIED BY 'testpassword' WITH GRANT OPTION;"
[root@rhel73bz ~]# docker stop localtest-mdb
localtest-mdb
[root@rhel73bz ~]# docker rm localtest-mdb
localtest-mdb
[root@rhel73bz ~]# uname -a
Linux rhel73bz.localdomain 3.10.0-625.el7.x86_64 #1 SMP Thu Mar 23 11:04:30 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@rhel73bz ~]# 
[root@rhel73bz ~]# 
[root@rhel73bz ~]# rpm -qa | grep oci-systemd-hook
oci-systemd-hook-0.1.6-1.gitfe22236.el7.x86_64

Comment 10 David Darrah/Red Hat QE 2017-03-28 21:03:56 UTC
Verified as above on RHEL with remote db service access across the network.

Comment 13 errata-xmlrpc 2017-04-12 14:51:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0949