Bug 1194542

Summary: /run mounted as tmpfs in container and causes changes in the directory to not be persistent
Product: Red Hat Enterprise Linux 7 Reporter: Chen Chang <cchang>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: ablum, cchang, cpelland, ghacker, sbonnevi, sct
Target Milestone: rcKeywords: Extras
Target Release: 7.2   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-20 19:02:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chen Chang 2015-02-20 06:24:51 UTC
Description of problem:

Directory /run/httpd created in a container by the installation of the httpd rpm is lost once container is exited.  When container is started again, httpd fails to run because it can't create /run/httpd/authdigest.shm.8 since the /run/httpd directory no longer exists.


Version-Release number of selected component (if applicable):

-bash-4.2# cat /etc/redhat-release
Red Hat Atomic Host release 7.1

-bash-4.2# rpm -q docker
docker-1.4.1-37.el7.x86_64


How reproducible:

Can be produced every time.


Steps to Reproduce:

On Atomic host, run:
1. docker pull rhel7
2. docker run -i -t rhel7 /bin/bash
   - runs image ID bef54b8f8a2f

In container, run:
1. yum install -y httpd
2. rpm -V httpd
   - verification passes with no output
3. ls -lad /run/httpd
   - confirms directory was created by httpd rpm install
4. exit

On Atomic host, run:
1. docker ps -a
   - displays ID of exited container
2. docker start CONTAINERID
3. docker attach CONTAINERID

In container, run:
1. rpm -V httpd
   - reports:
missing     /run/httpd
missing     /run/httpd/htcacheclean
2. ls -lad /run/httpd
   - fails and reports:
ls: cannot access /run/httpd: No such file or directory
3. /usr/sbin/httpd -DFOREGROUND; echo $?
   - httpd exits with status 1
   - /var/log/httpd/error_log reports error "No such file or directory: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.8"


Actual results:

/run contents are lost after container is exited.  When container is started again, absence of the /run/httpd directory causes httpd to fail to start.

Expected results:
/run contents should persist after container is exited.  When container is started again, directory /run/httpd should still exist and httpd should start without issues.


Additional info:

Same steps did not result in failure when run on prior release:

-bash-4.2# cat /etc/redhat-release
Red Hat Atomic Host Beta release 7.0 Beta

-bash-4.2# rpm -q docker
docker-1.3.2-4.el7.x86_64

In the above release, when running the same rhel7 image ID bef54b8f8a2f, the running container does not have /run mounted separately as tmpfs.  In the problematic release, the running container has /run mounted separately as tmpfs.

Looks like it might be related to issue mentioned here:
https://github.com/docker/docker/pull/5975

Comment 1 Steve Bonneville 2015-02-20 14:53:44 UTC
Since /run is mounted as tmpfs in regular RHEL 7, that's really weird that the httpd package owns files in there and is installing files in there.  I'm noticing that rpm -ql reports httpd owns /run/httpd and /run/httpd/htcacheclean.

I think you're right, there must be startup scripts that make sure those files are recreated properly in a traditional environment that aren't run in a container.

Comment 2 Andrew Blum 2015-02-20 16:26:17 UTC
I agree this is an issue we need some help with. Running the httpd container like the following isn't a good workaround [although this works]:

-bash-4.2# docker run -d -p 8080:80 rht-training:rhel7_httpd /bin/bash -c "mkdir /run/httpd; httpd -D FOREGROUND"

Also to note that containers built on older atomic versions [pre docker-1.4], don't have any issues running on the latest atomic build [2015-02-19 20:26:26  5799825b36]...but they also don't have tmpfs mounted /run either:

-bash-4.2# docker exec -it 77f0c3c187f0 df -h /run
Filesystem                                                                                          Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:1-16842786-77f0c3c187f0d969807bd235f58749150ddb35d91186863e3dc8dd137c01021c   10G  303M  9.7G   3% /

Comment 4 Stephen Tweedie 2015-02-20 16:56:26 UTC
/run is non-persistent on RHEL and Fedora too, this isn't container-specific.

In httpd's case, these files are created on boot via 
  /usr/lib/tmpfiles.d/httpd.conf
which causes the appropriate /run/httpd and /run/httpd/htcacheclean directories to be created at each boot.  A container start script for httpd will need to replicate this action.

Comment 5 Andrew Blum 2015-02-20 17:21:38 UTC
Ok. based on Stephen's comments and a quick chat on irc...I think this will need to be the httpd "hello world":


FROM https://github.com/fedora-cloud/Fedora-Dockerfiles/tree/master/apache

-bash-4.2# docker run -it registry.access.redhat.com/rhel7 /bin/bash

bash-4.2# yum install httpd -y
bash-4.2# echo kc5 > /var/www/html/index.html

Create startup script:

----------------------

bash-4.2# vi /usr/bin/my_httpd_startup.bash


#!/bin/bash
# Make sure we're not confused by old, incompletely-shutdown httpd
# context after restarting the container. httpd won't start correctly
# if it thinks it is already running.
rm -rf /run/httpd/*

# need to create directories per /usr/lib/tmpfiles.d/httpd.conf
## d /run/httpd   710 root apache
## d /run/httpd/htcacheclean   700 apache apache

mkdir -p /run/httpd/htcacheclean
chmod 710 /run/httpd
chmod 700 /run/httpd/htcacheclean

chown root:apache /run/httpd
chown apache:apache /run/httpd/htcacheclean

exec /usr/sbin/apachectl -D FOREGROUND

--------------------

bash-4.2# chmod 755 /usr/bin/my_httpd_startup.bash


-bash-4.2# docker run -p 8080:80 -d mytest /usr/bin/my_httpd_startup.bash
8518f03a28782745c4148667b1a936e7a052284231bde4c16d82b4ce3a0d0f72
-bash-4.2# docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                  NAMES
8518f03a2878        mytest:latest       "/usr/bin/my_httpd_s   3 seconds ago       Up 2 seconds        0.0.0.0:8080->80/tcp   goofy_tesla         
-bash-4.2# curl localhost:8080
kc5

-bash-4.2# docker stop 8518f03a2878
8518f03a2878

NOTE: docker stop will now take 10 sec, the default time before docker sends the sigkill since our wrapper script can't handle the sigterm.

Comment 6 Daniel Walsh 2015-02-20 19:02:53 UTC
You could also work with systemd in the container.  If you build an image using a docker file, this should work correctly also.

The /run patch will copy the content under /run on the image to /run on the image.