RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1194542 - /run mounted as tmpfs in container and causes changes in the directory to not be persistent
Summary: /run mounted as tmpfs in container and causes changes in the directory to not...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 7.2
Assignee: Daniel Walsh
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-20 06:24 UTC by Chen Chang
Modified: 2019-03-06 02:24 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-20 19:02:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Chen Chang 2015-02-20 06:24:51 UTC
Description of problem:

Directory /run/httpd created in a container by the installation of the httpd rpm is lost once container is exited.  When container is started again, httpd fails to run because it can't create /run/httpd/authdigest.shm.8 since the /run/httpd directory no longer exists.


Version-Release number of selected component (if applicable):

-bash-4.2# cat /etc/redhat-release
Red Hat Atomic Host release 7.1

-bash-4.2# rpm -q docker
docker-1.4.1-37.el7.x86_64


How reproducible:

Can be produced every time.


Steps to Reproduce:

On Atomic host, run:
1. docker pull rhel7
2. docker run -i -t rhel7 /bin/bash
   - runs image ID bef54b8f8a2f

In container, run:
1. yum install -y httpd
2. rpm -V httpd
   - verification passes with no output
3. ls -lad /run/httpd
   - confirms directory was created by httpd rpm install
4. exit

On Atomic host, run:
1. docker ps -a
   - displays ID of exited container
2. docker start CONTAINERID
3. docker attach CONTAINERID

In container, run:
1. rpm -V httpd
   - reports:
missing     /run/httpd
missing     /run/httpd/htcacheclean
2. ls -lad /run/httpd
   - fails and reports:
ls: cannot access /run/httpd: No such file or directory
3. /usr/sbin/httpd -DFOREGROUND; echo $?
   - httpd exits with status 1
   - /var/log/httpd/error_log reports error "No such file or directory: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.8"


Actual results:

/run contents are lost after container is exited.  When container is started again, absence of the /run/httpd directory causes httpd to fail to start.

Expected results:
/run contents should persist after container is exited.  When container is started again, directory /run/httpd should still exist and httpd should start without issues.


Additional info:

Same steps did not result in failure when run on prior release:

-bash-4.2# cat /etc/redhat-release
Red Hat Atomic Host Beta release 7.0 Beta

-bash-4.2# rpm -q docker
docker-1.3.2-4.el7.x86_64

In the above release, when running the same rhel7 image ID bef54b8f8a2f, the running container does not have /run mounted separately as tmpfs.  In the problematic release, the running container has /run mounted separately as tmpfs.

Looks like it might be related to issue mentioned here:
https://github.com/docker/docker/pull/5975

Comment 1 Steve Bonneville 2015-02-20 14:53:44 UTC
Since /run is mounted as tmpfs in regular RHEL 7, that's really weird that the httpd package owns files in there and is installing files in there.  I'm noticing that rpm -ql reports httpd owns /run/httpd and /run/httpd/htcacheclean.

I think you're right, there must be startup scripts that make sure those files are recreated properly in a traditional environment that aren't run in a container.

Comment 2 Andrew Blum 2015-02-20 16:26:17 UTC
I agree this is an issue we need some help with. Running the httpd container like the following isn't a good workaround [although this works]:

-bash-4.2# docker run -d -p 8080:80 rht-training:rhel7_httpd /bin/bash -c "mkdir /run/httpd; httpd -D FOREGROUND"

Also to note that containers built on older atomic versions [pre docker-1.4], don't have any issues running on the latest atomic build [2015-02-19 20:26:26  5799825b36]...but they also don't have tmpfs mounted /run either:

-bash-4.2# docker exec -it 77f0c3c187f0 df -h /run
Filesystem                                                                                          Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:1-16842786-77f0c3c187f0d969807bd235f58749150ddb35d91186863e3dc8dd137c01021c   10G  303M  9.7G   3% /

Comment 4 Stephen Tweedie 2015-02-20 16:56:26 UTC
/run is non-persistent on RHEL and Fedora too, this isn't container-specific.

In httpd's case, these files are created on boot via 
  /usr/lib/tmpfiles.d/httpd.conf
which causes the appropriate /run/httpd and /run/httpd/htcacheclean directories to be created at each boot.  A container start script for httpd will need to replicate this action.

Comment 5 Andrew Blum 2015-02-20 17:21:38 UTC
Ok. based on Stephen's comments and a quick chat on irc...I think this will need to be the httpd "hello world":


FROM https://github.com/fedora-cloud/Fedora-Dockerfiles/tree/master/apache

-bash-4.2# docker run -it registry.access.redhat.com/rhel7 /bin/bash

bash-4.2# yum install httpd -y
bash-4.2# echo kc5 > /var/www/html/index.html

Create startup script:

----------------------

bash-4.2# vi /usr/bin/my_httpd_startup.bash


#!/bin/bash
# Make sure we're not confused by old, incompletely-shutdown httpd
# context after restarting the container. httpd won't start correctly
# if it thinks it is already running.
rm -rf /run/httpd/*

# need to create directories per /usr/lib/tmpfiles.d/httpd.conf
## d /run/httpd   710 root apache
## d /run/httpd/htcacheclean   700 apache apache

mkdir -p /run/httpd/htcacheclean
chmod 710 /run/httpd
chmod 700 /run/httpd/htcacheclean

chown root:apache /run/httpd
chown apache:apache /run/httpd/htcacheclean

exec /usr/sbin/apachectl -D FOREGROUND

--------------------

bash-4.2# chmod 755 /usr/bin/my_httpd_startup.bash


-bash-4.2# docker run -p 8080:80 -d mytest /usr/bin/my_httpd_startup.bash
8518f03a28782745c4148667b1a936e7a052284231bde4c16d82b4ce3a0d0f72
-bash-4.2# docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                  NAMES
8518f03a2878        mytest:latest       "/usr/bin/my_httpd_s   3 seconds ago       Up 2 seconds        0.0.0.0:8080->80/tcp   goofy_tesla         
-bash-4.2# curl localhost:8080
kc5

-bash-4.2# docker stop 8518f03a2878
8518f03a2878

NOTE: docker stop will now take 10 sec, the default time before docker sends the sigkill since our wrapper script can't handle the sigterm.

Comment 6 Daniel Walsh 2015-02-20 19:02:53 UTC
You could also work with systemd in the container.  If you build an image using a docker file, this should work correctly also.

The /run patch will copy the content under /run on the image to /run on the image.


Note You need to log in before you can comment on or make changes to this bug.