Bug 1302167

Summary: When stopping httpd container that be startd via kubernetes with "docker kill" on the node, it fails to automatically re-start the container.
Product: Red Hat Enterprise Linux 7 Reporter: kyoneyama <kyoneyam>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED NOTABUG QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.2CC: lsm5
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-03 19:07:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description kyoneyama 2016-01-27 01:26:01 UTC
Description of problem:

When stopping httpd container that be startd via kubernetes with "docker kill" on the node, it fails to automatically re-start the container by recovery of kubernetes.


I commited the failed container, and I looked at the error_log file that has been recorded in the container. Then, httpd digest authentication module causes the error.

  # docker ps -f status=exited
  # docker commit <container id> test
  # docker run -it --rm --entrypoint=/bin/bash test
  
    # cat /var/log/httpd/error_log
    [root@ec2ae5cc6ccb httpd]# cat error_log 
    [Tue Jan 19 01:05:17.409548 2016] [suexec:notice] [pid 1] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
    AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.20.66.32. Set the 'ServerName' directive globally to suppress this message
    [Tue Jan 19 01:05:17.423792 2016] [auth_digest:notice] [pid 1] AH01757: generating secret for digest authentication ...
    [Tue Jan 19 01:05:17.424578 2016] [auth_digest:error] [pid 1] (17)File exists: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.1
    [Tue Jan 19 01:05:17.424606 2016] [auth_digest:error] [pid 1] (17)File exists: AH01760: failed to initialize shm - all nonce-count checking, one-time nonces, and MD5-sess algorithm disabled
    [Tue Jan 19 01:05:17.424611 2016] [:emerg] [pid 1] AH00020: Configuration Failed, exiting


Version-Release number of selected component (if applicable):

  Master:
  - kubernetes-1.0.3-0.2.gitb9a88a7.el7.
  - etcd-2.1.1-2.el7.x86_64

  Node:
  - kernel-3.10.0-327.3.1.el7
  - kubernetes-1.0.3-0.2.gitb9a88a7.el7
  - docker-1.8.2-10.el7

How reproducible:

Always

Steps to Reproduce:

-Configuration

  --- simpleweb-service.yml ---
  apiVersion: v1beta3
  kind: Service
  metadata:
    labels:
      name: simpleweb
    name: simpleweb-service
    namespace: default
  spec:
    ports:
    - port: 80
    selector:
      name: simpleweb

  --- simpleweb-rc.yml ---
  apiVersion: v1beta3
  kind: ReplicationController
  metadata:
    name: simpleweb-controller
  spec:
    replicas: 1
    selector:
      name: simpleweb
    template:
      spec:
        containers:
        - name: simpleweb
          image: simpleweb-image
          ports:
          - containerPort: 80
            hostPort: 80
      metadata:
        labels:
          name: simleweb
        selectorname: simpleweb
    labels:
      name: simpleweb


  --- Dockerfile ---
  FROM rhel7:latest
  USER root
  
  # Fix per https://bugzilla.redhat.com/show_bug.cgi?id=1192200 
  RUN yum -y install deltarpm yum-utils --disablerepo=*
  RUN yum-config-manager --disable *-eus-* *-htb-* *-ha-* *-rt-* *-lb-* *-rs-* *-sap-* > /dev/null
  
  # Update image
  RUN yum update -y
  RUN yum install httpd -y
  
  # Add httpd app and configuration
  RUN echo "The Web Server is Running" > /var/www/html/index.html
  EXPOSE 80
  
  # Start the service
  CMD ["-D", "FOREGROUND"]
  ENTRYPOINT ["/usr/sbin/httpd"]


-Reproducer

  1. (Node) Build httpd container
    # cp Dockerfile simpleweb/
    # docker build -t simpleweb-image simpleweb/

  2. (Master) Run httpd container via kubernetes
    # kubectl create -f simpleweb-service.yml
    # kubectl create -f simpleweb-rc.yml
    # kubectl get pod -w

  3. (Node) Stop the container

    (OK)
    # docker stop <simpleweb container id>
    # docker ps | grep simpleweb
    # docker ps -f status=exited | grep simpleweb

    (NG)
    # docker kill <simpleweb container id>
    # docker ps | grep simpleweb
    # docker ps -f status=exited | grep simpleweb


Actual results:

httpd container stops immediately.

Expected results:

httpd container continues running.


Additional info:

- Since it does not reproduce in httpd container based on RHEL6, it seems that the httpd container based on RHEL7 only problem.

- A workaround is adding "VOLUME /run" in the Dockerfile.

Comment 2 Daniel Walsh 2016-02-22 21:19:10 UTC
I don't see how this is a docker or k8s problem, Seems to be more of a bug about httpd running inside of a container.  I would prefer to use --tmpfs mounted on /run, although docker has pulled the code to handle this correctly.