Bug 1279404

Summary: The clustered mysql pod keeps CrashLoopBackOff for cannot create directory '/var/lib/mysql/data/mysql': Permission denied
Product: OpenShift Container Platform Reporter: zhou ying <yinzhou>
Component: ImageStreamsAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.0.0CC: aos-bugs, bleanhar, bparees, haowang, hhorak, jokerman, mmccomas, mnagy, pruan, wzheng, xiuwang
Target Milestone: ---Keywords: Regression, UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1280466 1373099 (view as bug list) Environment:
Last Closed: 2016-01-26 19:17:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1280466, 1373099    

Description zhou ying 2015-11-09 11:45:06 UTC
Description of problem:
Create clustered mysql-55 app, the pods can be running. The logs :
[zhouy@zhouy testjson]$ oc logs mysql-slave-1-0r12e
/usr/local/bin/run-mysqld.sh: line 59: exec: run-mysqld-slave: not found
[zhouy@zhouy testjson]$ oc logs mysql-master-1-90hhm
/usr/local/bin/run-mysqld.sh: line 59: exec: run-mysqld-master: not found

Version-Release number of selected component (if applicable):
openshift v3.1.0.2
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2
[root@openshift-158 ~]# docker images |grep mysql-55
virt-openshift-05.lab.eng.nay.redhat.com:5000/openshift3/mysql-55-rhel7                         latest              c3d990247510        7 weeks ago         306 MB
registry.access.redhat.com/openshift3/mysql-55-rhel7                                            latest              c3d990247510        7 weeks ago         306 MB

How reproducible:
always

Steps to Reproduce:
1. Create a persistentvolume;
2. Create the clustered mysql-55 app;
  `oc process -f https://raw.githubusercontent.com/openshift/mysql/master/5.5/examples/replica/mysql_replica.json |oc create -f -`
3. Check the pod 


Actual results:
The pod can not be running, with logs like description. When the pod run failed, the status was:CrashLoopBackOff

Expected results:
The pods should be running ok.

Additional info:
[
{
    "Id": "c3d990247510bcd3e1dbc3093a97bad5cfad753ea7bba9d74457515aa5d62406",
    "Parent": "82ad5fa11820c2889c60f7f748d67aab04400700c581843db0d1e68735327443",
    "Comment": "",
    "Created": "2015-09-15T18:50:50.846298Z",
    "Container": "",
    "ContainerConfig": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": null,
        "PublishService": "",
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": null,
        "Image": "",
        "Volumes": null,
        "VolumeDriver": "",
        "WorkingDir": "",
        "Entrypoint": null,
        "NetworkDisabled": false,
        "MacAddress": "",
        "OnBuild": null,
        "Labels": null
    },
    "DockerVersion": "1.6.2",
    "Author": "",
    "Config": {
        "Hostname": "",
        "Domainname": "",
        "User": "27",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "3306/tcp": {}
        },
        "PublishService": "",
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "container=docker",
            "PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin",
            "MYSQL_VERSION=5.5",
            "HOME=/var/lib/mysql",
            "BASH_ENV=/var/lib/mysql/scl_enable",
            "ENV=/var/lib/mysql/scl_enable",
            "PROMPT_COMMAND=. /var/lib/mysql/scl_enable"
        ],
        "Cmd": [
            "mysqld"
        ],
        "Image": "82ad5fa11820c2889c60f7f748d67aab04400700c581843db0d1e68735327443",
        "Volumes": {
            "/var/lib/mysql/data": {}
        },
        "VolumeDriver": "",
        "WorkingDir": "",
        "Entrypoint": [
            "run-mysqld.sh"
        ],
        "NetworkDisabled": false,
        "MacAddress": "",
        "OnBuild": [],
        "Labels": {
            "Architecture": "x86_64",
            "Authoritative_Registry": "registry.access.redhat.com",
            "BZComponent": "openshift-mysql-docker",
            "Build_Host": "rcm-img-docker01.build.eng.bos.redhat.com",
            "Name": "openshift3/mysql-55-rhel7",
            "Release": "5",
            "Vendor": "Red Hat, Inc.",
            "Version": "5.5",
            "io.k8s.description": "MySQL is a multi-user, multi-threaded SQL database server",
            "io.k8s.display-name": "MySQL 5.5",
            "io.openshift.expose-services": "3306:mysql",
            "io.openshift.tags": "database,mysql,mysql55"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Size": 147708402,
    "VirtualSize": 305959122,
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "24",
            "DeviceName": "docker-253:0-13539188-c3d990247510bcd3e1dbc3093a97bad5cfad753ea7bba9d74457515aa5d62406",
            "DeviceSize": "107374182400"
        }
    }
}
]

Comment 2 Ben Parees 2015-11-09 13:50:31 UTC
This is almost certainly because the image is 7 weeks old:

registry.access.redhat.com/openshift3/mysql-55-rhel7                                            latest              c3d990247510        7 weeks ago         306 MB


Assigning to Scott to move to ON_QA after a new version is built.

Comment 6 Wenjing Zheng 2015-11-11 07:08:02 UTC
mysql-56-rhel7 has same issue with log as below:
-bash-4.2# docker logs a3472a940df7
The 'master' server-id is 1
Running mysql_install_db ...
2015-11-11 02:06:07 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-11 02:06:07 0 [Note] /opt/rh/rh-mysql56/root/usr/libexec/mysqld (mysqld 5.6.26-log) starting as process 15 ...
2015-11-11 02:06:07 15 [Warning] Can't create test file /var/lib/mysql/data/mysql-master-1-f5903.lower-test
2015-11-11 02:06:07 15 [Warning] Can't create test file /var/lib/mysql/data/mysql-master-1-f5903.lower-test
/opt/rh/rh-mysql56/root/usr/libexec/mysqld: File '/var/lib/mysql/data/mysql-bin.index' not found (Errcode: 13 - Permission denied)
2015-11-11 02:06:07 15 [ERROR] Aborting

2015-11-11 02:06:07 15 [Note] Binlog end
2015-11-11 02:06:07 15 [Note] /opt/rh/rh-mysql56/root/usr/libexec/mysqld: Shutdown complete

Comment 8 Martin Nagy 2015-11-11 13:32:51 UTC
Wenjing, can you please provide steps to reproduce?

Comment 9 Honza Horak 2015-11-11 13:37:55 UTC
I might be off-topic here, so scratch that if that is not valid -- "Permission denied" errors are often caused by wrong selinux context being used. I don't see the code, but just be sure you use :Z modificator in -v docker option and use correct owner of the directory for data, that is volume-mounted into the container.

Comment 10 Martin Nagy 2015-11-11 13:47:39 UTC
Brenton explained to me that the steps are the same as in the description.

Wenjing, we test the MySQL image with a hostPath storage plugin pretty regularly and no such errors are there. Maybe there is something wrong with your PV setup?

Honza, your suggestion might not be off-base, though remember that they are running it in OS not Docker.

Ben, maybe somebody from the storage team can take a look at this and help QE out?

Comment 11 Ben Parees 2015-11-11 19:47:14 UTC
It looks to me like the 55 and 56 rhel images need to be republished to pick up the script renaming that happened.

the centos images work fine(I just tested them), and as Wenjing noted, the jenkins-built rhel images work fine.

This is not a blocker for 3.1 since
1) the images don't ship w/ the release
2) replication is a tech preview

Assigning to Scott to move to ON_QA when the mysql55 image is formally published (the image on RCM looks good: rcm-img-docker01.build.eng.bos.redhat.com:5001/openshift3/mysql-55-rhel7), and will create a clone for Honza to cover the mysql56 image.

Note that none of this relates to the permission denied error that's being reported in comment 5 and 6, i did not see those issues when testing with an NFS volume and the Centos 55 and 56 images, so I suspect it is a configuration issue.

Comment 12 Wenjing Zheng 2015-11-12 05:12:14 UTC
Sorry, the sebool virt_use_nfs is not turned on by default in AEP container installation which causes this issue. I verified again with this option turned on, both mysql-55 and mysql-56 works well now. Thanks for your patience. @sdodson, @mnagy please feel free to change this bug status to ON_QA, so that I can verify it.

Comment 13 Martin Nagy 2015-11-12 07:46:43 UTC
Thanks for the quick verification Wenjing :)

Comment 14 Wenjing Zheng 2015-11-12 09:14:54 UTC
Verified with below version:
mysql-56-rhel7 8101d9386118
mysql-55-rhel7 4aa22c02fb8a

Comment 16 errata-xmlrpc 2016-01-26 19:17:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070