Bug 1430627 - Logging deployment with NFS volume failed
Summary: Logging deployment with NFS volume failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: ewolinet
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On:
Blocks: 1426536
TreeView+ depends on / blocked
 
Reported: 2017-03-09 07:18 UTC by Gaoyun Pei
Modified: 2017-07-24 14:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
New logging role was missing using some defaults based on the previous hosted role
Clone Of:
Environment:
Last Closed: 2017-04-12 19:03:34 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0903 0 normal SHIPPED_LIVE OpenShift Container Platform atomic-openshift-utils bug fix and enhancement 2017-04-12 22:45:42 UTC

Description Gaoyun Pei 2017-03-09 07:18:47 UTC
Description of problem:
Enable logging deployment with NFS storage volume, start installation playbook.
The installation job failed when creating logging pvc.


Version-Release number of selected component (if applicable):
openshift-ansible-3.5.28-1.git.0.103513e.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1. Set the following options in ansible inventory, start installation playbook

openshift_hosted_logging_deploy=true
openshift_hosted_logging_storage_kind=nfs
openshift_hosted_logging_storage_access_modes=['ReadWriteOnce']
openshift_hosted_logging_storage_nfs_directory=/var/lib/exports
openshift_hosted_logging_storage_nfs_options=*(rw,root_squash)
openshift_hosted_logging_storage_volume_name=logging
openshift_hosted_logging_storage_volume_size=10Gi
openshift_hosted_logging_deployer_prefix=x.openshift.com/openshift3/
openshift_hosted_logging_deployer_version=3.5.0

ansible-playbook -i inventory_file /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml



Actual results:
TASK [openshift_logging : Applying /tmp/openshift-logging-ansible-4KJhN0/templates/logging--0-pvc.yaml] ***
Thursday 09 March 2017  03:15:28 +0000 (0:00:00.967)       0:20:09.370 ******** 
fatal: [ec2-54-87-19-41.compute-1.amazonaws.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "--config=/tmp/openshift-logging-ansible-4KJhN0/admin.kubeconfig", 
        "apply", 
        "-f", 
        "/tmp/openshift-logging-ansible-4KJhN0/templates/logging--0-pvc.yaml", 
        "-n", 
        "logging"
    ], 
    "delta": "0:00:00.160891", 
    "end": "2017-03-08 22:15:28.586282", 
    "failed": true, 
    "failed_when_result": true, 
    "rc": 1, 
    "start": "2017-03-08 22:15:28.425391", 
    "warnings": []
}

STDERR:

error: unable to decode "/tmp/openshift-logging-ansible-4KJhN0/templates/logging--0-pvc.yaml": [pos 107]: json: expect char '"' but got char '0'



Expected results:


Additional info:

Comment 3 Gaoyun Pei 2017-03-14 06:31:09 UTC
Test this with openshift-ansible-3.5.32-1.git.0.42cf266.el7.noarch.rpm


Set the following options in ansible inventory, start installation playbook
openshift_hosted_logging_deploy=true
openshift_hosted_logging_storage_kind=nfs
openshift_hosted_logging_storage_access_modes=['ReadWriteOnce']
openshift_hosted_logging_storage_nfs_directory=/var/lib/exports
openshift_hosted_logging_storage_nfs_options=*(rw,root_squash)
openshift_hosted_logging_storage_volume_name=logging
openshift_hosted_logging_storage_volume_size=10Gi
openshift_hosted_logging_deployer_prefix=x.com/openshift3/
openshift_hosted_logging_deployer_version=3.5.0


Installer failed when checking pvc of logging

TASK [openshift_logging : Checking generation of PersistentVolumeClaim -0] *****
Tuesday 14 March 2017  03:43:02 +0000 (0:00:03.900)       0:19:23.997 ********* 
fatal: [ec2-54-242-251-198.compute-1.amazonaws.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "--config=/tmp/openshift-logging-ansible-6TA28W/admin.kubeconfig", 
        "get", 
        "PersistentVolumeClaim", 
        "-0", 
        "-o", 
        "jsonpath={.metadata.resourceVersion}", 
        "-n", 
        "logging"
    ], 
    "delta": "0:00:00.155224", 
    "end": "2017-03-13 23:43:02.198647", 
    "failed": true, 
    "failed_when_result": true, 
    "rc": 1, 
    "start": "2017-03-13 23:43:02.043423", 
    "warnings": []
}

STDERR:

Error: unknown shorthand flag: '0' in -0

Comment 4 ewolinet 2017-03-14 14:21:19 UTC
Looks like its not getting a meaningful PVC name, its thinking its "-0" which is getting treated as a flag.

Comment 6 Scott Dodson 2017-03-15 18:25:03 UTC
Additional changes merged in openshift-ansible-3.5.34-1.git.0.2f1700a

Comment 7 Gaoyun Pei 2017-03-16 06:00:19 UTC
Verify this bug with openshift-ansible-3.5.35-1.git.0.7aa4728.el7.noarch.rpm

Set the following options in ansible inventory, start installation playbook
openshift_hosted_logging_deploy=true
openshift_hosted_logging_storage_kind=nfs
openshift_hosted_logging_storage_access_modes=['ReadWriteOnce']
openshift_hosted_logging_storage_nfs_directory=/var/lib/exports
openshift_hosted_logging_storage_nfs_options=*(rw,root_squash)
openshift_hosted_logging_storage_volume_name=logging
openshift_hosted_logging_storage_volume_size=10Gi
openshift_hosted_logging_deployer_prefix=xxx.com/openshift3/
openshift_hosted_logging_deployer_version=3.5.0

After installation, check logging pod status:

[root@ip-172-18-6-43 ~]# oc get pod -n logging
NAME                          READY     STATUS    RESTARTS   AGE
logging-curator-1-swpq5       1/1       Running   0          3m
logging-es-7jjts29b-1-9lxz4   1/1       Running   0          1m
logging-fluentd-20c45         1/1       Running   0          1m
logging-fluentd-zmnqv         1/1       Running   0          1m
logging-kibana-1-zj783        2/2       Running   0          2m
[root@ip-172-18-6-43 ~]# oc get pvc -n logging
NAME           STATUS    VOLUME           CAPACITY   ACCESSMODES   AGE
logging-es-0   Bound     logging-volume   10Gi       RWO           2m
[root@ip-172-18-6-43 ~]# oc rsh -n logging logging-es-7jjts29b-1-9lxz4
sh-4.2$ df -h
Filesystem                                                                                         Size  Used Avail Use% Mounted on
/dev/mapper/docker-202:2-8390960-73835aa93ab91fa8da75c09c62d9cee5d0e9ee841aaa45df1c33461469f1c4cf   10G  442M  9.6G   5% /
tmpfs                                                                                              7.7G     0  7.7G   0% /dev
tmpfs                                                                                              7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/xvda2                                                                                          25G  5.7G   20G  23% /etc/hosts
x.compute-1.amazonaws.com:/var/lib/exports/logging                                  25G  3.2G   22G  13% /elasticsearch/persistent
shm                                                                                                 64M     0   64M   0% /dev/shm
tmpfs                                                                                              7.7G   32K  7.7G   1% /etc/elasticsearch/secret
tmpfs                                                                                              7.7G   16K  7.7G   1% /run/secrets/kubernetes.io/serviceaccount

Comment 9 errata-xmlrpc 2017-04-12 19:03:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903


Note You need to log in before you can comment on or make changes to this bug.