Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1466233 - catalog apiserver fail to be started due to a non-existing etcd ca file path when cluster is using embedded etcd.
catalog apiserver fail to be started due to a non-existing etcd ca file path ...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: ewolinet
Johnny Liu
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-29 06:07 EDT by Johnny Liu
Modified: 2017-08-16 15 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 01:28:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1716 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 05:02:50 EDT

  None (edit)
Description Johnny Liu 2017-06-29 06:07:44 EDT
Description of problem:
see the following details.

Version-Release number of selected component (if applicable):
openshift-ansible-3.6.126.3-1.git.0.178cea4.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Add the following lines into inventory host file to deploy catalog service.
openshift_hosted_etcd_storage_kind=nfs
openshift_hosted_etcd_storage_nfs_options="*(rw,root_squash,sync,no_wdelay)"
openshift_hosted_etcd_storage_nfs_directory=/exports
openshift_hosted_etcd_storage_volume_name=etcd
openshift_hosted_etcd_storage_access_modes=["ReadWriteOnce"]
openshift_hosted_etcd_storage_volume_size=10G
openshift_hosted_etcd_storage_labels={'storage': 'etcd'}

openshift_enable_service_catalog=true
openshift_service_catalog_image_prefix=docker.io/openshift/origin-
openshift_service_catalog_image_version=latest
ansible_service_broker_image_prefix=ansibleplaybookbundle/
ansible_service_broker_image_tag=latest
ansible_service_broker_etcd_image_prefix=quay.io/coreos/
ansible_service_broker_etcd_image_tag=latest

2. Trigger installation.
3.

Actual results:
Failed at the following task:
TASK [openshift_service_catalog : wait for api server to be ready] *************
Thursday 29 June 2017  07:16:38 +0000 (0:00:01.511)       0:32:12.027 ********* 

FAILED - RETRYING: TASK: openshift_service_catalog : wait for api server to be ready (120 retries left).
<--snip-->
FAILED - RETRYING: TASK: openshift_service_catalog : wait for api server to be ready (1 retries left).

fatal: [openshift-143.lab.sjc.redhat.com]: FAILED! => {
    "attempts": 120, 
    "changed": false, 
    "cmd": [
        "curl", 
        "-k", 
        "https://apiserver.kube-service-catalog.svc/healthz"
    ], 
    "delta": "0:00:01.264269", 
    "end": "2017-06-29 03:22:04.963677", 
    "failed": true, 
    "rc": 7, 
    "start": "2017-06-29 03:22:03.699408", 
    "warnings": []
}

STDERR:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused

Log into cluster, check apiserver logs, found that:
# oc get po
NAME                       READY     STATUS             RESTARTS   AGE
apiserver-7xgxn            0/1       CrashLoopBackOff   6          11m
controller-manager-3ljw3   0/1       CrashLoopBackOff   15         1h

# oc logs apiserver-7xgxn
<--snip-->
F0629 08:11:41.686596       1 storage_decorator.go:61] Unable to create storage backend: config (&{ /k8s.io/service-catalog [https://192.168.2.41:4001 https://192.168.2.41:4001] /etc/origin/master/master.etcd-client.key /etc/origin/master/master.etcd-client.crt /etc/origin/master/master.etcd-ca.crt false 0 {0xc4204498c0 0xc420449950} 0xc420377a00}), err (open /etc/origin/master/master.etcd-ca.crt: no such file or directory)
<--snip-->

On host, there is no /etc/origin/master/master.etcd-ca.crt, only /etc/origin/master/ca-bundle.crt is available for embedded etcd.

After update daemonset to /etc/origin/master/ca-bundle.crt, apiserver pod is running now.


Expected results:
api server pod is running well.

Additional info:
Comment 1 Scott Dodson 2017-06-30 08:31:26 EDT
We should probably check if /etc/origin/master/master.etcd-ca.crt exists or not and then if not then use /etc/origin/master/ca-bundle.crt
Comment 4 Johnny Liu 2017-07-04 02:29:33 EDT
Verified this bug with openshift-ansible-3.6.132-1.git.0.0d0f54a.el7.noarch, and PASS.
# oc edit ds apiserver -n kube-service-catalog
<--snip-->
      - args:
        - --storage-type
        - etcd
        - --secure-port
        - "6443"
        - --etcd-servers
        - https://openshift-136.lab.sjc.redhat.com:4001
        - --etcd-cafile
        - /etc/origin/master/ca-bundle.crt
        - --etcd-certfile
        - /etc/origin/master/master.etcd-client.crt
        - --etcd-keyfile
        - /etc/origin/master/master.etcd-client.key
<--snip-->


# cat /etc/origin/master/master-config.yaml
<--snip-->
etcdClientInfo:
  ca: ca-bundle.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
  - https://openshift-136.lab.sjc.redhat.com:4001
etcdConfig:
  address: openshift-136.lab.sjc.redhat.com:4001
  peerAddress: openshift-136.lab.sjc.redhat.com:7001
  peerServingInfo:
    bindAddress: 0.0.0.0:7001
    certFile: etcd.server.crt
    clientCA: ca-bundle.crt
    keyFile: etcd.server.key
  servingInfo:
    bindAddress: 0.0.0.0:4001
    certFile: etcd.server.crt
    clientCA: ca-bundle.crt
    keyFile: etcd.server.key
  storageDirectory: /var/lib/origin/openshift.local.etcd
<--snip-->
Comment 6 errata-xmlrpc 2017-08-10 01:28:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.