Bug 1420618 - [intservice_public_324]Deploy logging stack with --check flag, failed to generate signing.conf
Summary: [intservice_public_324]Deploy logging stack with --check flag, failed to gene...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 3.5.z
Assignee: Jeff Cantrill
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-09 05:41 UTC by Junqi Zhao
Modified: 2017-10-25 13:00 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-10-25 13:00:48 UTC
Target Upstream Version:


Attachments (Terms of Use)
full ansible running log (139.49 KB, text/plain)
2017-02-09 05:41 UTC, Junqi Zhao
no flags Details
ansible log, not support check mode (14.05 KB, text/plain)
2017-03-10 01:56 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3049 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update 2017-10-25 15:57:15 UTC

Description Junqi Zhao 2017-02-09 05:41:02 UTC
Created attachment 1248734 [details]
full ansible running log

Description of problem:
Deployed logging stack via ansible with --check flag, failed to generate signing.conf.
If without --check flag, deployment can be successful and can generate signing.conf.
It's a regression defect.


Version-Release number of selected component (if applicable):
# openshift version
openshift v3.5.0.18+9a5d1aa
kubernetes v1.5.2+43a9be4
etcd 3.1.0


Image id:
openshift3/logging-elasticsearch    d715f4d34ad4
openshift3/logging-kibana    e0ab09c2cbeb
openshift3/logging-fluentd    47057624ecab
openshift3/logging-auth-proxy    139f7943475e
openshift3/logging-curator    7f034fdf7702

How reproducible:
Always


Steps to Reproduce:
1. prepare the inventory file

[oo_first_master]
$master-public-dns ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="~/cfile/libra.pem" openshift_public_hostname=$master-public-dns

[oo_first_master:vars]
deployment_type=openshift-enterprise
openshift_release=v3.5.0
openshift_logging_install_logging=true

openshift_logging_kibana_hostname=kibana.$sub-domain
public_master_url=https://$master-public-dns:8443
openshift_logging_fluentd_hosts=$node

openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/
openshift_logging_image_version=3.5.0

openshift_logging_namespace=juzhao
openshift_logging_fluentd_use_journal=true


2. Running the playbook from a control machine (my laptop) which is not oo_master:
git clone https://github.com/openshift/openshift-ansible
ansible-playbook -vvv -i ~/inventory   playbooks/common/openshift-cluster/openshift_logging.yml

Actual results:
Can not find signing.conf, see the trace:

TASK [openshift_logging : Sign cert request with CA for system.logging.fluentd] 
task path: /home/fedora/openshift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:30
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<ec2-54-86-165-237.compute-1.amazonaws.com> ESTABLISH SSH CONNECTION FOR USER: root
<ec2-54-86-165-237.compute-1.amazonaws.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/fedora/Downloads/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r ec2-54-86-165-237.compute-1.amazonaws.com '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
fatal: [ec2-54-86-165-237.compute-1.amazonaws.com]: FAILED! => {
    "changed": true, 
    "cmd": [
        "openssl", 
        "ca", 
        "-in", 
        "/etc/origin/logging/system.logging.fluentd.csr", 
        "-notext", 
        "-out", 
        "/etc/origin/logging/system.logging.fluentd.crt", 
        "-config", 
        "/etc/origin/logging/signing.conf", 
        "-extensions", 
        "v3_req", 
        "-batch", 
        "-extensions", 
        "server_ext"
    ], 
    "delta": "0:00:00.006133", 
    "end": "2017-02-08 22:52:09.814993", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "openssl ca -in /etc/origin/logging/system.logging.fluentd.csr -notext -out /etc/origin/logging/system.logging.fluentd.crt -config /etc/origin/logging/signing.conf -extensions v3_req -batch -extensions server_ext", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-02-08 22:52:09.808860", 
    "warnings": []
}

STDERR:

Using configuration from /etc/origin/logging/signing.conf
error loading the config file '/etc/origin/logging/signing.conf'
139728809564064:error:02001002:system library:fopen:No such file or directory:bss_file.c:169:fopen('/etc/origin/logging/signing.conf','rb')
139728809564064:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:172:
139728809564064:error:0E078072:configuration file routines:DEF_LOAD:no such file:conf_def.c:197:


PLAY RECAP *********************************************************************
ec2-54-86-165-237.compute-1.amazonaws.com : ok=57   changed=7    unreachable=0    failed=1  


search for signing.conf, it is not generated.

# ls -al /etc/origin/logging/
total 40
drwxr-xr-x. 2 root root 4096 Feb  9 00:20 .
drwx------. 7 root root 4096 Feb  9 00:19 ..
-rw-r--r--. 1 root root 1050 Feb  9 00:19 ca.crt
-rw-r--r--. 1 root root    0 Feb  9 00:20 ca.crt.srl
-rw-r--r--. 1 root root    0 Feb  9 00:19 ca.db
-rw-------. 1 root root 1679 Feb  9 00:19 ca.key
-rw-r--r--. 1 root root    2 Feb  9 00:19 ca.serial.txt
-rw-r--r--. 1 root root 2242 Feb  9 00:19 kibana-internal.crt
-rw-------. 1 root root 1679 Feb  9 00:19 kibana-internal.key
-rw-r--r--. 1 root root  321 Feb  9 00:19 server-tls.json
-rw-r--r--. 1 root root  960 Feb  9 00:20 system.logging.fluentd.csr
-rw-r--r--. 1 root root 1704 Feb  9 00:20 system.logging.fluentd.key


Expected results:
Deployment can be successful.

Additional info:
Attached full ansible log

Comment 1 Junqi Zhao 2017-02-09 08:58:12 UTC
correct step 2:
run ansible with the following command:

ansible-playbook --check -i ~/inventory -vvv playbooks/common/openshift-cluster/openshift_logging.yml

Comment 3 Junqi Zhao 2017-02-20 01:38:50 UTC
Jeff,

In card, we see the following messages:
https://trello.com/c/zjOqOXET/324-2-8-logging-administration-via-ansible-logging-epic-ois-agl-exp-ops-rfe


The administration actions and tasks should:

* allow for a dry run


If we decide to fix this defect in the UpcomingRelease, I don't think we can mark this card as 'accepted'. 

What is your opinion?

Comment 5 Junqi Zhao 2017-02-21 05:38:55 UTC
Remove Regression keyword, this defect is not one regression issue

Comment 6 Xia Zhao 2017-02-21 05:43:54 UTC
@juzhao @sdodson Adding back the keyword Regression since I've seen the --check flag worked fine without this issue in the early days when this card is newly completed: https://trello.com/c/zjOqOXET/324-2-8-logging-administration-via-ansible-logging-epic-ois-agl-exp-ops-rfe, we did support it then.

Comment 8 Jeff Cantrill 2017-03-09 17:06:18 UTC
@Junqi can you retest this issue.  I have used '--check' with recent versions of openshift-ansible and not seen errors.

Comment 9 Junqi Zhao 2017-03-10 01:55:42 UTC
@Jeff, 
Tested with command 'ansible-playbook --check -vvv -i $INVENTORY_FILE  playbooks/common/openshift-cluster/openshift_logging.yml' ($INVENTORY_FILE is inventory file)

Log shows "remote module (command) does not support check mode".
It seems check mode is not supported, as Scott mentioned in Comment 4.

See the attached ansible log.

Our openshift-ansible and playbooks are yum installed. 
version:
openshift-ansible-3.5.25-1.git.0.a40beae.el7.noarch
openshift-ansible-playbooks-3.5.25-1.git.0.a40beae.el7.noarch

# ansible --version
ansible 2.2.1.0

Comment 10 Junqi Zhao 2017-03-10 01:56:51 UTC
Created attachment 1261764 [details]
ansible log, not support check mode

Comment 12 Jeff Cantrill 2017-03-10 14:03:17 UTC
Can we close this bug since check mode is not supported and the original issue looks to have been resolved.

Comment 13 Junqi Zhao 2017-03-13 00:43:07 UTC
OK, it can be closed.

Comment 14 Jeff Cantrill 2017-03-13 13:43:02 UTC
Moving to ON_QA to place into verified since the original issue was fixed as part of installer work

Comment 15 Junqi Zhao 2017-03-14 00:23:35 UTC
Remote module (command) does not support check mode. without --check flag, signing.conf can generated successfully.

Close it as VERIFIED.

Comment 18 Junqi Zhao 2017-03-29 00:21:05 UTC
got it, thanks

Comment 20 errata-xmlrpc 2017-10-25 13:00:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049


Note You need to log in before you can comment on or make changes to this bug.