Bug 1697507

Summary: Forward logging to haproxy.log file task failing in dry-run mode
Product: Red Hat OpenStack Reporter: Victor Voronkov <vvoronko>
Component: openstack-tripleo-heat-templatesAssignee: Cédric Jeanneret <cjeanner>
Status: CLOSED ERRATA QA Contact: Victor Voronkov <vvoronko>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: aschultz, bperkins, cjeanner, emacchi, jschluet, mburns, mcornea, shdunne, slinaber
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190513171729.9995be9.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-02 20:08:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Victor Voronkov 2019-04-08 14:29:27 UTC
Description of problem:
Running deployment in dry run mode (only validating config files without actually deploying, performed by running ansible playbooks in check mode) fails while trying to run Forward logging to haproxy.log file task, here the log:

TASK [Forward logging to haproxy.log file] *************************************
fatal: [controller-2]: FAILED! => {"msg": "The conditional check 'rsyslog_config.rc == 0' failed. The error was: error while evaluating conditional (rsyslog_config.rc == 0): 'dict object' has no attribute 'rc'\n\nThe error appears to have been in '/home/stack/config-download/Controller/host_prep_tasks.yaml': line 479, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n- block:\n  - blockinfile:\n    ^ here\n"}

Version-Release number of selected component (if applicable):


How reproducible:
Run overcloud deployment with dry-run flag

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Cédric Jeanneret 2019-04-09 06:13:57 UTC
Hello Victor,

It probably don't hit just Rocky, Master should also get it. A patch is provided for Master right now, and will be backported to Rocky once merged.

I'll do some more tests on my own in parallel.

Cheers,

C.

Comment 4 Marius Cornea 2019-04-12 17:31:08 UTC
Issue is still present:

puddle id - 2019-04-12.1
openstack-tripleo-heat-templates.noarch 0:9.3.1-0.20190314162756.d0a6cb1.el7ost 

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/rfe/job/DFG-df-rfe-14-virsh-3cont_2comp_3ceph-dry-run-verification/19/artifact/.sh/verification-run.log

Comment 5 Victor Voronkov 2019-04-13 03:36:55 UTC
Solution that worked for sure for previous bug [1] was to disable the whole block where variable was in use [2]

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1684864
[2] https://review.openstack.org/#/c/639402/1/roles/tripleo-ssh-known-hosts/tasks/main.yml

Comment 6 Cédric Jeanneret 2019-04-15 11:40:38 UTC
Hello,

This is the "right" solution (at least, it's working): https://review.openstack.org/#/c/652621/

I've tested it with a simple playbook:
- hosts: localhost
  tasks:
    - name: check command
      shell: systemctl is-active rsyslog
      register: rsyslog_config
    - name: push output
      when:
        - rsyslog_config is changed
        - rsyslog_config.rc == 0
      debug:
        var: rsyslog_config

When we run it with --check:
PLAY [localhost] ******************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [check command] **************************************************************************************************************************************************************************************************************************
skipping: [localhost]

TASK [push output] ****************************************************************************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0

And with standard:
PLAY [localhost] ******************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [check command] **************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [push output] ****************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "rsyslog_config": {
        "changed": true, 
        "cmd": "systemctl is-active rsyslog", 
        "delta": "0:00:00.006942", 
        "end": "2019-04-15 13:31:58.830151", 
        "failed": false, 
        "rc": 0, 
        "start": "2019-04-15 13:31:58.823209", 
        "stderr": "", 
        "stderr_lines": [], 
        "stdout": "active", 
        "stdout_lines": [
            "active"
        ]
    }
}

PLAY RECAP ************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0

Have to make it merge, then backport it into rocky, and finally downstream.

Comment 8 Cédric Jeanneret 2019-04-16 13:05:09 UTC
New package should sort the issue. Sorry for the delay :(

Comment 12 Victor Voronkov 2019-06-04 13:25:13 UTC
"TASK [Forward logging to haproxy.log file] *************************************", "fatal: [controller-2]: FAILED! => {\"msg\": \"The conditional check 'rsyslog_config.rc == 0' failed. The error was: error while evaluating conditional (rsyslog_config.rc == 0): 'dict object' has no attribute 'rc'\\n\\nThe error appears to have been in '/home/stack/config-download/Controller/host_prep_tasks.yaml': line 479, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n  - blockinfile:\\n    ^ here\\n\"}", "fatal: [controller-1]: FAILED! => {\"msg\": \"The conditional check 'rsyslog_config.rc == 0' failed. The error was: error while evaluating conditional (rsyslog_config.rc == 0): 'dict object' has no attribute 'rc'\\n\\nThe error appears to have been in '/home/stack/config-download/Controller/host_prep_tasks.yaml': line 479, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n  - blockinfile:\\n    ^ here\\n\"}", "fatal: [controller-0]: FAILED! => {\"msg\": \"The conditional check 'rsyslog_config.rc == 0' failed. The error was: error while evaluating conditional (rsyslog_config.rc == 0): 'dict object' has no attribute 'rc'\\n\\nThe error appears to have been in '/home/stack/config-download/Controller/host_prep_tasks.yaml': line 479, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n  - blockinfile:\\n    ^ here\\n\"}",


core_puddle 2019-05-31.1

openstack-tripleo-heat-templates-9.3.1-0.20190513171725.9995be9.el7ost.noarch

The fix should be in the compose, but it's missing:

- name: Check if rsyslog exists
  register: rsyslog_config
  shell: systemctl is-active rsyslog
- block:
  - blockinfile:
      content: 'if $syslogfacility-text == ''{{facility}}'' and $programname == ''haproxy''
        then -/var/log/containers/haproxy/haproxy.log

        & stop

        '
      create: true
      path: /etc/rsyslog.d/openstack-haproxy.conf
    name: Forward logging to haproxy.log file
    register: logconfig
    vars:
      facility: local0
  - name: restart rsyslog service after logging conf change
    service:
      name: rsyslog
      state: restarted
    when: logconfig is changed
  when: rsyslog_config.rc == 0
- file:

Comment 13 Cédric Jeanneret 2019-06-07 19:40:00 UTC
Hello,

so the backport for pacemaker wasn't done for rocky - the stein one took a long time and got merged only last week - since I was off it wasn't taken down to rocky.

I've just pushed https://review.opendev.org/#/c/664046/ - hopefully it will be merged next week upstream, and I'll backport it now downstream.

Sorry for all those delays.

Comment 14 Cédric Jeanneret 2019-06-07 19:43:56 UTC
(added downstream backport for osp-14)

Comment 16 Victor Voronkov 2019-06-12 15:55:55 UTC
Verified on compose 2019-06-10.3

Comment 19 errata-xmlrpc 2019-07-02 20:08:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1672