Bug 1589063

Summary: Audit log not configured correctly in master yaml
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: InstallerAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED ERRATA QA Contact: Gaoyun Pei <gpei>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.10.0CC: anli, aos-bugs, byount, cshereme, denis, gpei, jmalde, jokerman, mmccomas, sgaikwad, vrutkovs
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-31 06:18:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gaoyun Pei 2018-06-08 09:50:39 UTC
Description of problem:
Set openshift_master_audit_config option in ansible inventory file, start ocp-3.10 installation.

openshift_master_audit_config={"enabled": true, "auditFilePath": "/var/log/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": 14, "maximumFileSizeMegabytes": 500, "maximumRetainedFiles": 5}


Control plane pods didn't come up during installation and the logs shows:
[root@qe-gpei-t3104master-etcd-1 ~]# master-logs controllers controllers
F0608 09:42:40.906693       1 start_controllers.go:67] could not load config file "/etc/origin/master/master-config.yaml" due to an error: error reading config: only encoded map or array can be decoded into a struct


Check /etc/origin/master/master-config.yaml

[root@qe-gpei-t3104master-etcd-1 ~]# grep auditConfig -A5 -B3 /etc/origin/master/master-config.yaml
apiLevels:
- v1
apiVersion: v1
auditConfig: '{"enabled": true, "auditFilePath": "/var/log/openpaas-oscp-audit/openpaas-oscp-audit.log",
  "maximumFileRetentionDays": 14, "maximumFileSizeMegabytes": 500, "maximumRetainedFiles":
  5}'
authConfig:
  requestHeader:
    clientCA: front-proxy-ca.crt



Version-Release number of the following components:
openshift-ansible-3.10.0-0.64.0.git.20.48df973.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Vadim Rutkovsky 2018-06-18 14:37:27 UTC
(In reply to Gaoyun Pei from comment #0)
> openshift_master_audit_config={"enabled": true, "auditFilePath":
> "/var/log/openpaas-oscp-audit/openpaas-oscp-audit.log",
> "maximumFileRetentionDays": 14, "maximumFileSizeMegabytes": 500,
> "maximumRetainedFiles": 5}

This is not a valid yaml string, so it gets pasted as a string.

All values need to be quoted - and the path should be in /var/lib/origin:

openshift_master_audit_config={"enabled": "true", "auditFilePath": "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": "14", "maximumFileSizeMegabytes": "500", "maximumRetainedFiles": "5"}

Created https://github.com/openshift/openshift-ansible/pull/8825 to fix hosts.examples

Please verify this works with updated setting

Comment 2 Gaoyun Pei 2018-06-19 09:48:35 UTC
Thanks for the update Vadim, I tried ocp-3.10 installation with the following option:
openshift_master_audit_config={"enabled": "true", "auditFilePath": "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": "14", "maximumFileSizeMegabytes": "500", "maximumRetainedFiles": "5"}

During installation, master api/controllers pod still couldn't get started. 

[root@ip-172-18-14-17 ~]# master-logs controllers controllers
F0619 09:35:32.644687       1 start_controllers.go:67] could not load config file "/etc/origin/master/master-config.yaml" due to an error: error reading config: [pos 844]: json: decode bool: got first char "
[root@ip-172-18-14-17 ~]# master-logs api api
F0619 09:35:46.189916       1 start_api.go:68] could not load config file "/etc/origin/master/master-config.yaml" due to an error: error reading config: [pos 844]: json: decode bool: got first char "


Audit config in master-config.yaml
[root@ip-172-18-14-17 ~]# grep -A6 auditConfig /etc/origin/master/master-config.yaml 
auditConfig:
  auditFilePath: /var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log
  enabled: 'true'
  maximumFileRetentionDays: '14'
  maximumFileSizeMegabytes: '500'
  maximumRetainedFiles: '5'
authConfig:


Installer used: openshift-ansible-3.10.1-1.git.157.2bb6250.el7.noarch.rpm

Comment 3 Vadim Rutkovsky 2018-06-19 15:34:37 UTC
Oops, I must have not cleaned existing master config when I tried that.

It seems "enabled": "true" always leaves quotes around 'true' - and that breaks controller.

The correct fix for this would be using YAML-style inventory. Setting :

openshift_master_audit_config:
  enabled: true
  auditFilePath: /var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log
  maximumFileRetentionDays: 5
  maximumFileSizeMegabytes: 500
  maximumRetainedFiles: 5


in inventory/group_vars/OSEv3 generates correct master-config.yaml.

Decreasing priority as there is a workaround for that.

Comment 7 Vadim Rutkovsky 2018-08-16 09:37:47 UTC
audit config was incorrectly converted, PR for master branch - https://github.com/openshift/openshift-ansible/pull/9620

Comment 8 Vadim Rutkovsky 2018-08-16 11:51:58 UTC
3.10 cherrypick - https://github.com/openshift/openshift-ansible/pull/9629

Comment 11 Gaoyun Pei 2018-08-22 03:28:20 UTC
Test with openshift-ansible-3.10.34-1.git.0.48df172None.noarch.rpm, master still fail to get started.

Audit config parameter in ansible inventory
openshift_master_audit_config={"enabled": "true", "auditFilePath": "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": "14", "maximumFileSizeMegabytes": "500", "maximumRetainedFiles": "5"}

[root@ip-172-18-1-138 ~]# master-logs api api
F0822 03:22:53.621619       1 start_api.go:68] could not load config file "/etc/origin/master/master-config.yaml" due to an error: error reading config: [pos 844]: json: decode bool: got first char "


[root@ip-172-18-1-138 ~]# grep -A5 auditConfig /etc/origin/master/master-config.yaml
auditConfig:
  auditFilePath: /var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log
  enabled: 'true'
  maximumFileRetentionDays: '14'
  maximumFileSizeMegabytes: '500'
  maximumRetainedFiles: '5'

Comment 12 Vadim Rutkovsky 2018-08-22 11:46:45 UTC
(In reply to Gaoyun Pei from comment #11)
> Test with openshift-ansible-3.10.34-1.git.0.48df172None.noarch.rpm, master
> still fail to get started.
> 
> Audit config parameter in ansible inventory
> openshift_master_audit_config={"enabled": "true", "auditFilePath":
> "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log",
> "maximumFileRetentionDays": "14", "maximumFileSizeMegabytes": "500",
> "maximumRetainedFiles": "5"}


It seems I was mistaken in comment #1 - quotes around booleans would actually break api server.

It seems to work with (no quotes around true in 'enabled')

openshift_master_audit_config={"enabled": true, "auditFilePath": "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": "14", "maximumFileSizeMegabytes": "500", "maximumRetainedFiles": "5"}

Could you re-verify that? Keeping this in ASSIGNED for now

Comment 13 Gaoyun Pei 2018-08-23 05:49:49 UTC
Yes, it works without quotes on true. Master service could run well.

openshift_master_audit_config={"enabled": true, "auditFilePath": "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log", "maximumFileRetentionDays": "14", "maximumFileSizeMegabytes": "500", "maximumRetainedFiles": "5"}


[root@ip-172-18-2-206 ~]# grep -A5 auditConfig /etc/origin/master/master-config.yaml
auditConfig:
  auditFilePath: /var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log
  enabled: true
  maximumFileRetentionDays: '14'
  maximumFileSizeMegabytes: '500'
  maximumRetainedFiles: '5'

Comment 14 Vadim Rutkovsky 2018-08-23 09:36:11 UTC
Thanks. Moving this to ON_QA so that other scenarios (if any) would be tested.

Comment 15 Gaoyun Pei 2018-08-27 03:20:45 UTC
Move this bug to verified according to Comment 13, bz#1622044 is tracking the 3.10 doc correction correspondingly.

Comment 21 errata-xmlrpc 2018-08-31 06:18:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2376