Bug 1920839

Created attachment 1751138 [details]
ocp4-7 config error when applying machine config

Created attachment 1751139 [details]
OCP 4-6 node service logs

This was working fine in ocp 4.6

Created attachment 1751141 [details]
OCP 4-6 node service logs in Kibana

This was working fine in ocp 4.6

The error appears to be in whatever your environment has $jrnl_cnf defined as. It appears it does not like something when that is expanded. 

Can you provide the output of

 echo $jnrl_cnf 

as well as the contents of the journal.conf file you created? I did not see either attached.

Hi,

please find below output of $jnrl-cnf and input file.
[root@bastion ~]# echo $jrnl_cnf
Q29tcHJlc3M9eWVzIApGb3J3YXJkVG9Db25zb2xlPW5vIApGb3J3YXJkVG9TeXNsb2c9bm8KTWF4UmV0ZW50aW9uU2VjPTFtb250aCAKUmF0ZUxpbWl0QnVyc3Q9MTAwMDAgClJhdGVMaW1pdEludGVydmFsPTFzClN0b3JhZ2U9cGVyc2lzdGVudCAKU3luY0ludGVydmFsU2VjPTFzIApTeXN0ZW1NYXhVc2U9OGcgClN5c3RlbUtlZXBGcmVlPTIwJSAKU3lzdGVtTWF4RmlsZVNpemU9MTBNIAo=
[root@bastion ~]#
[root@bastion ~]# cat /journald.conf
Compress=yes
ForwardToConsole=no
ForwardToSyslog=no
MaxRetentionSec=1month
RateLimitBurst=10000
RateLimitInterval=1s
Storage=persistent
SyncIntervalSec=1s
SystemMaxUse=8g
SystemKeepFree=20%
SystemMaxFileSize=10M
[root@bastion ~]#
[root@bastion ~]# cat machineconfiguration-worker.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-corp-journald
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,${jrnl_cnf}
        mode: 0644
        overwrite: true
        path: /etc/systemd/journald.conf
[root@bastion ~]# oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-7cf67c5854a43275b73735c04016803b   True      False      False      3              3                   3                     0                      14d
worker   rendered-worker-0026e514b5a56359c5a2827f360d4cbe   True      False      True       3              3                   3                     0                      14d


Thanks.

Created attachment 1751563 [details]
journald-setup-and-its-output

Hi Sanjaya, 

The issue is that the jrnl_cnf variable needs to be expanded in the yaml file before applying the machine config. If you follow this document you should be successful in making the change: https://docs.openshift.com/container-platform/4.6/post_installation_configuration/machine-configuration-tasks.html#machineconfig-modify-journald_post-install-machine-configuration-tasks - note the cat << EOF, which will create the file with the variable expanded. I've made a PR to update the documentation you linked to match this approach: https://github.com/openshift/openshift-docs/pull/

(In reply to Andy McCrae from comment #8)
> Hi Sanjaya, 
> 
> The issue is that the jrnl_cnf variable needs to be expanded in the yaml
> file before applying the machine config. If you follow this document you
> should be successful in making the change:
> https://docs.openshift.com/container-platform/4.6/
> post_installation_configuration/machine-configuration-tasks.
> html#machineconfig-modify-journald_post-install-machine-configuration-tasks
> - note the cat << EOF, which will create the file with the variable
> expanded. I've made a PR to update the documentation you linked to match
> this approach: https://github.com/openshift/openshift-docs/pull/

Thanks Andy, I've followed the above steps as per the attached doc, now the var jrnl_cnf able to expended using "EOF-here document" ,and MCP is working as expected. post that  the "journald.conf" is reflected on each worker nodes,  and able to collect kernel,daemon,service logs from worker nodes too.

#oc describe mc/40-worker-custom-journald
....
      Version:  3.1.0
    Networkd:
    Passwd:
    Storage:
      Files:
        Contents:
          Source:  data:text/plain;charset=utf-8;base64,Q29tcHJlc3M9eWVzCkZvcndhcmRUb0NvbnNvbGU9bm8KRm9yd2FyZFRvU3lzbG9nPW5vCk1heFJldGVudGlvblNlYz0xbW9udGgKUmF0ZUxpbWl0QnVyc3Q9MTAwMDAKUmF0ZUxpbWl0SW50ZXJ2YWw9MXMKU3RvcmFnZT1wZXJzaXN0ZW50ClN5bmNJbnRlcnZhbFNlYz0xcwpTeXN0ZW1NYXhVc2U9OGcKU3lzdGVtS2VlcEZyZWU9MjAlClN5c3RlbU1heEZpbGVTaXplPTEwTQo=
          Verification:
        Filesystem:  root
        Mode:        420
        Path:        /etc/systemd/journald.conf
    Systemd:
  Os Image URL:
Events:          <none>

Hi Andy, should this bug still be assigned to you since it's Documentation?

Hi Andy, following up - can we link your PR in Comment 8 to this bug or put a full path of the PR in a comment (currently the address is incomplete)

Apologies, I missed this update - and I didn't paste the full link accidentally.
I've tracked down the change so the full PR URL should be: https://github.com/openshift/openshift-docs/pull/28995 - which I've added as a link to the bz.