Bug 1718150 - openvswtich logrotate is missing due to sdn container change
Summary: openvswtich logrotate is missing due to sdn container change
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.11.z
Assignee: Phil Cameron
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-07 05:56 UTC by Takayoshi Kimura
Modified: 2019-07-23 19:56 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Consequence: Fix: don't write to host filesystem. Log to stdout/stderr in container. Result:
Clone Of:
Environment:
Last Closed: 2019-07-23 19:56:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1753 0 None None None 2019-07-23 19:56:33 UTC

Description Takayoshi Kimura 2019-06-07 05:56:58 UTC
Description of problem:

In 3.9 /etc/logrotate.d/openvswitch was effective. In 3.10+ this no longer work because the logrotate configuration file sits inside sdn pod filesystem and logrotate is working at the container host side.

Version-Release number of selected component (if applicable):

OpenShift 3.11

How reproducible:

Always

Steps to Reproduce:
1. n/a
2.
3.

Actual results:

No log rotations

Expected results:

Logs in /var/log/openvswitch/ are rotated

Additional info:

Workaround is to copy the /etc/logrotate.d/openvswitch file from the sdn pod to container hosts.

Comment 1 Takayoshi Kimura 2019-06-07 06:30:25 UTC
> Workaround is to copy the /etc/logrotate.d/openvswitch file from the sdn pod to container hosts.

Unfortunately this won't work as the ovs-appctl command is not available at the container host side. Installing openvswitch on the container host side may lead some conflict issue, so I'm not sure if it's feasible option.

Comment 2 Casey Callendrello 2019-06-07 12:28:01 UTC
Is this a bug? Openshift-sdn doesn't write any logs to /var/log/ on the host?

Comment 3 Takayoshi Kimura 2019-06-08 01:13:30 UTC
sdn/ovs pods openvswitch writes logs on the host via hostmount:

 $ ls -la /var/log/openvswitch/
total 2968
drwxr-xr-x.  2 root root      54 Jan 28 16:35 .
drwxr-xr-x. 20 root root    4096 Jun  4 03:34 ..
-rw-r-----.  1 root root   34526 Jun  3 12:47 ovsdb-server.log
-rw-r-----.  1 root root 2993478 Jun  8 03:30 ovs-vswitchd.log

          volumeMounts:
          - mountPath: /var/log/openvswitch
            name: log-openvswitch

Comment 4 Casey Callendrello 2019-06-11 11:43:46 UTC
Oh of course it does, sorry for missing that. In openshift 4, we removed the log file completely and just use kubectl logs. Let me talk to the team and find the correct solution.

Comment 5 Casey Callendrello 2019-06-11 11:47:37 UTC
To Phil. Maybe he can think of something.

Comment 6 Takayoshi Kimura 2019-06-12 01:34:39 UTC
Customer provided workaround logrotate config /etc/logrotate.d/openshift-ovs:

/var/log/openvswitch/*.log {
    daily
    compress
    sharedscripts
    missingok
    postrotate
        # Tell Open vSwitch daemons to reopen their log files
        if [ -d /var/run/openvswitch ]; then
            OVS_CONTAINER_ID=$(docker ps -f name=k8s_openvswitch_ovs --format "{{.ID}}")
            for ctl in /var/run/openvswitch/*.ctl; do
                docker exec $OVS_CONTAINER_ID ovs-appctl -t "$ctl" vlog/reopen 2>/dev/null || :
            done
        fi
    endscript
}

It uses ovs container via docker. Likely doesn't work for cri-o setup.

Comment 7 Phil Cameron 2019-07-09 17:39:17 UTC
https://github.com/openshift/openshift-ansible/pull/11702 - MERGED
openvswtich logrotate is missing due to sdn container change

Fix is to not write to host filesystem.

Fix also cleans up existing host system ovs log files.

Comment 9 zhaozhanqi 2019-07-11 05:54:16 UTC
Verified this bug on 3.11.128 with openshift-ansible-3.11.128-1.git.0.85b789b.el7.noarch

oc rsh -n openshift-sdn ovs-s2c5p
sh-4.2# ls /var/log/openvswitch
ovs-vswitchd.log  ovsdb-server.log
sh-4.2# ls /var/log/openvswitch-old/
/var/log/openvswitch is exist on node
exit

[root@sdn311-master ~]# ls /var/log/openvswitch/
[root@sdn311-master ~]#

Comment 11 errata-xmlrpc 2019-07-23 19:56:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1753


Note You need to log in before you can comment on or make changes to this bug.