Bug 1351666 - virtlogd breaks apps that monitor guest console logs
Summary: virtlogd breaks apps that monitor guest console logs
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Retired
Component: tests
Version: 21
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: 23.2
Assignee: Jan Stancek
QA Contact: Dan Callaghan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-30 14:05 UTC by Jan Stancek
Modified: 2016-08-29 06:22 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1351209
Environment:
Last Closed: 2016-08-29 06:18:27 UTC
Embargoed:


Attachments (Terms of Use)

Description Jan Stancek 2016-06-30 14:05:50 UTC
+++ This bug was initially created as a clone of Bug #1351209 +++

Description of problem:
KVM guest is set up to redirect console output into a file, which is monitored by part of Beaker's /distribution/virt/install test, that uploads new chunks to Beaker server:

   <serial type='file'>
      <source path='/mnt/tests/distribution/virt/install/guests/ppc64le_kvm8_2cpu/logs/ppc64le_kvm8_2cpu_console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </serial>
    <console type='file'>
      <source path='/mnt/tests/distribution/virt/install/guests/ppc64le_kvm8_2cpu/logs/ppc64le_kvm8_2cpu_console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </console>

virtlogd starts rotating logs and any application monitoring original file is left with fd that is no longer getting updates:

-rw-------. 1 root root  24159 Jun 29 08:37 ppc64le_kvm8_2cpu_console.log
-rw-------. 1 root root 131072 Jun 29 07:19 ppc64le_kvm8_2cpu_console.log.0
-rwxrwxrwx. 1 qemu qemu 131072 Jun 29 07:01 ppc64le_kvm8_2cpu_console.log.1

# fuser `pwd`/ppc64le_kvm8_2cpu_console.log
/mnt/tests/distribution/virt/install/guests/ppc64le_kvm8_2cpu/logs/ppc64le_kvm8_2cpu_console.log:  6416

# fuser `pwd`/ppc64le_kvm8_2cpu_console.log.1
/mnt/tests/distribution/virt/install/guests/ppc64le_kvm8_2cpu/logs/ppc64le_kvm8_2cpu_console.log.1:  6217

# ps afx | grep -e 6416 -e 6217
113741 pts/2    S+     0:00          \_ grep --color=auto -e 6416 -e 6217
  6217 ?        Ss     0:05 /usr/local/bin/logguestconsoles --config /usr/local/etc/logguestconsoles.conf
  6416 ?        Ssl    0:02 /usr/sbin/virtlogd


Version-Release number of selected component (if applicable):
libvirt-1.3.5-1.el7

How reproducible:
100%

Steps to Reproduce:
1. redirect serial console to file
2. open file
3. let guest write more than 128kb to serial console
4. file is rotated, and fd open at 2) is not getting updates anymore

Actual results:
virtlogd breaks apps that monitor guest console logs

Expected results:
1. larger default value for DEFAULT_FILE_SIZE, say 2M
2. option to override DEFAULT_FILE_SIZE

Additional info:

--- Additional comment from RHEL Product and Program Management on 2016-06-29 10:25:47 EDT ---

Since this bug report was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-06-29 10:25:54 EDT ---

This bug report has Keywords: Regression or TestBlocker.

Since no regressions or test blockers are allowed between releases, it is also being [proposed|marked] as a blocker for this release.

Please resolve ASAP.

--- Additional comment from Daniel Berrange on 2016-06-30 05:29:23 EDT ---

You have the option of disabling virtlogd in /etc/libvirt/qemu.conf via the stdio_handler property.

Allowing the max size before rollover to be configurable is a valid future RFE too.

--- Additional comment from Jan Stancek on 2016-06-30 07:19:10 EDT ---

(In reply to Daniel Berrange from comment #3)
> You have the option of disabling virtlogd in /etc/libvirt/qemu.conf via the
> stdio_handler property.

Thanks for the tip, I'll look into creating a patch for beaker, that configures the old unlimited behaviour.

Comment 1 Jan Stancek 2016-06-30 14:10:43 UTC
Proposed patch posted for review:
 https://gerrit.beaker-project.org/#/c/5017/

Comment 2 Jan Stancek 2016-08-15 11:14:14 UTC
fyi, we had a virtlogd AVC issue, which has been fixed couple days ago:
  Bug 1358140 - virtlogd AVC prevents ppc KVM guests from starting
From what I understand, it shouldn't have effect on patch in comment 1.

libvirt version of this BZ introduced a parameter and bumped default size to 2M:
  Bug 1351209 - virtlogd breaks apps that monitor guest console logs
  http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-July/msg00242.html

which is certainly better, but we still need some kind of patch to make it "unlimited" to not break logguestconsoles (script from /distribution/virt/install), either by changing stdio_handler to file or by increasing new option "max_size" to some very large number.

Comment 3 Roman Joost 2016-08-16 00:46:01 UTC
Dear Jan,

just wondering if the patch can be merged and therefore included in the next release?

Comment 4 Jan Stancek 2016-08-16 10:15:24 UTC
(In reply to Roman Joost from comment #3)
> Dear Jan,
> 
> just wondering if the patch can be merged and therefore included in the next
> release?

I checked it still works fine with RHEL-7.3-20160815.n.0, so from my point of view it can be merged.

Comment 5 Dan Callaghan 2016-08-29 03:55:31 UTC
Tagged as /distribution/virt/install 4.0-92.


Note You need to log in before you can comment on or make changes to this bug.