This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 2177684 - Libvirt virt-launcher pods logs containing "custom-ga-command" are flooding the virt-launcher pods logs
Summary: Libvirt virt-launcher pods logs containing "custom-ga-command" are flooding t...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 4.12.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.15.0
Assignee: sgott
QA Contact: Kedar Bidarkar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-13 11:43 UTC by Shirly Radco
Modified: 2023-12-14 16:09 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-12-14 16:09:57 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   CNV-26827 0 None None None 2023-12-14 16:09:56 UTC

Description Shirly Radco 2023-03-13 11:43:50 UTC 
Description of problem:
Libvirt virt-launcher pods logs containing "custom-ga-command" are flooding the virt-launcher pods logs.

Example of a log:
{"component":"virt-launcher","level":"warning","msg":"Domain id=1 name='alitke_fedora-blue-shark' uuid=b4cf836b-1677-5741-8069-da0cf7fe50d8 is tainted: custom-ga-command","pos":"qemuDomainObjTaintMsg:6376","subcomponent":"libvirt","thread":"33","timestamp":"2023-03-13T10:23:44.243000Z"}



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
These logs are sent for all virtual machines and are flooding the logs and are also not very helpful.

Expected results:
We need to check if these logs are valuable, why they are sent in such great quantity and if we should/can stop logging them.

Additional info:
Comment 1 sgott 2023-03-15 12:27:23 UTC 
The trouble is that these logs are almost always valuable when triaging issues. While it might be possible to perhaps move them to DEBUG level, the result is that they won't be there when they could have, turning them on would be a very high fricition process, and turning them on would provide exponentially more logs than necessary for normal purposes.

For those reasons we don't think it's a good idea to suppress them. Closing this BZ, but please feel free to re-open if you can think of a better way to deal with them.
Comment 2 Igor Bezukh 2023-03-20 08:48:00 UTC 
Hi,

The bug is only about the specific above-mentioned log with "custom-ga-command" .The log isn't self-explainable enough, and its being fired pretty often. Considering a large-scale setup of VMs, that message can indeed flood the user-defined logging stack.
My suggestion is at least to understand why this log is being fired often and why its level is warning
Comment 3 Igor Bezukh 2023-03-20 08:51:04 UTC 
One possible way to resolve it is to make it a k8s warning event and silence this log. I think it's better because repeated k8s events are not spamming the log, unlike this situation
Comment 4 Itamar Holder 2023-03-20 09:30:02 UTC 
In general, I think that libvirt logs are valuable, as Stu said above. We can definitely discuss which of them are important to always show, and which are better to show only under DEBUG level, but in general they are important.

I've once pushed a PR regarding libvirt logs which can be found here: https://github.com/kubevirt/kubevirt/pull/7000. I think this PR provides valuable information regarding the different logs and why they are being showed. As I've said, based on this PR, we can definitely discuss if we should filter some of them.

Please also note that labels such as "subcomponent":"libvirt", "pos":"qemuDomainObjTaintMsg:6376", etc are also important for better filtering logs, for example with logsviewer https://github.com/vladikr/logsviewer.

Regarding the specific log entry you showed in your example - it might very well be redundant.
Comment 5 Shirly Radco 2023-03-20 09:34:31 UTC 
I only meant the specific log that is stated in the bug description. Not libvirt logs in general.
Comment 6 Itamar Holder 2023-03-20 09:46:04 UTC 
Oh, alright, then I misunderstood.

Please clarify the BZ, as you wrote "Example of a log:" and "Libvirt virt-launcher pods related logs are flooding..." which imply that you refer to libvirt logs in general, not a specific log entry.
Comment 7 Shirly Radco 2023-03-20 10:42:15 UTC 
Updated the header and description to only refer to the logs that contain "custom-ga-command".
Comment 9 sgott 2023-03-22 13:13:46 UTC 
Given the attached screenshot, it appears you have the ability to filter logs. Given this, we're guessing the severity is low. It's unclear what the difficulty will be to address this programmatically.

Deferring this to 4.15 due to anticipated capacity.
Comment 10 Shirly Radco 2023-04-25 13:37:22 UTC 
This has both storage aspects and user experience impact.

I'll add the w/a for it here for now as well.

To get the virt-launcher logs without these logs use:
{ log_type=~".+", kubernetes_container_name="compute" } | json | !="custom-ga-command"
Note You need to log in before you can comment on or make changes to this bug.