This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 696943 - logs got by virtio are different from the originals on guest
logs got by virtio are different from the originals on guest
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: lorax (Show other bugs)
15
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Ales Kozumplik
Fedora Extras Quality Assurance
AcceptedNTH
:
Depends On:
Blocks: F15-accepted/F15FinalFreezeExcept
  Show dependency treegraph
 
Reported: 2011-04-15 07:17 EDT by Hongqing Yang
Modified: 2014-09-30 19:40 EDT (History)
8 users (show)

See Also:
Fixed In Version: lorax-0.4.4-1.fc15
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-05 01:00:36 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
anaconda.log got by virtio (32.77 KB, text/plain)
2011-04-15 07:21 EDT, Hongqing Yang
no flags Details
ifcfg.log got by virtio (1.35 KB, text/plain)
2011-04-15 07:21 EDT, Hongqing Yang
no flags Details
program.log got by virtio (20.96 KB, text/plain)
2011-04-15 07:22 EDT, Hongqing Yang
no flags Details
storage.log got by virtio (73.24 KB, text/plain)
2011-04-15 07:22 EDT, Hongqing Yang
no flags Details
syslog got by virtio (51.98 KB, text/plain)
2011-04-15 07:23 EDT, Hongqing Yang
no flags Details
X.log got by virtio (25.14 KB, text/plain)
2011-04-15 07:24 EDT, Hongqing Yang
no flags Details
original anaconda.log on guest (34.85 KB, text/plain)
2011-04-15 07:33 EDT, Hongqing Yang
no flags Details
original program.log on guest (23.78 KB, text/plain)
2011-04-15 07:34 EDT, Hongqing Yang
no flags Details
original storage.log on guest (117.46 KB, text/plain)
2011-04-15 07:35 EDT, Hongqing Yang
no flags Details
original syslog on guest (51.82 KB, text/plain)
2011-04-15 07:35 EDT, Hongqing Yang
no flags Details
original xlog on guest (24.62 KB, text/plain)
2011-04-15 07:36 EDT, Hongqing Yang
no flags Details
original ifcfg.log on guest (1.29 KB, text/plain)
2011-04-15 07:36 EDT, Hongqing Yang
no flags Details
debug_all log (168.81 KB, text/plain)
2011-04-17 22:57 EDT, Hongqing Yang
no flags Details
debug unknown source log (381 bytes, text/plain)
2011-04-17 22:58 EDT, Hongqing Yang
no flags Details
debug unknown source log after turn off rate limit (253 bytes, text/plain)
2011-04-18 07:26 EDT, Hongqing Yang
no flags Details
debug all log after turn off rate limit (464 bytes, text/plain)
2011-04-18 07:27 EDT, Hongqing Yang
no flags Details
rsyslogd.conf after turn off rate limit (2.85 KB, text/plain)
2011-04-18 07:29 EDT, Hongqing Yang
no flags Details
anaconda logs, debug logs, ks.cfg, create_command.sh (86.97 KB, application/zip)
2011-04-26 23:05 EDT, Hongqing Yang
no flags Details

  None (edit)
Description Hongqing Yang 2011-04-15 07:17:04 EDT
Description of problem:

The logs got by virtio are different from the originals on the guest.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. start rsyslog service
2. run analog
3. create a guest
4. edit guest xml to add channel
5. restart guest
  
Actual results:


Expected results:


Additional info:
Comment 1 Hongqing Yang 2011-04-15 07:21:08 EDT
Created attachment 492336 [details]
anaconda.log got by virtio
Comment 2 Hongqing Yang 2011-04-15 07:21:44 EDT
Created attachment 492337 [details]
ifcfg.log got by virtio
Comment 3 Hongqing Yang 2011-04-15 07:22:13 EDT
Created attachment 492338 [details]
program.log got by virtio
Comment 4 Hongqing Yang 2011-04-15 07:22:52 EDT
Created attachment 492339 [details]
storage.log got by virtio
Comment 5 Hongqing Yang 2011-04-15 07:23:35 EDT
Created attachment 492340 [details]
syslog got by virtio
Comment 6 Hongqing Yang 2011-04-15 07:24:45 EDT
Created attachment 492341 [details]
X.log got by virtio
Comment 7 Hongqing Yang 2011-04-15 07:33:33 EDT
Created attachment 492342 [details]
original anaconda.log on guest
Comment 8 Hongqing Yang 2011-04-15 07:34:13 EDT
Created attachment 492344 [details]
original program.log on guest
Comment 9 Hongqing Yang 2011-04-15 07:35:12 EDT
Created attachment 492347 [details]
original storage.log on guest
Comment 10 Hongqing Yang 2011-04-15 07:35:43 EDT
Created attachment 492348 [details]
original syslog on guest
Comment 11 Hongqing Yang 2011-04-15 07:36:18 EDT
Created attachment 492349 [details]
original  xlog on guest
Comment 12 Hongqing Yang 2011-04-15 07:36:52 EDT
Created attachment 492350 [details]
original ifcfg.log on guest
Comment 13 Hongqing Yang 2011-04-15 07:40:54 EDT
some difference in anaconda.log 

there is record line in the original anaconda.log

moving (1) to step storagedone

but it cannot be found in the anaconda.log got by virtio.
Comment 14 Ales Kozumplik 2011-04-15 09:24:58 EDT
Hi,

Please attach debug_all.log and debug_unknown.log from the analog output directory.
Comment 15 Ales Kozumplik 2011-04-15 09:26:17 EDT
Btw,

Why is the version 14? Isn't this supposed to be 15 or rawhide?
Comment 16 Hongqing Yang 2011-04-17 22:57:26 EDT
Created attachment 492791 [details]
debug_all log
Comment 17 Hongqing Yang 2011-04-17 22:58:27 EDT
Created attachment 492792 [details]
debug unknown source log
Comment 18 Hongqing Yang 2011-04-17 22:59:16 EDT
It should be F15, the host is F14 x86_64
Comment 19 Ales Kozumplik 2011-04-18 03:55:22 EDT
I think this is caused by this rsyslogd feature: http://www.rsyslog.com/how-to-use-rate-limiting-in-rsyslog/ which is turned on by default.

Patch 99f5ecf219c4668bd2b8c41499b504683b90a2ca on master is fixing this, please retest with analog from anaconda-16.6-1 once that becomes available or for a quick fix try adding 

$SystemLogRateLimitInterval 0

after 

$ModLoad imuxsock

in the generated rsyslogd file (you'll need to do that before rsyslogd starts though).
Comment 20 Hongqing Yang 2011-04-18 07:24:54 EDT
I have checked out the latest anaconda, I saw you have add $SystemLogRateLimitInterval 0 in it. but when I run 
eval `./analog -p 6080 -s -o .rsyslogd.conf /tmp/autoqa/log/`

it gives error in the debug logs, please find it in the attachments.

I have tried to add $SystemLogRateLimitInterval 0  into /etc/rsyslog.conf,
it also does not work.
Comment 21 Hongqing Yang 2011-04-18 07:26:37 EDT
Created attachment 492860 [details]
debug unknown source log after turn off rate limit
Comment 22 Hongqing Yang 2011-04-18 07:27:09 EDT
Created attachment 492861 [details]
debug all log after turn off rate limit
Comment 23 Hongqing Yang 2011-04-18 07:29:09 EDT
Created attachment 492862 [details]
rsyslogd.conf after turn off rate limit
Comment 24 Ales Kozumplik 2011-04-18 09:09:55 EDT
Hm,

can you also try like this:

$ModLoad imtcp.so
$ModLoad imuxsock.so
$SystemLogRateLimitInterval 0
$OmitLocalLogging on
$InputTCPServerRun 6080

that is, swapping $OmitLocalLogging and $SystemLogRateLimitInterval?

Thanks.
Comment 25 Hongqing Yang 2011-04-18 23:02:32 EDT
It seems the situation is the same. even I add it to /etc/rsyslog.conf, restart the rsyslog service,  the same error found in /var/log/messages. Do you think it is a problem of rsyslog? Thanks.
Comment 26 Hongqing Yang 2011-04-26 02:19:29 EDT
Hi, Ales

I have contacted with Rainer, I was told the rsyslog version 4.6.3 does not support the new type of rate limiting.
Comment 27 Ales Kozumplik 2011-04-26 04:32:11 EDT
Can you please retest with an f15/rawhide which should already have rsyslogd >= 5.7 ?

Thanks.
Comment 28 Hongqing Yang 2011-04-26 07:16:41 EDT
Hi, Ales,

I have tested f15 with rsyslog 5.7.9,  the error messages in attachment  https://bugzilla.redhat.com/attachment.cgi?id=492861  disappear, but the logs are still different.
Comment 29 Ales Kozumplik 2011-04-26 07:25:40 EDT
(In reply to comment #28)
> Hi, Ales,
> 
> I have tested f15 with rsyslog 5.7.9,  the error messages in attachment 
> https://bugzilla.redhat.com/attachment.cgi?id=492861  disappear, but the logs
> are still different.

Please re-attach all the logs from the 5.7 attempt, can be as a single, reasonably structured zip file.
Comment 30 Hongqing Yang 2011-04-26 23:05:07 EDT
Created attachment 495104 [details]
anaconda logs, debug logs, ks.cfg, create_command.sh
Comment 31 Ales Kozumplik 2011-04-27 02:43:43 EDT
(In reply to comment #30)
> Created attachment 495104 [details]
> anaconda logs, debug logs, ks.cfg, create_command.sh

That's interesting. Assuming you are using the latest 'analog' to generate the rsyslogd config which disables rate limiting via

$SystemLogRateLimitInterval 0

it looks like an rsyslogd bug to still have messages dropped as indicated in debug_unknown_source.log:

<46>2011-04-26T02:50:50.006142+00:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 364 due to rate-limiting
<46>2011-04-26T02:50:51.552511+00:00 localhost rsyslogd-2177: imuxsock lost 458 messages from pid 364 due to rate-limiting
<46>2011-04-26T02:56:26.284391+00:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 364 due to rate-limiting

I will talk to the rsyslogd maintainer about this later today.
Comment 32 Ales Kozumplik 2011-04-27 07:57:10 EDT
The rsyslogd maintainer explained that this is caused by the guest side dropping messages. Rawhide fix in the rsyslogd config will follow. 

Hongqing, I don't think we will be able to get this to F15, it is hard to imagine the issue as a blocker.
Comment 33 James Laska 2011-04-27 13:14:32 EDT
(In reply to comment #32)
> Hongqing, I don't think we will be able to get this to F15, it is hard to
> imagine the issue as a blocker.

Depending on the scope of the fix, since this is really helpful for our test automation, I think this would fit as a nice-to-have for F15.
Comment 34 Ales Kozumplik 2011-04-28 03:32:36 EDT
In master lorax this is fixed by:
http://git.fedorahosted.org/git/?p=lorax.git;a=commitdiff;h=7a77fdc7abefe71aa7c427890ff716e87a95ebc2

Will cherry-pick to f15 if this (low risk) patch becomes an accepted NTH.
Comment 35 Adam Williamson 2011-04-29 14:32:55 EDT
Discussed at 2011-04-29 NTH review meeting, accepted NTH per comments #33 and #34.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 36 Ales Kozumplik 2011-05-02 02:55:50 EDT
Fixed by 78709fe0291b9a9568fc193a350cd3716a6f05a1 on the f15-branch.
Comment 37 James Laska 2011-05-02 15:21:57 EDT
(In reply to comment #36)
> Fixed by 78709fe0291b9a9568fc193a350cd3716a6f05a1 on the f15-branch.

We still need a build submitted for this so it can land in F15.
Comment 38 James Laska 2011-05-02 15:59:10 EDT
(In reply to comment #37)
> (In reply to comment #36)
> > Fixed by 78709fe0291b9a9568fc193a350cd3716a6f05a1 on the f15-branch.
> 
> We still need a build submitted for this so it can land in F15.

Many thanks to mcepl who submitted a bodhi update for this issue

https://admin.fedoraproject.org/updates/lorax-0.4.4-1.fc15
Comment 39 Fedora Update System 2011-05-02 16:02:02 EDT
lorax-0.4.4-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/lorax-0.4.4-1.fc15
Comment 40 Fedora Update System 2011-05-03 00:28:34 EDT
Package lorax-0.4.4-1.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing lorax-0.4.4-1.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/lorax-0.4.4-1.fc15
then log in and leave karma (feedback).
Comment 41 Fedora Update System 2011-05-05 01:00:29 EDT
lorax-0.4.4-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 42 Hongqing Yang 2011-05-09 01:54:55 EDT
it works well with rsyslog 5.7.9-2.fc15.x86_64 and lorax 0.4.5.1.fc15.x86_64

Note You need to log in before you can comment on or make changes to this bug.