Bug 696943

Created attachment 492336 [details]
anaconda.log got by virtio

Created attachment 492337 [details]
ifcfg.log got by virtio

Created attachment 492338 [details]
program.log got by virtio

Created attachment 492339 [details]
storage.log got by virtio

Created attachment 492340 [details]
syslog got by virtio

Created attachment 492341 [details]
X.log got by virtio

Created attachment 492342 [details]
original anaconda.log on guest

Created attachment 492344 [details]
original program.log on guest

Created attachment 492347 [details]
original storage.log on guest

Created attachment 492348 [details]
original syslog on guest

Created attachment 492349 [details]
original  xlog on guest

Created attachment 492350 [details]
original ifcfg.log on guest

some difference in anaconda.log 

there is record line in the original anaconda.log

moving (1) to step storagedone

but it cannot be found in the anaconda.log got by virtio.

Hi,

Please attach debug_all.log and debug_unknown.log from the analog output directory.

Btw,

Why is the version 14? Isn't this supposed to be 15 or rawhide?

Created attachment 492791 [details]
debug_all log

Created attachment 492792 [details]
debug unknown source log

It should be F15, the host is F14 x86_64

I think this is caused by this rsyslogd feature: http://www.rsyslog.com/how-to-use-rate-limiting-in-rsyslog/ which is turned on by default.

Patch 99f5ecf219c4668bd2b8c41499b504683b90a2ca on master is fixing this, please retest with analog from anaconda-16.6-1 once that becomes available or for a quick fix try adding 

$SystemLogRateLimitInterval 0

after 

$ModLoad imuxsock

in the generated rsyslogd file (you'll need to do that before rsyslogd starts though).

I have checked out the latest anaconda, I saw you have add $SystemLogRateLimitInterval 0 in it. but when I run 
eval `./analog -p 6080 -s -o .rsyslogd.conf /tmp/autoqa/log/`

it gives error in the debug logs, please find it in the attachments.

I have tried to add $SystemLogRateLimitInterval 0  into /etc/rsyslog.conf,
it also does not work.

Created attachment 492860 [details]
debug unknown source log after turn off rate limit

Created attachment 492861 [details]
debug all log after turn off rate limit

Created attachment 492862 [details]
rsyslogd.conf after turn off rate limit

Hm,

can you also try like this:

$ModLoad imtcp.so
$ModLoad imuxsock.so
$SystemLogRateLimitInterval 0
$OmitLocalLogging on
$InputTCPServerRun 6080

that is, swapping $OmitLocalLogging and $SystemLogRateLimitInterval?

Thanks.

It seems the situation is the same. even I add it to /etc/rsyslog.conf, restart the rsyslog service,  the same error found in /var/log/messages. Do you think it is a problem of rsyslog? Thanks.

Hi, Ales

I have contacted with Rainer, I was told the rsyslog version 4.6.3 does not support the new type of rate limiting.

Can you please retest with an f15/rawhide which should already have rsyslogd >= 5.7 ?

Thanks.

Hi, Ales,

I have tested f15 with rsyslog 5.7.9,  the error messages in attachment  https://bugzilla.redhat.com/attachment.cgi?id=492861  disappear, but the logs are still different.

(In reply to comment #28)
> Hi, Ales,
> 
> I have tested f15 with rsyslog 5.7.9,  the error messages in attachment 
> https://bugzilla.redhat.com/attachment.cgi?id=492861  disappear, but the logs
> are still different.

Please re-attach all the logs from the 5.7 attempt, can be as a single, reasonably structured zip file.

Created attachment 495104 [details]
anaconda logs, debug logs, ks.cfg, create_command.sh

(In reply to comment #30)
> Created attachment 495104 [details]
> anaconda logs, debug logs, ks.cfg, create_command.sh

That's interesting. Assuming you are using the latest 'analog' to generate the rsyslogd config which disables rate limiting via

$SystemLogRateLimitInterval 0

it looks like an rsyslogd bug to still have messages dropped as indicated in debug_unknown_source.log:

<46>2011-04-26T02:50:50.006142+00:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 364 due to rate-limiting
<46>2011-04-26T02:50:51.552511+00:00 localhost rsyslogd-2177: imuxsock lost 458 messages from pid 364 due to rate-limiting
<46>2011-04-26T02:56:26.284391+00:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 364 due to rate-limiting

I will talk to the rsyslogd maintainer about this later today.

The rsyslogd maintainer explained that this is caused by the guest side dropping messages. Rawhide fix in the rsyslogd config will follow. 

Hongqing, I don't think we will be able to get this to F15, it is hard to imagine the issue as a blocker.

(In reply to comment #32)
> Hongqing, I don't think we will be able to get this to F15, it is hard to
> imagine the issue as a blocker.

Depending on the scope of the fix, since this is really helpful for our test automation, I think this would fit as a nice-to-have for F15.

In master lorax this is fixed by:
http://git.fedorahosted.org/git/?p=lorax.git;a=commitdiff;h=7a77fdc7abefe71aa7c427890ff716e87a95ebc2

Will cherry-pick to f15 if this (low risk) patch becomes an accepted NTH.

Discussed at 2011-04-29 NTH review meeting, accepted NTH per comments #33 and #34.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Fixed by 78709fe0291b9a9568fc193a350cd3716a6f05a1 on the f15-branch.

(In reply to comment #36)
> Fixed by 78709fe0291b9a9568fc193a350cd3716a6f05a1 on the f15-branch.

We still need a build submitted for this so it can land in F15.

(In reply to comment #37)
> (In reply to comment #36)
> > Fixed by 78709fe0291b9a9568fc193a350cd3716a6f05a1 on the f15-branch.
> 
> We still need a build submitted for this so it can land in F15.

Many thanks to mcepl who submitted a bodhi update for this issue

https://admin.fedoraproject.org/updates/lorax-0.4.4-1.fc15

lorax-0.4.4-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/lorax-0.4.4-1.fc15

Package lorax-0.4.4-1.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing lorax-0.4.4-1.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/lorax-0.4.4-1.fc15
then log in and leave karma (feedback).

lorax-0.4.4-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

it works well with rsyslog 5.7.9-2.fc15.x86_64 and lorax 0.4.5.1.fc15.x86_64