Bug 1014942

1. Can you share the code running in the automated tests that reproduce this?
2. Is is possible that a previous test was not finished before the new test began? Do you run the test with a fresh storage domain or run several tests on the same domain one after another?

Created attachment 809892 [details]
vdsm logs from failed run and successful run

(In reply to Nir Soffer from comment #1)
> 1. Can you share the code running in the automated tests that reproduce this?
> 2. Is is possible that a previous test was not finished before the new test
> began? Do you run the test with a fresh storage domain or run several tests
> on the same domain one after another?
Storage domain is a new storage domain each time.

Attaching archive with a successful run and another failed run for comparison.

Can you provide /var/log/messages log for the duration of the test of both a successful and failed test?

Created attachment 809991 [details]
var/log/messages for failed and successful run

Added /var/log/messages for both a successful and a failed run

First, please complete the basic details:

RHEL version:
multipathd version:
lvm version:

Second, please run the same test with the previous release: IS 16.

Please run the same test with build IS 15; we have a theory that the failure is caused by change introduced in IS 16.

Nir, 

please try to ask for all missing info you need at once.

Like Gadi wrote, we added logs of successful run and another failed run for comparison.
we can't rerun as we have other things in our pipe. I am sure you can recheck it without us (you asked to run using is15 and then asked is16)

I agree that Gadi could attach the automation code, but you can understand the flow from the logs - also he mentioned what the test does.

we will try to do our best to help but based on our priorities.

(In reply to Aharon Canan from comment #7)

When your priorities allow and you run this test again, do enable lvm logging.

lvm default log level is 0, means no messages are logged even for critical errors. Please set lvm log level to 7 (LOG_DEBUG) while running this test.

To change lvm log level, edit /etc/lvm/lvm.conf, find the log section and edit level to 7:

log {
    ...
    level = 7
    ...
}

This bug is caused by lvm regresion in 6.5, see bug 1020401.

A fix for lvm bug 1020401 is in a qa now, and will be available in lvm2-2.02.100-7.el6.

Please test again when this version is available.

If you want to test now, you can edit /etc/lvm/lvm.conf and disable obtain_device_list_from_udev option:

devices {
    obtain_device_list_from_udev = 0
}

Patch was merge in upstream.

(In reply to Nir Soffer from comment #10)
> A fix for lvm bug 1020401 is in a qa now, and will be available in
> lvm2-2.02.100-7.el6.
This package is installed on our hosts and so far I have not seen this error.
> 
> Please test again when this version is available.
> 
> If you want to test now, you can edit /etc/lvm/lvm.conf and disable
> obtain_device_list_from_udev option:
> 
> devices {
>     obtain_device_list_from_udev = 0
> }

(In reply to Gadi Ickowicz from comment #12)
> (In reply to Nir Soffer from comment #10)
> > A fix for lvm bug 1020401 is in a qa now, and will be available in
> > lvm2-2.02.100-7.el6.
> This package is installed on our hosts and so far I have not seen this error.

It the depend on how you install the host. If you do a clean install of 6.5, then this issue should be resolve with any vdsm build. But if you install an older that includes lvm2 < 2.02.100-7, then your lvm.conf would not be updated with new lvm defaults, and the issue will be reproduced with current vdsm builds, but should be fixed with the next build containing the fix.

Do test machines upgraded from 6.4 to 6.5?

(In reply to Nir Soffer from comment #13)
> (In reply to Gadi Ickowicz from comment #12)
> > (In reply to Nir Soffer from comment #10)
> > > A fix for lvm bug 1020401 is in a qa now, and will be available in
> > > lvm2-2.02.100-7.el6.
> > This package is installed on our hosts and so far I have not seen this error.
> 
> It the depend on how you install the host. If you do a clean install of 6.5,
> then this issue should be resolve with any vdsm build. But if you install an
> older that includes lvm2 < 2.02.100-7, then your lvm.conf would not be
> updated with new lvm defaults, and the issue will be reproduced with current
> vdsm builds, but should be fixed with the next build containing the fix.
> 
> Do test machines upgraded from 6.4 to 6.5?

The machine is a clean install of 6.5

Unless you want to test machine upgraded from 6.4, you can close this bug.

Now when we included our own fix, we do not depend any more on bug 1020401.

verified on is22 - error did not show up in any nightly automation runs since is22

This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html