Bug 483578

Summary: LVM2 problem with VIRTIO devices on Kernels 2.6.27.xxx
Product: [Fedora] Fedora Reporter: Eric Goirand <eric.goirand>
Component: lvm2Assignee: Peter Rajnoha <prajnoha>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: low    
Version: 10CC: agk, bmarzins, bmr, dwysocha, heinzm, lvm-team, mbroz, prajnoha, prockai
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-12-18 07:46:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
log of vgchange -ay -vvvv vg_data_01
none
dmesg extract after vgchange -ay -vvvv vg_data_01
none
lvmdump after vgchange -ay -vvvv vg_data_01 none

Description Eric Goirand 2009-02-02 14:56:30 UTC
Description of problem:

LVM is not working correctly when :
 - I use a KVM virtual machine
 - the kernel is 2.6.27.xxx (both on Fedora 9 and Fedora 10)
 - 2 disks are defined by virtio ==> vda and vdb
 - vda is the system disk with 1 PV, 1 VG and 1 LV for /
 - vdb is the data disk with 1 PV, 1 VG and 1 LV for data

Version-Release number of selected component (if applicable):


How reproducible:

All the time

Steps to Reproduce:
1. Install a virtual machine only 1 disk using virtio ==> vda and let's configure it with :
  - vda1 : ext3 = /boot
  - vda2 : physical volume, with 1 volume group (vg_sys_01) with 1 LV (lv_f10)
2. Create a new disk with dd and add it to the virtual machine configuration as a virtio disk ==> vdb
3. Boot the new configuration virtual machine and set up the second disk
  - create 1 partition vdb1 with ID 8e (Linux LVM)
  - create the PV on vdb1
  - create the VG on vdb1 (vg_data_01)
  - create the LV on vdb1 (lv_data)
4. When you try to format /dev/vg_data_01/lv_data it does not work, as vgchange -ay vg_data_01 gets KILLED.

Any time you reboot your virtual machine, if you do vgscan ; vgchange -ay then you have the KILLED message and /dev/vg_data_01/lv_data is never accessible

I didn't have the problem with kernel 2.6.26 on Fedora 9, but since kernel 2.6.27, I always have this problem in Fedora 9 and Fedora 10.
  
Actual results:

It is not possible to use any LV from the second VG on the second virtio disk


Expected results:

vgchange -ay should not get killed, and we could access all the LV on any VG

Additional info:

Comment 1 Peter Rajnoha 2009-02-04 13:35:34 UTC
I tried to reproduce this bug with the configuration described above, but it seems to be working for me on current Fedora 10 installation (kvm 74-5, kernel 2.6.27.5, lvm 2.02.39). No error occurs.

Could you please post exact LVM commands that you run and attach verbose output of the command that gets killed if it's possible (for example output of 'vgchange -vvvv -ay vg_data_01'). Thanks.

Comment 2 Eric Goirand 2009-02-05 10:21:20 UTC
Created attachment 330978 [details]
log of vgchange -ay -vvvv vg_data_01

Comment 3 Eric Goirand 2009-02-05 10:26:09 UTC
I put in attachment file 20090205-log-vgchange-bug-lvm.txt that corresponds to the output of the command : vgchange -ay -vvvv vg_data_01 before it gets killed.
I used version lvm2-2.02.39-7.fc10.x86_64 and tried with kernel-2.6.27.5-117.fc10.x86_64 but it does the same with all 2.6.27 kernels for me on both Fedora 9 and Fedora 10.

Comment 4 Eric Goirand 2009-02-05 10:31:19 UTC
Sorry, I forgot the kvm version : I use kvm-74-10.fc10.x86_64 on Fedora 10 and I use kvm-65-10.fc9.x86_64 on Fedora 9.

Comment 5 Peter Rajnoha 2009-02-05 11:41:59 UTC
Can you spot any relevant information written by syslog that is related to the actions you make (...vgchange getting killed), e.g. try to check /var/log/messages or try to send a tail of this file after running the 'vgchange' command. Also, try to send a dump by running the 'lvmdump' command...

Comment 6 Eric Goirand 2009-02-05 16:47:43 UTC
Created attachment 331030 [details]
dmesg extract after vgchange -ay -vvvv vg_data_01

Comment 7 Eric Goirand 2009-02-05 16:48:31 UTC
Created attachment 331031 [details]
lvmdump after vgchange -ay -vvvv vg_data_01

Comment 8 Milan Broz 2009-02-06 12:01:04 UTC
lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Pid: 1058, comm: lvm Not tainted 2.6.27.5-117.fc10.x86_64 #1
...
According the log, lvm triggers OOM and lvm is killed.
There is 256M RAM, it should be enough...

You said that it works with 2.6.26 kernel - are you sure that the lvm2 version was exactly the same?
IOW with the lvm2-2.02.39-7.fc10.x86_64 and some 2.6.26 kernel it works correctly?

Comment 9 Eric Goirand 2009-02-06 12:16:38 UTC
It is working correctly only on Fedora 9 with kvm kvm-65-10.fc9.x86_64 and kernel kernel-2.6.26.6-79.fc9.x86_64 with lvm version lvm2-2.02.33-11.fc9.x86_64

Comment 10 Eric Goirand 2009-03-09 09:17:50 UTC
I made a test by increasing the memory of the virtual machine from 256MiB to 384Mib and I no longer have the problem both in Fedora 9 and Fedora 10.

It seems that 256Mib RAM is not enough with kernel 2.6.27.xxx and LVM.

Best regards,
Eric.

Comment 11 Peter Rajnoha 2009-03-09 09:46:13 UTC
Yes, I got the same results when I tested it. However I was able to run it with less memory (about 128 MB was enough, giving less resulted in oom killer activation, this was tested in 32-bit environment). I tested the configuration on old kernels as well (2.6.25, 2.6.23...) and the memory needed was only about a half of that in 2.6.27. So more investigation is still needed to find out the cause of this relatively huge increase in memory consumption.

Comment 12 Peter Rajnoha 2009-03-09 12:09:49 UTC
Also, I noticed that this does not seem to depend on whether virtio is used or not -- the same problem appears on non-virtio devices, too. The problem does not appear with the exact same system (fresh install of Fedora 10) but using different virtualization tool (with different HW emulated). So, I'm a little bit suspicious that this rather depends on the (virtual) hardware configuration and the cause of the problem lies in here.

Comment 13 Bug Zapper 2009-11-18 09:28:08 UTC
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 14 Bug Zapper 2009-12-18 07:46:57 UTC
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.