Red Hat Bugzilla – Bug 483578
LVM2 problem with VIRTIO devices on Kernels 2.6.27.xxx
Last modified: 2009-12-18 02:46:57 EST
Description of problem:
LVM is not working correctly when :
- I use a KVM virtual machine
- the kernel is 2.6.27.xxx (both on Fedora 9 and Fedora 10)
- 2 disks are defined by virtio ==> vda and vdb
- vda is the system disk with 1 PV, 1 VG and 1 LV for /
- vdb is the data disk with 1 PV, 1 VG and 1 LV for data
Version-Release number of selected component (if applicable):
All the time
Steps to Reproduce:
1. Install a virtual machine only 1 disk using virtio ==> vda and let's configure it with :
- vda1 : ext3 = /boot
- vda2 : physical volume, with 1 volume group (vg_sys_01) with 1 LV (lv_f10)
2. Create a new disk with dd and add it to the virtual machine configuration as a virtio disk ==> vdb
3. Boot the new configuration virtual machine and set up the second disk
- create 1 partition vdb1 with ID 8e (Linux LVM)
- create the PV on vdb1
- create the VG on vdb1 (vg_data_01)
- create the LV on vdb1 (lv_data)
4. When you try to format /dev/vg_data_01/lv_data it does not work, as vgchange -ay vg_data_01 gets KILLED.
Any time you reboot your virtual machine, if you do vgscan ; vgchange -ay then you have the KILLED message and /dev/vg_data_01/lv_data is never accessible
I didn't have the problem with kernel 2.6.26 on Fedora 9, but since kernel 2.6.27, I always have this problem in Fedora 9 and Fedora 10.
It is not possible to use any LV from the second VG on the second virtio disk
vgchange -ay should not get killed, and we could access all the LV on any VG
I tried to reproduce this bug with the configuration described above, but it seems to be working for me on current Fedora 10 installation (kvm 74-5, kernel 184.108.40.206, lvm 2.02.39). No error occurs.
Could you please post exact LVM commands that you run and attach verbose output of the command that gets killed if it's possible (for example output of 'vgchange -vvvv -ay vg_data_01'). Thanks.
Created attachment 330978 [details]
log of vgchange -ay -vvvv vg_data_01
I put in attachment file 20090205-log-vgchange-bug-lvm.txt that corresponds to the output of the command : vgchange -ay -vvvv vg_data_01 before it gets killed.
I used version lvm2-2.02.39-7.fc10.x86_64 and tried with kernel-220.127.116.11-117.fc10.x86_64 but it does the same with all 2.6.27 kernels for me on both Fedora 9 and Fedora 10.
Sorry, I forgot the kvm version : I use kvm-74-10.fc10.x86_64 on Fedora 10 and I use kvm-65-10.fc9.x86_64 on Fedora 9.
Can you spot any relevant information written by syslog that is related to the actions you make (...vgchange getting killed), e.g. try to check /var/log/messages or try to send a tail of this file after running the 'vgchange' command. Also, try to send a dump by running the 'lvmdump' command...
Created attachment 331030 [details]
dmesg extract after vgchange -ay -vvvv vg_data_01
Created attachment 331031 [details]
lvmdump after vgchange -ay -vvvv vg_data_01
lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Pid: 1058, comm: lvm Not tainted 18.104.22.168-117.fc10.x86_64 #1
According the log, lvm triggers OOM and lvm is killed.
There is 256M RAM, it should be enough...
You said that it works with 2.6.26 kernel - are you sure that the lvm2 version was exactly the same?
IOW with the lvm2-2.02.39-7.fc10.x86_64 and some 2.6.26 kernel it works correctly?
It is working correctly only on Fedora 9 with kvm kvm-65-10.fc9.x86_64 and kernel kernel-22.214.171.124-79.fc9.x86_64 with lvm version lvm2-2.02.33-11.fc9.x86_64
I made a test by increasing the memory of the virtual machine from 256MiB to 384Mib and I no longer have the problem both in Fedora 9 and Fedora 10.
It seems that 256Mib RAM is not enough with kernel 2.6.27.xxx and LVM.
Yes, I got the same results when I tested it. However I was able to run it with less memory (about 128 MB was enough, giving less resulted in oom killer activation, this was tested in 32-bit environment). I tested the configuration on old kernels as well (2.6.25, 2.6.23...) and the memory needed was only about a half of that in 2.6.27. So more investigation is still needed to find out the cause of this relatively huge increase in memory consumption.
Also, I noticed that this does not seem to depend on whether virtio is used or not -- the same problem appears on non-virtio devices, too. The problem does not appear with the exact same system (fresh install of Fedora 10) but using different virtualization tool (with different HW emulated). So, I'm a little bit suspicious that this rather depends on the (virtual) hardware configuration and the cause of the problem lies in here.
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '10'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 10's end of life.
Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 10 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
The process we are following is described here:
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version.
Thank you for reporting this bug and we are sorry it could not be fixed.