Bug 892458

(In reply to comment #0)
> Description of problem:,
> boot a guest on rhel7 host with IDE/virtio_blk/virtio-scsi data disk, but
> fail to format it in guest using mkfs.ext4, but can write to the data disk
> in guest via 'dd'.
> BTW, 
> 1.if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can
> format the data disk successfully in guest.
> 2.If i just dd a disk in rhel7 local host, then also can format it
> successfully.
> 
> Version-Release number of selected component (if applicable):
> host info:
> # uname -r && rpm -q qemu-kvm
> 3.6.0-0.29.el7.x86_64
> qemu-kvm-1.3.0-3.el7.x86_64
> guest info:
> rhel7:  kernel-3.6.0-0.29.el7.x86_64
> rhel6.4:kernel-2.6.32-351.el6.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1.boot a guest on rhel7 host with IDE/virtio_blk/virtio-scsi data disk.
> e.g:...-drive
> file=/home/my-data-disk.qcow2,if=none,id=drive-data-disk,format=qcow2,
> cache=none,werror=stop,rerror=stop -device
> virtio-scsi-pci,id=scsi0,vectors=0,bus=pci.0,addr=0x7 -device
> scsi-hd,drive=drive-data-disk,bus=scsi0.0,id=data-disk
> 2.format the data disk in guest.
> # mkfs.ext4 /dev/sda
>   
> Actual results:
> after step 2, it will fail to format it with '/dev/sda is apparently in use
> by the system', and look at mount or fuser /dev/sda resulted in no
> information the drive wasn't mounted anywhere or in use.

So, you are sure sda is not in use as the root filesystem of the guest?

> # mkfs.ext4 /dev/sda
> mke2fs 1.42.6 (21-Sep-2012)
> /dev/sda is entire device, not just one partition!
> Proceed anyway? (y,n) y
> /dev/sda is apparently in use by the system; will not make a filesystem here!
> 
> Expected results:
> it can format the data disk successfully without any error.
> 
> Additional info:

(In reply to comment #4)
> (In reply to comment #0)
> > Description of problem:,
> > boot a guest on rhel7 host with IDE/virtio_blk/virtio-scsi data disk, but
> > fail to format it in guest using mkfs.ext4, but can write to the data disk
> > in guest via 'dd'.
> > BTW, 
> > 1.if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can
> > format the data disk successfully in guest.
> > 2.If i just dd a disk in rhel7 local host, then also can format it
> > successfully.
> > 
> > Version-Release number of selected component (if applicable):
> > host info:
> > # uname -r && rpm -q qemu-kvm
> > 3.6.0-0.29.el7.x86_64
> > qemu-kvm-1.3.0-3.el7.x86_64
> > guest info:
> > rhel7:  kernel-3.6.0-0.29.el7.x86_64
> > rhel6.4:kernel-2.6.32-351.el6.x86_64
> > 
> > How reproducible:
> > 100%
> > 
> > Steps to Reproduce:
> > 1.boot a guest on rhel7 host with IDE/virtio_blk/virtio-scsi data disk.
> > e.g:...-drive
> > file=/home/my-data-disk.qcow2,if=none,id=drive-data-disk,format=qcow2,
> > cache=none,werror=stop,rerror=stop -device
> > virtio-scsi-pci,id=scsi0,vectors=0,bus=pci.0,addr=0x7 -device
> > scsi-hd,drive=drive-data-disk,bus=scsi0.0,id=data-disk
> > 2.format the data disk in guest.
> > # mkfs.ext4 /dev/sda
> >   
> > Actual results:
> > after step 2, it will fail to format it with '/dev/sda is apparently in use
> > by the system', and look at mount or fuser /dev/sda resulted in no
> > information the drive wasn't mounted anywhere or in use.
> 
> So, you are sure sda is not in use as the root filesystem of the guest?
> 
Yes, I have looked at mount or fuser /dev/sda resulted in no information the drive wasn't mounted anywhere or in use.
> > # mkfs.ext4 /dev/sda
> > mke2fs 1.42.6 (21-Sep-2012)
> > /dev/sda is entire device, not just one partition!
> > Proceed anyway? (y,n) y
> > /dev/sda is apparently in use by the system; will not make a filesystem here!
> > 
> > Expected results:
> > it can format the data disk successfully without any error.
> > 
> > Additional info:

Hi,

    I tried it on kernel-3.8.0-0.40.el7.x86_64 and qemu-kvm-1.4.0-1.el7.x86_64, it can format(both the mkfs.ext4 and mkfs.ext3) the data disk successfully without any error.

Host info:
kernel-3.8.0-0.40.el7.x86_64
qemu-kvm-1.4.0-1.el7.x86_64
guest info:
kernel-3.8.0-0.40.el7.x86_64

qemu-kvm command line:
e.g:...-drive file=/home/my-data-disk.qcow2,if=none,id=drive-data-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,vectors=0,bus=pci.0,addr=0x8 -device scsi-hd,drive=drive-data-disk,bus=scsi0.0,id=data-disk

Results:
# mkfs.ext4 /dev/sda
mke2fs 1.42.6 (21-Sep-2012)
/dev/sda is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done 

# mkfs.ext3 /dev/sda
mke2fs 1.42.6 (21-Sep-2012)
/dev/sda is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Retried the kernel-3.6.0-0.29.el7.x86_64 and qemu-kvm-1.3.0-3.el7.x86_64, it still hit it.

host info:
kernel-3.6.0-0.29.el7.x86_64
qemu-kvm-1.3.0-3.el7.x86_64
guest info:
rhel7:  kernel-3.6.0-0.29.el7.x86_64

Results:
# mkfs.ext4 /dev/sda
mke2fs 1.42.6 (21-Sep-2012)
/dev/sda is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sda is apparently in use by the system; will not make a filesystem here!
# mkfs.ext3 /dev/sda
mke2fs 1.42.6 (21-Sep-2012)
/dev/sda is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sda is apparently in use by the system; will not make a filesystem here!
# mkfs.xfs /dev/sda
mkfs.xfs: cannot open /dev/sda: Device or resource busy

# mkfs /dev/sda
mke2fs 1.42.6 (21-Sep-2012)
/dev/sda is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sda is apparently in use by the system; will not make a filesystem here!

Here is the e2fsprogs code that determines whether the device is busy, lib/ext2fs/ismounted.c:ext2fs_check_mount_point():
int fd = open(device, O_RDONLY | O_EXCL);

if (fd >= 0)
        close(fd);
else if (errno == EBUSY)
        *mount_flags |= EXT2_MF_BUSY;

It seems that the device has already been opened with O_EXCL somewhere else.  This is probably a guest userspace issue or maybe a guest kernel issue.

> if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can format the data disk successfully in guest.

This sentence is ambiguous so I'm not sure if I'm interpreting it correctly:
 * RHEL7 guest fails
 * RHEL6.4 guest passes

You could set /sys/block/sdb/events_poll_msecs to "0" so that in-kernel disk polling is disabled.  Perhaps there is a bug in RHEL7 which is causing the in-kernel polling mechanism to claim the disk for a long time.

You could try a vanilla kernel with RHEL7 userspace to check if the RHEL7 guest kernel is broken.

You could install the RHEL7 iso on a physical machine and check if the problem happens there too.

(In reply to comment #10)
> Here is the e2fsprogs code that determines whether the device is busy,
> lib/ext2fs/ismounted.c:ext2fs_check_mount_point():
> int fd = open(device, O_RDONLY | O_EXCL);
> 
> if (fd >= 0)
>         close(fd);
> else if (errno == EBUSY)
>         *mount_flags |= EXT2_MF_BUSY;
> 
> It seems that the device has already been opened with O_EXCL somewhere else.
> This is probably a guest userspace issue or maybe a guest kernel issue.
> 
> > if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can format the data disk successfully in guest.
> 
> This sentence is ambiguous so I'm not sure if I'm interpreting it correctly:
>  * RHEL7 guest fails
>  * RHEL6.4 guest passes
> 
> You could set /sys/block/sdb/events_poll_msecs to "0" so that in-kernel disk
> polling is disabled.  Perhaps there is a bug in RHEL7 which is causing the
> in-kernel polling mechanism to claim the disk for a long time.
the /sys/block/sdb/events_poll_msecs default value is -1 in rhel7.

> You could try a vanilla kernel with RHEL7 userspace to check if the RHEL7
> guest kernel is broken.
> 
> You could install the RHEL7 iso on a physical machine and check if the
> problem happens there too.
yes, I tried it as your indication in the rhel7, it also hit this issue no matter i set the /sys/block/sdb/events_poll_msecs to "0" or not.

# cat /sys/block/sdb/events_poll_msecs
-1
# mkfs.ext4 /dev/sdb
mke2fs 1.42.6 (21-Sep-2012)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdb is apparently in use by the system; will not make a filesystem here!
# mkfs.ext3 /dev/sdb
mke2fs 1.42.6 (21-Sep-2012)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdb is apparently in use by the system; will not make a filesystem here!
# mkfs.xfs /dev/sdb
mkfs.xfs: cannot open /dev/sdb: Device or resource busy

# echo 0 > /sys/block/sdb/events_poll_msecs
# cat /sys/block/sdb/events_poll_msecs
0
# mkfs.ext4 /dev/sdb
mke2fs 1.42.6 (21-Sep-2012)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdb is apparently in use by the system; will not make a filesystem here!
# mkfs.ext3 /dev/sdb
mke2fs 1.42.6 (21-Sep-2012)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdb is apparently in use by the system; will not make a filesystem here!
# mkfs.xfs /dev/sdb
mkfs.xfs: cannot open /dev/sdb: Device or resource busy

(In reply to comment #11)
> (In reply to comment #10)
> > Here is the e2fsprogs code that determines whether the device is busy,
> > lib/ext2fs/ismounted.c:ext2fs_check_mount_point():
> > int fd = open(device, O_RDONLY | O_EXCL);
> > 
> > if (fd >= 0)
> >         close(fd);
> > else if (errno == EBUSY)
> >         *mount_flags |= EXT2_MF_BUSY;
> > 
> > It seems that the device has already been opened with O_EXCL somewhere else.
> > This is probably a guest userspace issue or maybe a guest kernel issue.
> > 
> > > if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can format the data disk successfully in guest.
> > 
> > This sentence is ambiguous so I'm not sure if I'm interpreting it correctly:
> >  * RHEL7 guest fails
> >  * RHEL6.4 guest passes
> > 
> > You could set /sys/block/sdb/events_poll_msecs to "0" so that in-kernel disk
> > polling is disabled.  Perhaps there is a bug in RHEL7 which is causing the
> > in-kernel polling mechanism to claim the disk for a long time.
> the /sys/block/sdb/events_poll_msecs default value is -1 in rhel7.
> 
> > You could try a vanilla kernel with RHEL7 userspace to check if the RHEL7
> > guest kernel is broken.
> > 
> > You could install the RHEL7 iso on a physical machine and check if the
> > problem happens there too.
> yes, I tried it as your indication in the rhel7, it also hit this issue no
> matter i set the /sys/block/sdb/events_poll_msecs to "0" or not.

Okay, this means the in-kernel polling mechanism is not the problem.

It's not clear yet whether this is a RHEL7 guest bug or a misconfiguration.

Please confirm that no process has the device open by running the following as root:
lsof -n | grep '\/dev\/sdb'

(In reply to comment #13)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > Here is the e2fsprogs code that determines whether the device is busy,
> > > lib/ext2fs/ismounted.c:ext2fs_check_mount_point():
> > > int fd = open(device, O_RDONLY | O_EXCL);
> > > 
> > > if (fd >= 0)
> > >         close(fd);
> > > else if (errno == EBUSY)
> > >         *mount_flags |= EXT2_MF_BUSY;
> > > 
> > > It seems that the device has already been opened with O_EXCL somewhere else.
> > > This is probably a guest userspace issue or maybe a guest kernel issue.
> > > 
> > > > if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can format the data disk successfully in guest.
> > > 
> > > This sentence is ambiguous so I'm not sure if I'm interpreting it correctly:
> > >  * RHEL7 guest fails
> > >  * RHEL6.4 guest passes
> > > 
> > > You could set /sys/block/sdb/events_poll_msecs to "0" so that in-kernel disk
> > > polling is disabled.  Perhaps there is a bug in RHEL7 which is causing the
> > > in-kernel polling mechanism to claim the disk for a long time.
> > the /sys/block/sdb/events_poll_msecs default value is -1 in rhel7.
> > 
> > > You could try a vanilla kernel with RHEL7 userspace to check if the RHEL7
> > > guest kernel is broken.
> > > 
> > > You could install the RHEL7 iso on a physical machine and check if the
> > > problem happens there too.
> > yes, I tried it as your indication in the rhel7, it also hit this issue no
> > matter i set the /sys/block/sdb/events_poll_msecs to "0" or not.
> 
> Okay, this means the in-kernel polling mechanism is not the problem.
> 
> It's not clear yet whether this is a RHEL7 guest bug or a misconfiguration.
> 
> Please confirm that no process has the device open by running the following
> as root:
> lsof -n | grep '\/dev\/sdb'

# lsof -n | grep '\/dev\/sdb'
multipath   543         root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543  556    root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543  559    root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543  560    root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543  811    root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543  812    root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543  813    root    9r      BLK               8,16       0t0      26359 /dev/sdb
multipath   543 2278    root    9r      BLK               8,16       0t0      26359 /dev/sdb

(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #11)
> > > (In reply to comment #10)
> > > > Here is the e2fsprogs code that determines whether the device is busy,
> > > > lib/ext2fs/ismounted.c:ext2fs_check_mount_point():
> > > > int fd = open(device, O_RDONLY | O_EXCL);
> > > > 
> > > > if (fd >= 0)
> > > >         close(fd);
> > > > else if (errno == EBUSY)
> > > >         *mount_flags |= EXT2_MF_BUSY;
> > > > 
> > > > It seems that the device has already been opened with O_EXCL somewhere else.
> > > > This is probably a guest userspace issue or maybe a guest kernel issue.
> > > > 
> > > > > if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can format the data disk successfully in guest.
> > > > 
> > > > This sentence is ambiguous so I'm not sure if I'm interpreting it correctly:
> > > >  * RHEL7 guest fails
> > > >  * RHEL6.4 guest passes
> > > > 
> > > > You could set /sys/block/sdb/events_poll_msecs to "0" so that in-kernel disk
> > > > polling is disabled.  Perhaps there is a bug in RHEL7 which is causing the
> > > > in-kernel polling mechanism to claim the disk for a long time.
> > > the /sys/block/sdb/events_poll_msecs default value is -1 in rhel7.
> > > 
> > > > You could try a vanilla kernel with RHEL7 userspace to check if the RHEL7
> > > > guest kernel is broken.
> > > > 
> > > > You could install the RHEL7 iso on a physical machine and check if the
> > > > problem happens there too.
> > > yes, I tried it as your indication in the rhel7, it also hit this issue no
> > > matter i set the /sys/block/sdb/events_poll_msecs to "0" or not.
> > 
> > Okay, this means the in-kernel polling mechanism is not the problem.
> > 
> > It's not clear yet whether this is a RHEL7 guest bug or a misconfiguration.
> > 
> > Please confirm that no process has the device open by running the following
> > as root:
> > lsof -n | grep '\/dev\/sdb'
> 
> # lsof -n | grep '\/dev\/sdb'
> multipath   543         root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543  556    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543  559    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543  560    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543  811    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543  812    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543  813    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb
> multipath   543 2278    root    9r      BLK               8,16       0t0    
> 26359 /dev/sdb

Great.  Is it possible that you have a multipath setup involving /dev/sdb?

Please try the following:
# service multipathd stop
# multipath -F
# lsof -n | grep '\/dev\/sdb'

You should now see no processes that have /dev/sdb open.

If you don't want to use /dev/sdb in a multipath configuration, please check /etc/multipath.conf to see why the device is being claimed.  For more information about multipath, see:

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/DM_Multipath/index.html

If you *do* want to use /dev/sdb in a multipath configuration, then you should not access /dev/sdb directly.  Instead, run mkfs on the device-mapper device that multipath created.

(In reply to comment #15)
> (In reply to comment #14)
> > (In reply to comment #13)
> > > (In reply to comment #11)
> > > > (In reply to comment #10)
> > > > > Here is the e2fsprogs code that determines whether the device is busy,
> > > > > lib/ext2fs/ismounted.c:ext2fs_check_mount_point():
> > > > > int fd = open(device, O_RDONLY | O_EXCL);
> > > > > 
> > > > > if (fd >= 0)
> > > > >         close(fd);
> > > > > else if (errno == EBUSY)
> > > > >         *mount_flags |= EXT2_MF_BUSY;
> > > > > 
> > > > > It seems that the device has already been opened with O_EXCL somewhere else.
> > > > > This is probably a guest userspace issue or maybe a guest kernel issue.
> > > > > 
> > > > > > if i boot guest in rhel6.4 host with the same qemu-kvm commands, then can format the data disk successfully in guest.
> > > > > 
> > > > > This sentence is ambiguous so I'm not sure if I'm interpreting it correctly:
> > > > >  * RHEL7 guest fails
> > > > >  * RHEL6.4 guest passes
> > > > > 
> > > > > You could set /sys/block/sdb/events_poll_msecs to "0" so that in-kernel disk
> > > > > polling is disabled.  Perhaps there is a bug in RHEL7 which is causing the
> > > > > in-kernel polling mechanism to claim the disk for a long time.
> > > > the /sys/block/sdb/events_poll_msecs default value is -1 in rhel7.
> > > > 
> > > > > You could try a vanilla kernel with RHEL7 userspace to check if the RHEL7
> > > > > guest kernel is broken.
> > > > > 
> > > > > You could install the RHEL7 iso on a physical machine and check if the
> > > > > problem happens there too.
> > > > yes, I tried it as your indication in the rhel7, it also hit this issue no
> > > > matter i set the /sys/block/sdb/events_poll_msecs to "0" or not.
> > > 
> > > Okay, this means the in-kernel polling mechanism is not the problem.
> > > 
> > > It's not clear yet whether this is a RHEL7 guest bug or a misconfiguration.
> > > 
> > > Please confirm that no process has the device open by running the following
> > > as root:
> > > lsof -n | grep '\/dev\/sdb'
> > 
> > # lsof -n | grep '\/dev\/sdb'
> > multipath   543         root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543  556    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543  559    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543  560    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543  811    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543  812    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543  813    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> > multipath   543 2278    root    9r      BLK               8,16       0t0    
> > 26359 /dev/sdb
> 
> Great.  Is it possible that you have a multipath setup involving /dev/sdb?
> 
> Please try the following:
> # service multipathd stop
> # multipath -F
> # lsof -n | grep '\/dev\/sdb'
> 
> You should now see no processes that have /dev/sdb open.
> 
> If you don't want to use /dev/sdb in a multipath configuration, please check
> /etc/multipath.conf to see why the device is being claimed.  For more
> information about multipath, see:
> 
> https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/
> html-single/DM_Multipath/index.html
> 
> If you *do* want to use /dev/sdb in a multipath configuration, then you
> should not access /dev/sdb directly.  Instead, run mkfs on the device-mapper
> device that multipath created.

yes, just as expected, there is no processes that have /dev/sdb open after i disable the multipath, and i can format the /dev/sdb successfully on host no matter i set the /sys/block/sdb/events_poll_msecs to "0" or not.
# lsof -n | grep '\/dev\/sdb'
                  <------output nothing

Closing as NOTABUG since this was a multipath configuration issue inside the guest.

If multipath was enabled manually by mistake (e.g. testing multipath and forgot to turn it off in this guest), then you do not need to do anything.

If the RHEL7 guest configured multipath automatically then this is a RHEL multipath or anaconda bug.  Please raise the issue with those teams instead of qemu-kvm.