RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 981251 - Libvirt should stop starting the fs and netfs pool using inexistent/unreachable source device
Summary: Libvirt should stop starting the fs and netfs pool using inexistent/unreachab...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 983539
TreeView+ depends on / blocked
 
Reported: 2013-07-04 10:11 UTC by Hu Jianwei
Modified: 2014-06-18 00:52 UTC (History)
7 users (show)

Fixed In Version: libvirt-1.1.1-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 983539 (view as bug list)
Environment:
Last Closed: 2014-06-13 09:32:52 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Hu Jianwei 2013-07-04 10:11:50 UTC 
Description of problem:
Libvirt should stop starting the fs and netfs pool using inexistent/unreachable source device
 
Version-Release number of selected component (if applicable):
libvirt-1.1.0-1.el7.x86_64
qemu-kvm-1.5.1-1.el7.x86_64
kernel-3.9.0-0.55.el7.x86_64

How reproducible:
100%

Steps:
1. Define one fs and nfs pool, like below:
fs pool:
<pool type='fs'>
  <name>fs_pool_sdb1</name>
  <uuid>591d601d-fe96-470c-87ac-cd4bc6d6d064</uuid>
  <capacity unit='bytes'>42141548544</capacity>
  <allocation unit='bytes'>33044459520</allocation>
  <available unit='bytes'>9097089024</available>
  <source>
    <device path='/dev/sdb1'/>
    <format type='ext3'/>
  </source>
  <target>
    <path>/machine/sdb1</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

netfs pool:
<pool type='netfs'>
  <name>nfs_pool</name>
  <uuid>78f87cd3-bc2c-429a-b9b6-826bf168faf8</uuid>
  <capacity unit='bytes'>42141548544</capacity>
  <allocation unit='bytes'>33044541440</allocation>
  <available unit='bytes'>9097007104</available>
  <source>
    <host name='10.66.106.33'/>
    <dir path='/libvirt_nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/nfs_pool</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

2.Two source devices are available and start them.

[root@hujianweitest nfs_pool]# parted /dev/sdb print
Model: Generic- SD/MMC (scsi)
Disk /dev/sdb: 1967MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      32.3kB  525MB  525MB  primary  ext3

[root@hujianweitest nfs_pool]# showmount -e 10.66.106.33
Export list for 10.66.106.33:
/libvirt_nfs *

[root@hujianweitest nfs_pool]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@hujianweitest nfs_pool]# virsh pool-start nfs_pool
Pool nfs_pool started

[root@hujianweitest nfs_pool]# virsh pool-list --all --details
Name             State     Autostart  Persistent    Capacity  Allocation   Available
------------------------------------------------------------------------------------
default          running   yes        yes          39.25 GiB   30.77 GiB    8.47 GiB
fs_pool_sdb1     running   no         yes         476.58 MiB    2.30 MiB  474.28 MiB
nfs_pool         running   no         yes          39.25 GiB    1.88 GiB   37.37 GiB

[root@hujianweitest nfs_pool]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@hujianweitest nfs_pool]# virsh pool-destroy fs_pool_sdb1
Pool fs_pool_sdb1 destroyed

3. Stop/block the nfs server and remove sdb1 partition, then restart these two pools.
[root@hujianweitest nfs_pool]# showmount -e 10.66.106.33
clnt_create: RPC: Program not registered
[root@hujianweitest nfs_pool]# parted /dev/sdb print
Error: Could not stat device /dev/sdb - No such file or directory.
Retry/Cancel?

[root@hujianweitest nfs_pool]# virsh pool-start fs_pool_sdb1
Pool fs_pool_sdb1 started

[root@hujianweitest nfs_pool]# virsh pool-start nfs_pool
Pool nfs_pool started

[root@hujianweitest nfs_pool]# virsh pool-list --all --details
Name             State     Autostart  Persistent    Capacity  Allocation   Available
------------------------------------------------------------------------------------
default          running   yes        yes          39.25 GiB   30.77 GiB    8.47 GiB
fs_pool_sdb1     running   no         yes          39.25 GiB   30.78 GiB    8.47 GiB       <==== the pool is wrong
nfs_pool         running   no         yes          39.25 GiB   30.78 GiB    8.47 GiB       <==== the pool is wrong


Actual results:

The pools with inexistent/unreachable source devices still can start, even if the errors happened.
...
214713 2013-07-04 08:05:47.788+0000: 1565: error : virCommandWait:2351 : internal error Child process (/usr/bin/mount -t ext3 /dev/sdb1 /machine/sdb1) unexpected exit status 32: 2013-07-04 08:05:47.7       86+0000: 2157: debug : virFileClose:90 : Closed fd 24
214714 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 26
214715 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 22
214716 mount: special device /dev/sdb1 does not exist
214717
214718 2013-07-04 08:05:47.788+0000: 1565: debug : virCommandRun:2115 : Result status 0, stdout: '' stderr: '2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 24
214719 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 26
214720 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 22
214721 mount: special device /dev/sdb1 does not exist
214722 '
214723 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214724 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 25
214725 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214726 2013-07-04 08:05:47.788+0000: 1565: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '..'
214727 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214728 2013-07-04 08:05:47.788+0000: 1565: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '.'
214729 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileProbeFormatFromBuf:688 : path=/machine/sdb1/image.img
214730 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileProbeFormatFromBuf:716 : format=1
214731 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileGetMetadataInternal:772 : path=/machine/sdb1/image.img, fd=23, format=1
214732 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214733 2013-07-04 08:05:47.788+0000: 1565: info : storagePoolCreate:748 : Starting up storage pool 'fs_pool_sdb1'
214734 2013-07-04 08:05:47.788+0000: 1565: debug : virStoragePoolFree:13657 : pool=0x7f14fc0009a0
214735 /lib64/libvirt.so.0(virLogMessage+0x87)[0x7f153a36cf47]
214736 /lib64/libvirt.so.0(virStoragePoolFree+0x3c)[0x7f153a41394c]
214737 /usr/sbin/libvirtd(+0x2cdf9)[0x7f153adfbdf9]
214738 /lib64/libvirt.so.0(virNetServerProgramDispatch+0x367)[0x7f153a45e9e7]
214739 /lib64/libvirt.so.0(+0x162af8)[0x7f153a459af8]
214740 /lib64/libvirt.so.0(+0x92e65)[0x7f153a389e65]
214741 /lib64/libvirt.so.0(+0x928f1)[0x7f153a3898f1]
214742 /lib64/libpthread.so.0(+0x30d6a07c53)[0x7f1537c1ec53]
...
219387 2013-07-04 08:08:01.236+0000: 1567: error : virCommandWait:2351 : internal error Child process (/usr/bin/mount -t nfs 10.66.106.33:/libvirt_nfs /nfs_pool) unexpected exit status 32: 2013-07-04        08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 24
219388 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 26
219389 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 22
219390 mount.nfs: Connection timed out
219391
219392 2013-07-04 08:08:01.236+0000: 1567: debug : virCommandRun:2115 : Result status 0, stdout: '' stderr: '2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 24
219393 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 26
219394 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 22
219395 mount.nfs: Connection timed out
219396 '
219397 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219398 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 25
219399 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219400 2013-07-04 08:08:01.236+0000: 1567: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '..'
219401 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219402 2013-07-04 08:08:01.236+0000: 1567: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '.'
219403 2013-07-04 08:08:01.236+0000: 1567: info : storagePoolCreate:748 : Starting up storage pool 'nfs_pool'
219404 2013-07-04 08:08:01.236+0000: 1567: debug : virStoragePoolFree:13657 : pool=0x7f1504007b80
219405 /lib64/libvirt.so.0(virLogMessage+0x87)[0x7f153a36cf47]
219406 /lib64/libvirt.so.0(virStoragePoolFree+0x3c)[0x7f153a41394c]
219407 /usr/sbin/libvirtd(+0x2cdf9)[0x7f153adfbdf9]
219408 /lib64/libvirt.so.0(virNetServerProgramDispatch+0x367)[0x7f153a45e9e7]
219409 /lib64/libvirt.so.0(+0x162af8)[0x7f153a459af8]
219410 /lib64/libvirt.so.0(+0x92e65)[0x7f153a389e65]
219411 /lib64/libvirt.so.0(+0x928f1)[0x7f153a3898f1]
219412 /lib64/libpthread.so.0(+0x30d6a07c53)[0x7f1537c1ec53]
219413 /lib64/libc.so.6(clone+0x6d)[0x7f1537544ecd]
...


Expected results:

I thinks libvirt should block these pools to start, just like disk and iscsi pools.

Error info:
[root@hujianweitest nfs_pool]# virsh pool-start test-iscsi
error: Failed to start pool test-iscsi
error: internal error Child process (/usr/sbin/iscsiadm --mode discovery --type sendtargets --portal 10.66.90.100:3260,1) unexpected exit status 4

[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]# virsh pool-start sdb
error: Failed to start pool sdb
error: invalid argument: device path '/dev/sdb' doesn't exist
Comment 2 Ján Tomko 2013-07-11 10:43:40 UTC 
Upstream patch proposed:
https://www.redhat.com/archives/libvir-list/2013-July/msg00678.html
Comment 3 Ján Tomko 2013-07-11 11:49:21 UTC 
Now fixed upstream:
commit 13fde7ceab556804dc6cfb3e56938fb948ffe83d
Author:     Ján Tomko <jtomko>
AuthorDate: 2013-07-11 12:36:59 +0200
Commit:     Ján Tomko <jtomko>
CommitDate: 2013-07-11 12:42:15 +0200

    storage: return -1 when fs pool can't be mounted
    
    Don't reuse the return value of virStorageBackendFileSystemIsMounted.
    If it's 0, we'd return it even if the mount command failed.
    
    Also, don't report another error if it's -1, since one has already
    been reported.
    
    Introduced by 258e06c.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=981251

git describe: v1.1.0-146-g13fde7c
Comment 4 Luwen Su 2013-08-02 03:40:19 UTC 
Test with
libvirt-1.1.1-1.el7.x86_64
qemu-kvm-1.5.2-2.el7.x86_64

1.Prepare two pools
# virsh pool-list --all
Name                 State      Autostart 
-----------------------------------------
default              active     yes       
fs                   active     yes       
nfs                  active     yes    

xml same as comment 0

2.
#virsh pool-destroy fs
#virsh pool-destroy nfs

3.
hotunplug the sdb whose fs's source dev
stop the nfs server on the nfs pool source host

4.Try to start those pools
#virsh pool-start fs
error: Failed to start pool fs
error: internal error: Child process (/usr/bin/mount -t auto /dev/sdb /var/lib/libvirt/images/fs) unexpected exit status 32: mount: special device /dev/sdb does not exist

# virsh pool-start nfs
error: Failed to start pool nfs
error: internal error: Child process (/usr/bin/mount 10.66.6.56:/test /var/lib/libvirt/images/nfs) unexpected exit status 32: mount.nfs: Connection timed o

5.Resume the environment , then start these pools again
# virsh pool-start fs
Pool fs started

# virsh pool-start nfs
Pool nfs started

# virsh pool-list --all
Name                 State      Autostart 
-----------------------------------------
default              active     yes       
fs                   active     yes       
nfs                  active     yes  

So change the bug to VERIFIED
Comment 5 Ludek Smid 2014-06-13 09:32:52 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Note You need to log in before you can comment on or make changes to this bug.