RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 983539 - Libvirt should stop starting the fs and netfs pool using inexistent/unreachable source device
Summary: Libvirt should stop starting the fs and netfs pool using inexistent/unreachab...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.4
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 981251
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-11 12:01 UTC by Ján Tomko
Modified: 2013-11-21 09:05 UTC (History)
7 users (show)

Fixed In Version: libvirt-0.10.2-20.el6
Doc Type: Bug Fix
Doc Text:
Cause: The function virStorageBackendFileSystemMount returned success even if the mount command failed. Consequence: Libvirt showed the pool as running even though it was unusable. Fix: Return an error if the mount command failed. Result: Libvirt no longer says starting of the pool with an unreachable source device succeeded.
Clone Of: 981251
Environment:
Last Closed: 2013-11-21 09:05:07 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1581 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2013-11-21 01:11:35 UTC

Description Ján Tomko 2013-07-11 12:01:18 UTC 
Also reproducible with libvirt-0.10.2-18.el6.x86_64.

+++ This bug was initially created as a clone of Bug #981251 +++

Description of problem:
Libvirt should stop starting the fs and netfs pool using inexistent/unreachable source device
 
Version-Release number of selected component (if applicable):
libvirt-1.1.0-1.el7.x86_64
qemu-kvm-1.5.1-1.el7.x86_64
kernel-3.9.0-0.55.el7.x86_64

How reproducible:
100%

Steps:
1. Define one fs and nfs pool, like below:
fs pool:
<pool type='fs'>
  <name>fs_pool_sdb1</name>
  <uuid>591d601d-fe96-470c-87ac-cd4bc6d6d064</uuid>
  <capacity unit='bytes'>42141548544</capacity>
  <allocation unit='bytes'>33044459520</allocation>
  <available unit='bytes'>9097089024</available>
  <source>
    <device path='/dev/sdb1'/>
    <format type='ext3'/>
  </source>
  <target>
    <path>/machine/sdb1</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

netfs pool:
<pool type='netfs'>
  <name>nfs_pool</name>
  <uuid>78f87cd3-bc2c-429a-b9b6-826bf168faf8</uuid>
  <capacity unit='bytes'>42141548544</capacity>
  <allocation unit='bytes'>33044541440</allocation>
  <available unit='bytes'>9097007104</available>
  <source>
    <host name='10.66.106.33'/>
    <dir path='/libvirt_nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/nfs_pool</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

2.Two source devices are available and start them.

[root@hujianweitest nfs_pool]# parted /dev/sdb print
Model: Generic- SD/MMC (scsi)
Disk /dev/sdb: 1967MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      32.3kB  525MB  525MB  primary  ext3

[root@hujianweitest nfs_pool]# showmount -e 10.66.106.33
Export list for 10.66.106.33:
/libvirt_nfs *

[root@hujianweitest nfs_pool]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@hujianweitest nfs_pool]# virsh pool-start nfs_pool
Pool nfs_pool started

[root@hujianweitest nfs_pool]# virsh pool-list --all --details
Name             State     Autostart  Persistent    Capacity  Allocation   Available
------------------------------------------------------------------------------------
default          running   yes        yes          39.25 GiB   30.77 GiB    8.47 GiB
fs_pool_sdb1     running   no         yes         476.58 MiB    2.30 MiB  474.28 MiB
nfs_pool         running   no         yes          39.25 GiB    1.88 GiB   37.37 GiB

[root@hujianweitest nfs_pool]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@hujianweitest nfs_pool]# virsh pool-destroy fs_pool_sdb1
Pool fs_pool_sdb1 destroyed

3. Stop/block the nfs server and remove sdb1 partition, then restart these two pools.
[root@hujianweitest nfs_pool]# showmount -e 10.66.106.33
clnt_create: RPC: Program not registered
[root@hujianweitest nfs_pool]# parted /dev/sdb print
Error: Could not stat device /dev/sdb - No such file or directory.
Retry/Cancel?

[root@hujianweitest nfs_pool]# virsh pool-start fs_pool_sdb1
Pool fs_pool_sdb1 started

[root@hujianweitest nfs_pool]# virsh pool-start nfs_pool
Pool nfs_pool started

[root@hujianweitest nfs_pool]# virsh pool-list --all --details
Name             State     Autostart  Persistent    Capacity  Allocation   Available
------------------------------------------------------------------------------------
default          running   yes        yes          39.25 GiB   30.77 GiB    8.47 GiB
fs_pool_sdb1     running   no         yes          39.25 GiB   30.78 GiB    8.47 GiB       <==== the pool is wrong
nfs_pool         running   no         yes          39.25 GiB   30.78 GiB    8.47 GiB       <==== the pool is wrong


Actual results:

The pools with inexistent/unreachable source devices still can start, even if the errors happened.
...
214713 2013-07-04 08:05:47.788+0000: 1565: error : virCommandWait:2351 : internal error Child process (/usr/bin/mount -t ext3 /dev/sdb1 /machine/sdb1) unexpected exit status 32: 2013-07-04 08:05:47.7       86+0000: 2157: debug : virFileClose:90 : Closed fd 24
214714 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 26
214715 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 22
214716 mount: special device /dev/sdb1 does not exist
214717
214718 2013-07-04 08:05:47.788+0000: 1565: debug : virCommandRun:2115 : Result status 0, stdout: '' stderr: '2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 24
214719 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 26
214720 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 22
214721 mount: special device /dev/sdb1 does not exist
214722 '
214723 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214724 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 25
214725 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214726 2013-07-04 08:05:47.788+0000: 1565: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '..'
214727 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214728 2013-07-04 08:05:47.788+0000: 1565: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '.'
214729 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileProbeFormatFromBuf:688 : path=/machine/sdb1/image.img
214730 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileProbeFormatFromBuf:716 : format=1
214731 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileGetMetadataInternal:772 : path=/machine/sdb1/image.img, fd=23, format=1
214732 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214733 2013-07-04 08:05:47.788+0000: 1565: info : storagePoolCreate:748 : Starting up storage pool 'fs_pool_sdb1'
214734 2013-07-04 08:05:47.788+0000: 1565: debug : virStoragePoolFree:13657 : pool=0x7f14fc0009a0
214735 /lib64/libvirt.so.0(virLogMessage+0x87)[0x7f153a36cf47]
214736 /lib64/libvirt.so.0(virStoragePoolFree+0x3c)[0x7f153a41394c]
214737 /usr/sbin/libvirtd(+0x2cdf9)[0x7f153adfbdf9]
214738 /lib64/libvirt.so.0(virNetServerProgramDispatch+0x367)[0x7f153a45e9e7]
214739 /lib64/libvirt.so.0(+0x162af8)[0x7f153a459af8]
214740 /lib64/libvirt.so.0(+0x92e65)[0x7f153a389e65]
214741 /lib64/libvirt.so.0(+0x928f1)[0x7f153a3898f1]
214742 /lib64/libpthread.so.0(+0x30d6a07c53)[0x7f1537c1ec53]
...
219387 2013-07-04 08:08:01.236+0000: 1567: error : virCommandWait:2351 : internal error Child process (/usr/bin/mount -t nfs 10.66.106.33:/libvirt_nfs /nfs_pool) unexpected exit status 32: 2013-07-04        08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 24
219388 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 26
219389 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 22
219390 mount.nfs: Connection timed out
219391
219392 2013-07-04 08:08:01.236+0000: 1567: debug : virCommandRun:2115 : Result status 0, stdout: '' stderr: '2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 24
219393 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 26
219394 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 22
219395 mount.nfs: Connection timed out
219396 '
219397 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219398 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 25
219399 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219400 2013-07-04 08:08:01.236+0000: 1567: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '..'
219401 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219402 2013-07-04 08:08:01.236+0000: 1567: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '.'
219403 2013-07-04 08:08:01.236+0000: 1567: info : storagePoolCreate:748 : Starting up storage pool 'nfs_pool'
219404 2013-07-04 08:08:01.236+0000: 1567: debug : virStoragePoolFree:13657 : pool=0x7f1504007b80
219405 /lib64/libvirt.so.0(virLogMessage+0x87)[0x7f153a36cf47]
219406 /lib64/libvirt.so.0(virStoragePoolFree+0x3c)[0x7f153a41394c]
219407 /usr/sbin/libvirtd(+0x2cdf9)[0x7f153adfbdf9]
219408 /lib64/libvirt.so.0(virNetServerProgramDispatch+0x367)[0x7f153a45e9e7]
219409 /lib64/libvirt.so.0(+0x162af8)[0x7f153a459af8]
219410 /lib64/libvirt.so.0(+0x92e65)[0x7f153a389e65]
219411 /lib64/libvirt.so.0(+0x928f1)[0x7f153a3898f1]
219412 /lib64/libpthread.so.0(+0x30d6a07c53)[0x7f1537c1ec53]
219413 /lib64/libc.so.6(clone+0x6d)[0x7f1537544ecd]
...


Expected results:

I thinks libvirt should block these pools to start, just like disk and iscsi pools.

Error info:
[root@hujianweitest nfs_pool]# virsh pool-start test-iscsi
error: Failed to start pool test-iscsi
error: internal error Child process (/usr/sbin/iscsiadm --mode discovery --type sendtargets --portal 10.66.90.100:3260,1) unexpected exit status 4

[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]# virsh pool-start sdb
error: Failed to start pool sdb
error: invalid argument: device path '/dev/sdb' doesn't exist

--- Additional comment from Jan Tomko on 2013-07-11 06:43:40 EDT ---

Upstream patch proposed:
https://www.redhat.com/archives/libvir-list/2013-July/msg00678.html

--- Additional comment from Jan Tomko on 2013-07-11 07:49:21 EDT ---

Now fixed upstream:
commit 13fde7ceab556804dc6cfb3e56938fb948ffe83d
Author:     Ján Tomko <jtomko>
AuthorDate: 2013-07-11 12:36:59 +0200
Commit:     Ján Tomko <jtomko>
CommitDate: 2013-07-11 12:42:15 +0200

    storage: return -1 when fs pool can't be mounted
    
    Don't reuse the return value of virStorageBackendFileSystemIsMounted.
    If it's 0, we'd return it even if the mount command failed.
    
    Also, don't report another error if it's -1, since one has already
    been reported.
    
    Introduced by 258e06c.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=981251

git describe: v1.1.0-146-g13fde7c
Comment 1 Ján Tomko 2013-07-11 12:04:14 UTC 
Moving to POST:
http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-July/msg00291.html
Comment 3 Hu Jianwei 2013-07-16 08:17:35 UTC 
The bug can not reproduce on libvirt-0.10.2-20.el6.x86_64.

1. Start these pools.
[root@test ~]# virsh pool-list --all --details
Name          State    Autostart  Persistent    Capacity  Allocation   Available
--------------------------------------------------------------------------------
default       running  yes        yes         147.65 GiB   12.97 GiB  134.68 GiB
fs_pool_sdb1  running  no         yes         485.45 MiB   10.30 MiB  475.16 MiB
nfs_pool      running  no         yes          49.09 GiB    3.38 GiB   45.71 GiB

[root@test ~]# 
[root@test ~]# virsh pool-dumpxml fs_pool_sdb1
<pool type='fs'>
  <name>fs_pool_sdb1</name>
  <uuid>9a28db7c-de6e-752c-d70a-0ef0a215eebe</uuid>
  <capacity unit='bytes'>509035520</capacity>
  <allocation unit='bytes'>10796032</allocation>
  <available unit='bytes'>498239488</available>
  <source>
    <device path='/dev/sdb1'/>
    <format type='ext3'/>
  </source>
  <target>
    <path>/machine/sdb1</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

[root@test ~]# virsh pool-dumpxml nfs_pool
<pool type='netfs'>
  <name>nfs_pool</name>
  <uuid>1b4a7db0-e755-55df-e690-cebfe07c8ac2</uuid>
  <capacity unit='bytes'>52710866944</capacity>
  <allocation unit='bytes'>3634364416</allocation>
  <available unit='bytes'>49076502528</available>
  <source>
    <host name='*.*.*.*'/>
    <dir path='/libvirt_nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/nfs_pool</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

2. Destroy these pools and block the source of pools from host machine.
[root@test ~]# virsh pool-list --all --details
Name          State    Autostart  Persistent    Capacity  Allocation   Available
--------------------------------------------------------------------------------
default       running  yes        yes         147.65 GiB   12.97 GiB  134.68 GiB
fs_pool_sdb1  running  no         yes         485.45 MiB   10.30 MiB  475.16 MiB
nfs_pool      running  no         yes          49.09 GiB    3.38 GiB   45.71 GiB

[root@test ~]# virsh pool-destroy fs_pool_sdb1
Pool fs_pool_sdb1 destroyed

[root@test ~]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@test ~]# parted /dev/sdb print
Error: Could not stat device /dev/sdb - No such file or directory.        
Retry/Cancel? ^C                                                          
[root@test ~]# showmount -e *.*.*.* 
Export list for 10.66.100.107:
/libvirt_nfs *
[root@test ~]# showmount -e *.*.*.*
clnt_create: RPC: Program not registered

[root@test ~]# virsh pool-start fs_pool_sdb1
error: Failed to start pool fs_pool_sdb1
error: internal error Child process (/bin/mount -t ext3 /dev/sdb1 /machine/sdb1) unexpected exit status 32: mount: special device /dev/sdb1 does not exist

[root@test ~]# virsh pool-start nfs_pool
error: Failed to start pool nfs_pool
error: internal error Child process (/bin/mount -t nfs 10.66.100.107:/libvirt_nfs /nfs_pool) unexpected exit status 32: mount.nfs: Connection timed out

3. Resume the source of pools, and restart these pools, they are started normally.

[root@test ~]# parted /dev/sdb print
Model: Generic- SD/MMC (scsi)
Disk /dev/sdb: 1967MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  527MB  526MB  primary  ext3

[root@test ~]# showmount -e *.*.*.*
Export list for *.*.*.*:
/libvirt_nfs *

[root@test ~]# virsh pool-start fs_pool_sdb1
Pool fs_pool_sdb1 started

[root@test ~]# virsh pool-start nfs_pool
Pool nfs_pool started


We can get expected results, changed it to verified.
Comment 5 errata-xmlrpc 2013-11-21 09:05:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html
Note You need to log in before you can comment on or make changes to this bug.