Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 983539 - Libvirt should stop starting the fs and netfs pool using inexistent/unreachable source device
Libvirt should stop starting the fs and netfs pool using inexistent/unreachab...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.4
x86_64 Linux
medium Severity high
: rc
: ---
Assigned To: Ján Tomko
Virtualization Bugs
:
Depends On: 981251
Blocks:
  Show dependency tree / graph
 
Reported: 2013-07-11 08:01 EDT by Ján Tomko
Modified: 2013-11-21 04:05 EST (History)
7 users (show)

See Also:
Fixed In Version: libvirt-0.10.2-20.el6
Doc Type: Bug Fix
Doc Text:
Cause: The function virStorageBackendFileSystemMount returned success even if the mount command failed. Consequence: Libvirt showed the pool as running even though it was unusable. Fix: Return an error if the mount command failed. Result: Libvirt no longer says starting of the pool with an unreachable source device succeeded.
Story Points: ---
Clone Of: 981251
Environment:
Last Closed: 2013-11-21 04:05:07 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1581 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2013-11-20 20:11:35 EST

  None (edit)
Description Ján Tomko 2013-07-11 08:01:18 EDT 
Also reproducible with libvirt-0.10.2-18.el6.x86_64.

+++ This bug was initially created as a clone of Bug #981251 +++

Description of problem:
Libvirt should stop starting the fs and netfs pool using inexistent/unreachable source device
 
Version-Release number of selected component (if applicable):
libvirt-1.1.0-1.el7.x86_64
qemu-kvm-1.5.1-1.el7.x86_64
kernel-3.9.0-0.55.el7.x86_64

How reproducible:
100%

Steps:
1. Define one fs and nfs pool, like below:
fs pool:
<pool type='fs'>
  <name>fs_pool_sdb1</name>
  <uuid>591d601d-fe96-470c-87ac-cd4bc6d6d064</uuid>
  <capacity unit='bytes'>42141548544</capacity>
  <allocation unit='bytes'>33044459520</allocation>
  <available unit='bytes'>9097089024</available>
  <source>
    <device path='/dev/sdb1'/>
    <format type='ext3'/>
  </source>
  <target>
    <path>/machine/sdb1</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

netfs pool:
<pool type='netfs'>
  <name>nfs_pool</name>
  <uuid>78f87cd3-bc2c-429a-b9b6-826bf168faf8</uuid>
  <capacity unit='bytes'>42141548544</capacity>
  <allocation unit='bytes'>33044541440</allocation>
  <available unit='bytes'>9097007104</available>
  <source>
    <host name='10.66.106.33'/>
    <dir path='/libvirt_nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/nfs_pool</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

2.Two source devices are available and start them.

[root@hujianweitest nfs_pool]# parted /dev/sdb print
Model: Generic- SD/MMC (scsi)
Disk /dev/sdb: 1967MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      32.3kB  525MB  525MB  primary  ext3

[root@hujianweitest nfs_pool]# showmount -e 10.66.106.33
Export list for 10.66.106.33:
/libvirt_nfs *

[root@hujianweitest nfs_pool]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@hujianweitest nfs_pool]# virsh pool-start nfs_pool
Pool nfs_pool started

[root@hujianweitest nfs_pool]# virsh pool-list --all --details
Name             State     Autostart  Persistent    Capacity  Allocation   Available
------------------------------------------------------------------------------------
default          running   yes        yes          39.25 GiB   30.77 GiB    8.47 GiB
fs_pool_sdb1     running   no         yes         476.58 MiB    2.30 MiB  474.28 MiB
nfs_pool         running   no         yes          39.25 GiB    1.88 GiB   37.37 GiB

[root@hujianweitest nfs_pool]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@hujianweitest nfs_pool]# virsh pool-destroy fs_pool_sdb1
Pool fs_pool_sdb1 destroyed

3. Stop/block the nfs server and remove sdb1 partition, then restart these two pools.
[root@hujianweitest nfs_pool]# showmount -e 10.66.106.33
clnt_create: RPC: Program not registered
[root@hujianweitest nfs_pool]# parted /dev/sdb print
Error: Could not stat device /dev/sdb - No such file or directory.
Retry/Cancel?

[root@hujianweitest nfs_pool]# virsh pool-start fs_pool_sdb1
Pool fs_pool_sdb1 started

[root@hujianweitest nfs_pool]# virsh pool-start nfs_pool
Pool nfs_pool started

[root@hujianweitest nfs_pool]# virsh pool-list --all --details
Name             State     Autostart  Persistent    Capacity  Allocation   Available
------------------------------------------------------------------------------------
default          running   yes        yes          39.25 GiB   30.77 GiB    8.47 GiB
fs_pool_sdb1     running   no         yes          39.25 GiB   30.78 GiB    8.47 GiB       <==== the pool is wrong
nfs_pool         running   no         yes          39.25 GiB   30.78 GiB    8.47 GiB       <==== the pool is wrong


Actual results:

The pools with inexistent/unreachable source devices still can start, even if the errors happened.
...
214713 2013-07-04 08:05:47.788+0000: 1565: error : virCommandWait:2351 : internal error Child process (/usr/bin/mount -t ext3 /dev/sdb1 /machine/sdb1) unexpected exit status 32: 2013-07-04 08:05:47.7       86+0000: 2157: debug : virFileClose:90 : Closed fd 24
214714 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 26
214715 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 22
214716 mount: special device /dev/sdb1 does not exist
214717
214718 2013-07-04 08:05:47.788+0000: 1565: debug : virCommandRun:2115 : Result status 0, stdout: '' stderr: '2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 24
214719 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 26
214720 2013-07-04 08:05:47.786+0000: 2157: debug : virFileClose:90 : Closed fd 22
214721 mount: special device /dev/sdb1 does not exist
214722 '
214723 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214724 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 25
214725 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214726 2013-07-04 08:05:47.788+0000: 1565: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '..'
214727 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214728 2013-07-04 08:05:47.788+0000: 1565: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '.'
214729 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileProbeFormatFromBuf:688 : path=/machine/sdb1/image.img
214730 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileProbeFormatFromBuf:716 : format=1
214731 2013-07-04 08:05:47.788+0000: 1565: debug : virStorageFileGetMetadataInternal:772 : path=/machine/sdb1/image.img, fd=23, format=1
214732 2013-07-04 08:05:47.788+0000: 1565: debug : virFileClose:90 : Closed fd 23
214733 2013-07-04 08:05:47.788+0000: 1565: info : storagePoolCreate:748 : Starting up storage pool 'fs_pool_sdb1'
214734 2013-07-04 08:05:47.788+0000: 1565: debug : virStoragePoolFree:13657 : pool=0x7f14fc0009a0
214735 /lib64/libvirt.so.0(virLogMessage+0x87)[0x7f153a36cf47]
214736 /lib64/libvirt.so.0(virStoragePoolFree+0x3c)[0x7f153a41394c]
214737 /usr/sbin/libvirtd(+0x2cdf9)[0x7f153adfbdf9]
214738 /lib64/libvirt.so.0(virNetServerProgramDispatch+0x367)[0x7f153a45e9e7]
214739 /lib64/libvirt.so.0(+0x162af8)[0x7f153a459af8]
214740 /lib64/libvirt.so.0(+0x92e65)[0x7f153a389e65]
214741 /lib64/libvirt.so.0(+0x928f1)[0x7f153a3898f1]
214742 /lib64/libpthread.so.0(+0x30d6a07c53)[0x7f1537c1ec53]
...
219387 2013-07-04 08:08:01.236+0000: 1567: error : virCommandWait:2351 : internal error Child process (/usr/bin/mount -t nfs 10.66.106.33:/libvirt_nfs /nfs_pool) unexpected exit status 32: 2013-07-04        08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 24
219388 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 26
219389 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 22
219390 mount.nfs: Connection timed out
219391
219392 2013-07-04 08:08:01.236+0000: 1567: debug : virCommandRun:2115 : Result status 0, stdout: '' stderr: '2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 24
219393 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 26
219394 2013-07-04 08:05:56.219+0000: 2160: debug : virFileClose:90 : Closed fd 22
219395 mount.nfs: Connection timed out
219396 '
219397 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219398 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 25
219399 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219400 2013-07-04 08:08:01.236+0000: 1567: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '..'
219401 2013-07-04 08:08:01.236+0000: 1567: debug : virFileClose:90 : Closed fd 23
219402 2013-07-04 08:08:01.236+0000: 1567: info : virStorageBackendVolOpenCheckMode:1152 : Skipping special dir '.'
219403 2013-07-04 08:08:01.236+0000: 1567: info : storagePoolCreate:748 : Starting up storage pool 'nfs_pool'
219404 2013-07-04 08:08:01.236+0000: 1567: debug : virStoragePoolFree:13657 : pool=0x7f1504007b80
219405 /lib64/libvirt.so.0(virLogMessage+0x87)[0x7f153a36cf47]
219406 /lib64/libvirt.so.0(virStoragePoolFree+0x3c)[0x7f153a41394c]
219407 /usr/sbin/libvirtd(+0x2cdf9)[0x7f153adfbdf9]
219408 /lib64/libvirt.so.0(virNetServerProgramDispatch+0x367)[0x7f153a45e9e7]
219409 /lib64/libvirt.so.0(+0x162af8)[0x7f153a459af8]
219410 /lib64/libvirt.so.0(+0x92e65)[0x7f153a389e65]
219411 /lib64/libvirt.so.0(+0x928f1)[0x7f153a3898f1]
219412 /lib64/libpthread.so.0(+0x30d6a07c53)[0x7f1537c1ec53]
219413 /lib64/libc.so.6(clone+0x6d)[0x7f1537544ecd]
...


Expected results:

I thinks libvirt should block these pools to start, just like disk and iscsi pools.

Error info:
[root@hujianweitest nfs_pool]# virsh pool-start test-iscsi
error: Failed to start pool test-iscsi
error: internal error Child process (/usr/sbin/iscsiadm --mode discovery --type sendtargets --portal 10.66.90.100:3260,1) unexpected exit status 4

[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]#
[root@hujianweitest nfs_pool]# virsh pool-start sdb
error: Failed to start pool sdb
error: invalid argument: device path '/dev/sdb' doesn't exist

--- Additional comment from Jan Tomko on 2013-07-11 06:43:40 EDT ---

Upstream patch proposed:
https://www.redhat.com/archives/libvir-list/2013-July/msg00678.html

--- Additional comment from Jan Tomko on 2013-07-11 07:49:21 EDT ---

Now fixed upstream:
commit 13fde7ceab556804dc6cfb3e56938fb948ffe83d
Author:     Ján Tomko <jtomko@redhat.com>
AuthorDate: 2013-07-11 12:36:59 +0200
Commit:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2013-07-11 12:42:15 +0200

    storage: return -1 when fs pool can't be mounted
    
    Don't reuse the return value of virStorageBackendFileSystemIsMounted.
    If it's 0, we'd return it even if the mount command failed.
    
    Also, don't report another error if it's -1, since one has already
    been reported.
    
    Introduced by 258e06c.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=981251

git describe: v1.1.0-146-g13fde7c
Comment 3 Hu Jianwei 2013-07-16 04:17:35 EDT 
The bug can not reproduce on libvirt-0.10.2-20.el6.x86_64.

1. Start these pools.
[root@test ~]# virsh pool-list --all --details
Name          State    Autostart  Persistent    Capacity  Allocation   Available
--------------------------------------------------------------------------------
default       running  yes        yes         147.65 GiB   12.97 GiB  134.68 GiB
fs_pool_sdb1  running  no         yes         485.45 MiB   10.30 MiB  475.16 MiB
nfs_pool      running  no         yes          49.09 GiB    3.38 GiB   45.71 GiB

[root@test ~]# 
[root@test ~]# virsh pool-dumpxml fs_pool_sdb1
<pool type='fs'>
  <name>fs_pool_sdb1</name>
  <uuid>9a28db7c-de6e-752c-d70a-0ef0a215eebe</uuid>
  <capacity unit='bytes'>509035520</capacity>
  <allocation unit='bytes'>10796032</allocation>
  <available unit='bytes'>498239488</available>
  <source>
    <device path='/dev/sdb1'/>
    <format type='ext3'/>
  </source>
  <target>
    <path>/machine/sdb1</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

[root@test ~]# virsh pool-dumpxml nfs_pool
<pool type='netfs'>
  <name>nfs_pool</name>
  <uuid>1b4a7db0-e755-55df-e690-cebfe07c8ac2</uuid>
  <capacity unit='bytes'>52710866944</capacity>
  <allocation unit='bytes'>3634364416</allocation>
  <available unit='bytes'>49076502528</available>
  <source>
    <host name='*.*.*.*'/>
    <dir path='/libvirt_nfs'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/nfs_pool</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

2. Destroy these pools and block the source of pools from host machine.
[root@test ~]# virsh pool-list --all --details
Name          State    Autostart  Persistent    Capacity  Allocation   Available
--------------------------------------------------------------------------------
default       running  yes        yes         147.65 GiB   12.97 GiB  134.68 GiB
fs_pool_sdb1  running  no         yes         485.45 MiB   10.30 MiB  475.16 MiB
nfs_pool      running  no         yes          49.09 GiB    3.38 GiB   45.71 GiB

[root@test ~]# virsh pool-destroy fs_pool_sdb1
Pool fs_pool_sdb1 destroyed

[root@test ~]# virsh pool-destroy nfs_pool
Pool nfs_pool destroyed

[root@test ~]# parted /dev/sdb print
Error: Could not stat device /dev/sdb - No such file or directory.        
Retry/Cancel? ^C                                                          
[root@test ~]# showmount -e *.*.*.* 
Export list for 10.66.100.107:
/libvirt_nfs *
[root@test ~]# showmount -e *.*.*.*
clnt_create: RPC: Program not registered

[root@test ~]# virsh pool-start fs_pool_sdb1
error: Failed to start pool fs_pool_sdb1
error: internal error Child process (/bin/mount -t ext3 /dev/sdb1 /machine/sdb1) unexpected exit status 32: mount: special device /dev/sdb1 does not exist

[root@test ~]# virsh pool-start nfs_pool
error: Failed to start pool nfs_pool
error: internal error Child process (/bin/mount -t nfs 10.66.100.107:/libvirt_nfs /nfs_pool) unexpected exit status 32: mount.nfs: Connection timed out

3. Resume the source of pools, and restart these pools, they are started normally.

[root@test ~]# parted /dev/sdb print
Model: Generic- SD/MMC (scsi)
Disk /dev/sdb: 1967MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  527MB  526MB  primary  ext3

[root@test ~]# showmount -e *.*.*.*
Export list for *.*.*.*:
/libvirt_nfs *

[root@test ~]# virsh pool-start fs_pool_sdb1
Pool fs_pool_sdb1 started

[root@test ~]# virsh pool-start nfs_pool
Pool nfs_pool started


We can get expected results, changed it to verified.
Comment 5 errata-xmlrpc 2013-11-21 04:05:07 EST 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.