Bug 1464313

(In reply to xiaodwan from comment #0)

I'm not quite following your steps...  I've thrown some comments in the middle for extra context...

> Description of problem:
> Failed to activate the disk pool if host reboots or libvirtd restarts
> 
> Version-Release number of selected component (if applicable):
> libvirt-3.2.0-14.el7.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Remove the partition info of the disk.
> # dd if=/dev/zero of=/dev/sdb bs=1M count=10 
> 2. Build a disk pool.
> 3. Restart the libvirtd service, Destroy the disk pool, then start it.
> 
> Actual results:
> # virsh pool-start usbdisk
> error: Failed to start pool usbdisk
> error: Requested operation is not valid: Format of device '/dev/sdb' does
> not match the expected format 'unknown'
> 
> Expected results:
> The disk pool should be actived successfully.

I disagree if you're performing that dd and libvirt doesn't know about it until it restarts, then it figures it out. Then you'd be getting expected results. What I'm not really clear on is when you're doing the dd to zero things out.

> 
> Additional info:
> [root@xiaodwan-test ~]# dd if=/dev/zero of=/dev/sdb bs=1M count=10 
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.43006 s, 7.3 MB/s

So while a Disk Pool was active, behind libvirt's back you wiped it's contents (similar to vol-wipe, but you didn't tell libvirt you did this.  It's not 100% clear from the following steps *when* this happened though.

> [root@xiaodwan-test ~]# virsh pool-list --all
>  Name                 State      Autostart 
> -------------------------------------------
>  boot-scratch         active     yes       
>  default              active     yes       
>  nfspool              inactive   no        
>  nvram                active     yes       
>  python-virtinst      active     yes       
>  root                 active     yes       
>  usbdisk              active     no        
>  xiaodwan             active     yes       
> 

OK, still there, still active, not autostart

> [root@xiaodwan-test ~]# virsh pool-dumpxml usbdisk
> <pool type='disk'>
>   <name>usbdisk</name>
>   <uuid>83b4f716-f00b-41ee-8477-1213562f3062</uuid>
>   <capacity unit='bytes'>2015193600</capacity>
>   <allocation unit='bytes'>0</allocation>
>   <available unit='bytes'>2015161344</available>
>   <source>
>     <device path='/dev/sdb'>
>       <freeExtent start='32256' end='2015193600'/>
>     </device>
>     <format type='dos'/>
>   </source>
>   <target>
>     <path>/dev</path>
>   </target>
> </pool>

libvirt still doesn't know that you've done this because you haven't told it to "refresh" yet.  You just got a dump of libvirt's last known state. The dumpxml doesn't re-read anything - just dumps out the XML that's been saved regarding what libvirt "knew" about the disk when it was first started.

Perhaps an appropriately placed "fdisk -l /dev/sdb" would show things...

> 
> [root@xiaodwan-test ~]# service libvirtd restart
> Redirecting to /bin/systemctl restart libvirtd.service

Up through here I'm still following, but the next steps... not so much.

again another "disk -l /dev/sdb" would help and I'm assuming nothing else has changed /dev/sdb at this point...  and of course a virsh pool-list

> [root@xiaodwan-test ~]# virsh pool-dumpxml usbdisk
> <pool type='disk'>
>   <name>usbdisk</name>
>   <uuid>83b4f716-f00b-41ee-8477-1213562f3062</uuid>
>   <capacity unit='bytes'>0</capacity>
>   <allocation unit='bytes'>0</allocation>
>   <available unit='bytes'>0</available>
>   <source>
>     <device path='/dev/sdb'/>
>     <format type='dos'/>
>   </source>
>   <target>
>     <path>/dev</path>
>   </target>
> </pool>
> 

I don't see this, I get format type='unknown' like I'd expect because restart causes a re-read of the device and the *configuration* XML would be updated. I am using the upstream version (3.7.0 - recently released).
 
> [root@xiaodwan-test ~]# virsh pool-start usbdisk
> Pool usbdisk started
> 

Again not see the same results.


> [root@xiaodwan-test ~]# virsh pool-destroy usbdisk
> Pool usbdisk destroyed
> 

So now you've stopped usbdisk and then you start it right away and it fails. It seems I'm not getting your exact test order/results as I see no way for this to happen unless in some other window or somewhere else you're modifying /dev/sdb and not showing it.

> [root@xiaodwan-test ~]# virsh pool-start usbdisk
> error: Failed to start pool usbdisk
> error: Requested operation is not valid: Format of device '/dev/sdb' does
> not match the expected format 'unknown'
> 

Well that's for sure

> [root@xiaodwan-test ~]# virsh pool-dumpxml usbdisk
> <pool type='disk'>
>   <name>usbdisk</name>
>   <uuid>83b4f716-f00b-41ee-8477-1213562f3062</uuid>
>   <capacity unit='bytes'>0</capacity>
>   <allocation unit='bytes'>0</allocation>
>   <available unit='bytes'>0</available>
>   <source>
>     <device path='/dev/sdb'/>
>     <format type='unknown'/>
>   </source>
>   <target>
>     <path>/dev</path>
>   </target>
> </pool>

The above matches what I'd expect.

What follows is my example using a memory stick stuck into a USB port on my laptop:

# lsscsi
[0:0:0:0]    disk    ATA      SAMSUNG MZ7LN512 4L0Q  /dev/sda 
[5:0:0:0]    cd/dvd  HL-DT-ST DVDRAM GUB0N     LV20  /dev/sr0 
[8:0:0:0]    disk    PNY      USB 2.0 FD       1100  /dev/sdo 
#

# cat usbdisk.xml
<pool type='disk'>
  <name>usbdisk</name>
  <source>
    <device path='/dev/sdo'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>

# virsh pool-define usbdisk.xml
Pool usbdisk defined from usbdisk.xml

# virsh pool-start usbdisk --build --overwrite
Pool usbdisk started

# virsh pool-dumpxml usbdisk
<pool type='disk'>
  <name>usbdisk</name>
  <uuid>9b4e5969-e1f3-49e2-a9ab-456b65e5985a</uuid>
  <capacity unit='bytes'>16358645760</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>16358644736</available>
  <source>
    <device path='/dev/sdo'>
      <freeExtent start='1024' end='16358645760'/>
    </device>
    <format type='dos'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>

# fdisk -l /dev/sdo
Disk /dev/sdo: 15.2 GiB, 16358768640 bytes, 31950720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x874172ec

# dd if=/dev/zero of=/dev/sdo bs=1M count=10 
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 1.24977 s, 8.4 MB/s
# fdisk -l /dev/sdo
Disk /dev/sdo: 15.2 GiB, 16358768640 bytes, 31950720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
# virsh pool-dumpxml usbdisk
<pool type='disk'>
  <name>usbdisk</name>
  <uuid>9b4e5969-e1f3-49e2-a9ab-456b65e5985a</uuid>
  <capacity unit='bytes'>16358645760</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>16358644736</available>
  <source>
    <device path='/dev/sdo'>
      <freeExtent start='1024' end='16358645760'/>
    </device>
    <format type='dos'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>

# service libvirtd restart
Restarting libvirtd (via systemctl):                       [  OK  ]
# virsh pool-dumpxml usbdisk
<pool type='disk'>
  <name>usbdisk</name>
  <uuid>9b4e5969-e1f3-49e2-a9ab-456b65e5985a</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
    <device path='/dev/sdo'/>
    <format type='unknown'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>

# virsh pool-list
 Name                 State      Autostart 
-------------------------------------------
 default              active     yes       
 dirpool              active     yes       
 images               active     yes       
 LVM_Test             active     no        

# virsh pool-start usbdisk
error: Failed to start pool usbdisk
error: Storage pool probe failed: Device '/dev/sdo' is unrecognized, requires build

]# virsh version
Compiled against library: libvirt 3.7.0
Using library: libvirt 3.7.0
Using API: QEMU 3.7.0
Running hypervisor: QEMU 2.9.50

# 


Everything is as I expect. Please either let me know which of my steps you're not seeing exactly as I'm seeing or that I may have down out of order from your example... You could also re-run your test and explicitly show every step you take. It could very well be something that has been changed since 3.2.0 has been resolved by 3.7.0. Not sure what yet though as most of the sticky things that dealt with device formats were backported into 3.2.0-14.

> > [root@xiaodwan-test ~]# dd if=/dev/zero of=/dev/sdb bs=1M count=10 
> > 10+0 records in
> > 10+0 records out
> > 10485760 bytes (10 MB) copied, 1.43006 s, 7.3 MB/s
> 
> So while a Disk Pool was active, behind libvirt's back you wiped it's
> contents (similar to vol-wipe, but you didn't tell libvirt you did this. 
> It's not 100% clear from the following steps *when* this happened though.

I did 'dd' operation after plugging the usb disk and before creating the disk pool. Because i created the disk pool by virt-manager, and virt-manager doesn't support '--overwirte", i have to wipe out the usb disk to make virt-manager create disk pool successfully.

> 
> What follows is my example using a memory stick stuck into a USB port on my
> laptop:
> 
> # lsscsi
> [0:0:0:0]    disk    ATA      SAMSUNG MZ7LN512 4L0Q  /dev/sda 
> [5:0:0:0]    cd/dvd  HL-DT-ST DVDRAM GUB0N     LV20  /dev/sr0 
> [8:0:0:0]    disk    PNY      USB 2.0 FD       1100  /dev/sdo 
> #
> 
> # cat usbdisk.xml
> <pool type='disk'>
>   <name>usbdisk</name>
>   <source>
>     <device path='/dev/sdo'/>
>   </source>
>   <target>
>     <path>/dev</path>
>   </target>
> </pool>
> 
> # virsh pool-define usbdisk.xml
> Pool usbdisk defined from usbdisk.xml
> 
> # virsh pool-start usbdisk --build --overwrite
> Pool usbdisk started
> 
> # virsh pool-dumpxml usbdisk
> <pool type='disk'>
>   <name>usbdisk</name>
>   <uuid>9b4e5969-e1f3-49e2-a9ab-456b65e5985a</uuid>
>   <capacity unit='bytes'>16358645760</capacity>
>   <allocation unit='bytes'>0</allocation>
>   <available unit='bytes'>16358644736</available>
>   <source>
>     <device path='/dev/sdo'>
>       <freeExtent start='1024' end='16358645760'/>
>     </device>
>     <format type='dos'/>
>   </source>
>   <target>
>     <path>/dev</path>
>   </target>
> </pool>
> 
> # fdisk -l /dev/sdo
> Disk /dev/sdo: 15.2 GiB, 16358768640 bytes, 31950720 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x874172ec
> 

I followed you steps until here. then i restart libvirtd service and tried destroy and restart the disk pool several times, and failed at last. 
My operations are as below:

[root@xiaodwan-test ~]# virsh pool-define usbdisk.xml
Pool usbdisk defined from usbdisk.xml

[root@xiaodwan-test ~]# virsh pool-start usbdisk --build --overwrite
Pool usbdisk started

[root@xiaodwan-test ~]# cat usbdisk.xml
<pool type='disk'>
  <name>usbdisk</name>
  <source>
    <device path='/dev/sdb'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>
[root@xiaodwan-test ~]# virsh pool-list --all
 Name                 State      Autostart 
-------------------------------------------
 boot-scratch         active     yes       
 default              active     yes       
 nfspool              inactive   no        
 nvram                active     yes       
 python-virtinst      active     yes       
 root                 active     yes       
 usbdisk              active     no        
 virtio-win           active     yes       
 xiaodwan             active     yes       

[root@xiaodwan-test ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 2019 MB, 2019557376 bytes, 3944448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b6b75

   Device Boot      Start         End      Blocks   Id  System
[root@xiaodwan-test ~]# virsh pool-dumpxml usbdisk
<pool type='disk'>
  <name>usbdisk</name>
  <uuid>43105cc0-549f-4d5e-9b4e-531ea17a24bf</uuid>
  <capacity unit='bytes'>2015193600</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>2015161344</available>
  <source>
    <device path='/dev/sdb'>
      <freeExtent start='32256' end='2015193600'/>
    </device>
    <format type='dos'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>

[root@xiaodwan-test ~]# virsh pool-destroy usbdisk
Pool usbdisk destroyed

[root@xiaodwan-test ~]# virsh pool-start usbdisk
Pool usbdisk started

[root@xiaodwan-test ~]# virsh pool-destroy usbdisk
Pool usbdisk destroyed

[root@xiaodwan-test ~]# virsh pool-start usbdisk
Pool usbdisk started

[root@xiaodwan-test ~]# service libvirtd restart
Redirecting to /bin/systemctl restart libvirtd.service
[root@xiaodwan-test ~]# virsh pool-list --all
 Name                 State      Autostart 
-------------------------------------------
 boot-scratch         active     yes       
 default              active     yes       
 nfspool              inactive   no        
 nvram                active     yes       
 python-virtinst      active     yes       
 root                 active     yes       
 usbdisk              inactive   no        
 virtio-win           active     yes       
 xiaodwan             active     yes       

[root@xiaodwan-test ~]# virsh pool-start usbdisk
Pool usbdisk started

[root@xiaodwan-test ~]# virsh pool-destroy usbdisk
Pool usbdisk destroyed

[root@xiaodwan-test ~]# virsh pool-start usbdisk
error: Failed to start pool usbdisk
error: Requested operation is not valid: Format of device '/dev/sdb' does not match the expected format 'unknown'

[root@xiaodwan-test ~]# virsh pool-destroy usbdisk
error: Failed to destroy pool usbdisk
error: Requested operation is not valid: storage pool 'usbdisk' is not active

[root@xiaodwan-test ~]# virsh pool-dumpxml usbdisk
<pool type='disk'>
  <name>usbdisk</name>
  <uuid>43105cc0-549f-4d5e-9b4e-531ea17a24bf</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
    <device path='/dev/sdb'/>
    <format type='unknown'/>
  </source>
  <target>
    <path>/dev</path>
  </target>
</pool>


# virsh version
Compiled against library: libvirt 3.2.0
Using library: libvirt 3.2.0
Using API: QEMU 3.2.0
Running hypervisor: QEMU 2.9.0

Hope above info can give some helps. Thanks!

And my packages:

# rpm -q libvirt qemu-kvm-rhev
libvirt-3.2.0-14.el7_4.3.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.6.x86_64

ahh... I see the 'dd' is a red herring in the process.

So I was able to reproduce what you saw now and have proposed a fix upstream for the problem:

https://www.redhat.com/archives/libvir-list/2017-September/msg00124.html

Essentially the problem is that the build --overwrite functionality is not updating the "static" (inactive) configFile. So added that into the code.

If someone does do a 'dd' after the initial start/build, then all bets are off again as the format will change, but libvirt won't be notified thus resulting in a similar problem, but that's to be expected.

v1 of the patch needed changes, v2 does things slightly differently... Mostly it doesn't update/change the persistent format of the storage XML, see:

https://www.redhat.com/archives/libvir-list/2017-September/msg00247.html

Patch pushed upstream:

commit 239781e03ae01d28b7ca79ace77c30295c783ba8
Author: John Ferlan <jferlan>
Date:   Mon Sep 11 17:11:25 2017 -0400

    storage: Adjust expected format for Disk startup processing
    
...
    
    If a Disk pool was defined/created using XML that either didn't
    specify a specific format or specified format type='unknown', then
    restarting a pool after an initial disk backend build with overwrite
    would fail after a libvirtd restart for a non-autostarted pool.
    
    This is because the persistent pool data is not updated during pool
    build w/ overwrite processing to have the VIR_STORAGE_POOL_DISK_DOS
    default format.
    
    So in addition to the alteration done during disk build processing,
    alter the default expectation for disk startup to be DOS if nothing
    has been defined yet. That will either succeed if the pool had been
    successfully built previously using the default DOS format or fail
    with a message indicating the format is something else that does not
    match the expect format 'dos'.


$ git describe 239781e03ae01d28b7ca79ace77c30295c783ba8
v3.7.0-53-g239781e03
$

Verified on:
libvirt-3.8.0-1.virtcov.el7.x86_64
qemu-kvm-rhev-2.10.0-3.el7.x86_64
and passed.

Steps:
1. prepare the test script as follow
## cat bz1464313.sh 
#!/bin/sh
device="/dev/sdb"
xml_file="diskpool.xml"
pool_name="bz1464313"
round=10
disk_xml="<pool type='disk'>
  <name>$pool_name</name>
  <source>
    <device path='$device'/>
  </source>
  <target>
  <path>/dev</path>
  </target>
</pool>"
case_failed=false

echo "$disk_xml" > $xml_file
echo -e "***** Running: virsh pool-define $xml_file *****"
virsh pool-define $xml_file
echo -e "***** Running: virsh pool-start $pool_name --overwrite *****"
virsh pool-start $pool_name --overwrite
echo -e "***** Running: service libvirtd restart *****"
service libvirtd restart
echo -e "***** Running: pool-start/destroy $pool_name for $round times *****"
for i in $(seq $round)
do
    virsh pool-start $pool_name && virsh pool-destroy $pool_name
    if [ $? -ne 0 ]; then
        case_failed=true
        break
    fi
done
echo -e "***** Running: virsh pool-undefine $pool_name *****"
virsh pool-undefine $pool_name
if [ "$case_failed" = true ]; then
    echo "FAILED"
else
    echo "PASSED"
fi



2. run the script with latest libvirt:
###################
## sh bz1464313.sh 
***** Running: virsh pool-define diskpool.xml *****
Pool bz1464313 defined from diskpool.xml

***** Running: virsh pool-start bz1464313 --overwrite *****
Pool bz1464313 started

***** Running: service libvirtd restart *****
Redirecting to /bin/systemctl restart libvirtd.service
***** Running: pool-start/destroy bz1464313 for 10 times *****
Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

Pool bz1464313 started

Pool bz1464313 destroyed

***** Running: virsh pool-undefine bz1464313 *****
Pool bz1464313 has been undefined

PASSED
###################


3. run the script with libvirt-3.2.0-7.el7.x86_64 in which the problem is not fixed yet
###################
# sh bz1464313.sh 
***** Running: virsh pool-define diskpool.xml *****
Pool bz1464313 defined from diskpool.xml

***** Running: virsh pool-start bz1464313 --overwrite *****
Pool bz1464313 started

***** Running: service libvirtd restart *****
Redirecting to /bin/systemctl restart  libvirtd.service
***** Running: pool-start/destroy bz1464313 for 10 times *****
Pool bz1464313 started

Pool bz1464313 destroyed

error: Failed to start pool bz1464313
error: Requested operation is not valid: Format of device '/dev/sdb' does not match the expected format 'unknown'

***** Running: virsh pool-undefine bz1464313 *****
Pool bz1464313 has been undefined

FAILED
###################

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704