Bug 1504241 - Images over NFS:Could not start storage pool: cannot open volume '/images/EFI': Transport endpoint is not connected
Summary: Images over NFS:Could not start storage pool: cannot open volume '/images/EFI...
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-19 19:17 UTC by Konrad Rzeszutek Wilk
Modified: 2020-05-05 21:31 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)
virt-install --debug .. blah blah 2>standard_output.txt (4.01 KB, text/plain)
2017-10-20 02:25 UTC, Konrad Rzeszutek Wilk
no flags Details

Description Konrad Rzeszutek Wilk 2017-10-19 19:17:20 UTC
Description of problem:

When trying to create a new guest I get the above error.


Version-Release number of selected component (if applicable):

[root@tst063 ~]# rpm -q virt-install
virt-install-1.4.3-1.fc27.noarch
[root@tst063 ~]# rpm -q virt-manager
virt-manager-1.4.3-1.fc27.noarch
[root@tst063 ~]# rpm -q libvirt
libvirt-3.7.0-2.fc27.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install Fedora-Server-dvd-x86_64-27-20171017.n.0.iso
2. # virt-install --hvm -l /images/Fedora-Server-dvd-x86_64-27-20171017.n.0.iso.iso --initrd-inject=/root/rawhide-64-hvm-kvm-guest.ks --extra-args  "console=ttyS0,115200 vnc ip=dhcp ks=http://192.168.106.1/rawhide-64-hvm-kvm-guest.ks" --graphics vnc,listen=0.0.0.0,port=5901 --video vga --serial file,path=/root/rawhide-64-hvm.serial --ram 8192 --disk /dev/fedora_tst063/rawhide-64-hvm --name rawhide-64-hvm --network=bridge=switch,mac=C0:EB:24:DE:56:B7
ERROR    Error validating install location: Validating install media '/images/Fedora-Server-dvd-x86_64-27-20171017.n.0.iso.iso' failed: Could not start storage pool: cannot open volume '/images/EFI': Transport endpoint is not connected
3. The same thing happens if I replace the ISO with Fedora-Server-dvd-x86_64-26-1.5.iso

Actual results:

ERROR    Error validating install location: Validating install media '/images/Fedora-Server-dvd-x86_64-27-20171017.n.0.iso.iso' failed: Could not start storage pool: cannot open volume '/images/EFI': Transport endpoint is not connected

Expected results:

Installing..

Additional info:

Comment 1 Cole Robinson 2017-10-19 20:13:12 UTC
Thanks for the report, please post the full virt-install --debug output when reproducing

Comment 2 Konrad Rzeszutek Wilk 2017-10-20 02:25:11 UTC
Created attachment 1341127 [details]
virt-install --debug .. blah blah 2>standard_output.txt

Comment 3 Cole Robinson 2017-10-20 14:19:36 UTC
Thanks, can you show the output of ls -alZ /images

Comment 4 Konrad Rzeszutek Wilk 2017-10-25 20:02:59 UTC
-rw-r--r--. 1 konrad konrad system_u:object_r:nfs_t:s0 2507145216 Oct  5 07:42 /images/Fedora-Server-dvd-x86_64-27-20171005.n.0.iso
-rw-r--r--. 1 konrad konrad system_u:object_r:nfs_t:s0 2493513728 Oct  8 08:22 /images/Fedora-Server-dvd-x86_64-27-20171008.n.0.iso
-rw-r--r--. 1 konrad konrad system_u:object_r:nfs_t:s0 2494562304 Oct 12 14:06 /images/Fedora-Server-dvd-x86_64-27-20171012.n.0.iso
-rw-r--r--. 1 konrad konrad system_u:object_r:nfs_t:s0 2496659456 Oct 17 08:35 /images/Fedora-Server-dvd-x86_64-27-20171017.n.0.iso

Comment 5 Cole Robinson 2017-11-20 12:30:34 UTC
So /images is NFS it seems. The 'transport endpoint is not connected' error seems weird here, I wonder if NFS is acting up. are you still seeing this?

Comment 6 Konrad Rzeszutek Wilk 2017-11-20 22:08:45 UTC
I played with setsebool for xen_use_nfs or virt_use_nfs and having it on or off did not make any change.

[root@tst063 ~]# rpm -q virt-manager
virt-manager-1.4.3-1.fc28.noarch
[root@tst063 ~]# 


I still get:

 Installing rawhide-64-hvm with C0:EB:24:DE:56:B7 --hvm -l /images_nfs/rawhide-64.iso --initrd-inject=/root/rawhide-64-hvm-xen-guest.ks
virt-install --hvm -l /images_nfs/rawhide-64.iso --initrd-inject=/root/rawhide-64-hvm-xen-guest.ks --extra-args console=ttyS0,115200 vnc ip=dhcp ks=http://192.168.106.1/rawhide-64-hvm-xen-guest.ks --graphics vnc,listen=0.0.0.0,port=5901 --video vga --serial file,path=/root/rawhide-64-hvm.serial --ram 8192 --disk /dev/fedora_tst063/rawhide-64-hvm --name rawhide-64-hvm --network=bridge=switch,mac=C0:EB:24:DE:56:B7
.[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (cli:264) Launched with command line: /usr/share/virt-manager/virt-install --hvm -l /images_nfs/rawhide-64.iso --initrd-inject=/root/rawhide-64-hvm-xen-guest.ks --debug --extra-args console=ttyS0,115200 vnc ip=dhcp ks=http://192.168.106.1/rawhide-64-hvm-xen-guest.ks --graphics vnc,listen=0.0.0.0,port=5901 --video vga --serial file,path=/root/rawhide-64-hvm.serial --ram 8192 --disk /dev/fedora_tst063/rawhide-64-hvm --name rawhide-64-hvm --network=bridge=switch,mac=C0:EB:24:DE:56:B7
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (cli:278) Requesting libvirt URI default
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (cli:281) Received libvirt URI xen:///
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (virt-install:358) Requesting virt method 'hvm', hv type 'default'.
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (virt-install:583) Received virt method 'xen'
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (virt-install:584) Hypervisor name is 'hvm'
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (virt-install:270) Distilled --network options: ['bridge=switch,mac=C0:EB:24:DE:56:B7']
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (virt-install:183) Distilled --disk options: ['/dev/fedora_tst063/rawhide-64-hvm']
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (hostkeymap:117) Found keymap=us in /etc/vconsole.conf
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (distroinstaller:191) Error validating install location
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtinst/distroinstaller.py", line 187, in _validate_location
    dev.validate()
  File "/usr/share/virt-manager/virtinst/devicedisk.py", line 872, in validate
    "non-existent path '%s'.") % self.path)
ValueError: Must specify storage creation parameters for non-existent path '/images_nfs/rawhide-64.iso'.
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (cli:316)   File "/usr/share/virt-manager/virt-install", line 1008, in <module>
    sys.exit(main())
  File "/usr/share/virt-manager/virt-install", line 996, in main
    guest = build_guest_instance(conn, options)
  File "/usr/share/virt-manager/virt-install", line 619, in build_guest_instance
    options.distro_variant)
  File "/usr/share/virt-manager/virt-install", line 405, in set_install_media
    fail(_("Error validating install location: %s") % str(e))
  File "/usr/share/virt-manager/virtinst/cli.py", line 316, in fail
    logging.debug("".join(traceback.format_stack()))

[Mon, 20 Nov 2017 17:06:26 virt-install 1988] ERROR (cli:317) Error validating install location: Validating install media '/images_nfs/rawhide-64.iso' failed: Must specify storage creation parameters for non-existent path '/images_nfs/rawhide-64.iso'.
[Mon, 20 Nov 2017 17:06:26 virt-install 1988] DEBUG (cli:319) 
Traceback (most recent call last):
  File "/usr/share/virt-manager/virt-install", line 395, in set_install_media
    guest.installer.location = (location or cdpath)
  File "/usr/share/virt-manager/virtinst/installer.py", line 80, in set_location
    self._location = self._validate_location(val)
  File "/usr/share/virt-manager/virtinst/distroinstaller.py", line 193, in _validate_location
    (str(val), e))
ValueError: Validating install media '/images_nfs/rawhide-64.iso' failed: Must specify storage creation parameters for non-existent path '/images_nfs/rawhide-64.iso'.

Comment 7 Konrad Rzeszutek Wilk 2017-11-20 22:20:44 UTC
Ditto for KVM:
 Installing rawhide-64-hvm with C0:EB:24:DE:56:B7 --hvm -l /images_nfs/rawhide-64.iso --initrd-inject=/root/rawhide-64-hvm-kvm-guest.ks
virt-install --hvm -l /images_nfs/rawhide-64.iso --initrd-inject=/root/rawhide-64-hvm-kvm-guest.ks --extra-args console=ttyS0,115200 vnc ip=dhcp ks=http://192.168.106.1/rawhide-64-hvm-kvm-guest.ks --graphics vnc,listen=0.0.0.0,port=5901 --video vga --serial file,path=/root/rawhide-64-hvm.serial --ram 8192 --disk /dev/fedora_tst063/rawhide-64-hvm --name rawhide-64-hvm --network=bridge=switch,mac=C0:EB:24:DE:56:B7
.[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (cli:264) Launched with command line: /usr/share/virt-manager/virt-install --hvm -l /images_nfs/rawhide-64.iso --initrd-inject=/root/rawhide-64-hvm-kvm-guest.ks --debug --extra-args console=ttyS0,115200 vnc ip=dhcp ks=http://192.168.106.1/rawhide-64-hvm-kvm-guest.ks --graphics vnc,listen=0.0.0.0,port=5901 --video vga --serial file,path=/root/rawhide-64-hvm.serial --ram 8192 --disk /dev/fedora_tst063/rawhide-64-hvm --name rawhide-64-hvm --network=bridge=switch,mac=C0:EB:24:DE:56:B7
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (cli:278) Requesting libvirt URI default
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (cli:281) Received libvirt URI qemu:///system
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (virt-install:358) Requesting virt method 'hvm', hv type 'default'.
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (virt-install:583) Received virt method 'kvm'
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (virt-install:584) Hypervisor name is 'hvm'
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (virt-install:270) Distilled --network options: ['bridge=switch,mac=C0:EB:24:DE:56:B7']
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (virt-install:183) Distilled --disk options: ['/dev/fedora_tst063/rawhide-64-hvm']
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (distroinstaller:191) Error validating install location
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtinst/distroinstaller.py", line 187, in _validate_location
    dev.validate()
  File "/usr/share/virt-manager/virtinst/devicedisk.py", line 872, in validate
    "non-existent path '%s'.") % self.path)
ValueError: Must specify storage creation parameters for non-existent path '/images_nfs/rawhide-64.iso'.
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (cli:316)   File "/usr/share/virt-manager/virt-install", line 1008, in <module>
    sys.exit(main())
  File "/usr/share/virt-manager/virt-install", line 996, in main
    guest = build_guest_instance(conn, options)
  File "/usr/share/virt-manager/virt-install", line 619, in build_guest_instance
    options.distro_variant)
  File "/usr/share/virt-manager/virt-install", line 405, in set_install_media
    fail(_("Error validating install location: %s") % str(e))
  File "/usr/share/virt-manager/virtinst/cli.py", line 316, in fail
    logging.debug("".join(traceback.format_stack()))

[Mon, 20 Nov 2017 17:19:52 virt-install 1444] ERROR (cli:317) Error validating install location: Validating install media '/images_nfs/rawhide-64.iso' failed: Must specify storage creation parameters for non-existent path '/images_nfs/rawhide-64.iso'.
[Mon, 20 Nov 2017 17:19:52 virt-install 1444] DEBUG (cli:319) 
Traceback (most recent call last):
  File "/usr/share/virt-manager/virt-install", line 395, in set_install_media
    guest.installer.location = (location or cdpath)
  File "/usr/share/virt-manager/virtinst/installer.py", line 80, in set_location
    self._location = self._validate_location(val)
  File "/usr/share/virt-manager/virtinst/distroinstaller.py", line 193, in _validate_location
    (str(val), e))
ValueError: Validating install media '/images_nfs/rawhide-64.iso' failed: Must specify storage creation parameters for non-existent path '/images_nfs/rawhide-64.iso'.

Comment 8 Cole Robinson 2017-11-21 20:36:52 UTC
In those last examples the root directory changing /images -> /images_nfs, and the error message changed 'transport not connected' (from libvirt) to file non-existent.

Can you provide ls -lZ for /images_nfs

Comment 9 Christophe de Dinechin 2018-10-20 18:27:47 UTC
(In reply to Cole Robinson from comment #8)
> In those last examples the root directory changing /images -> /images_nfs,
> and the error message changed 'transport not connected' (from libvirt) to
> file non-existent.
> 
> Can you provide ls -lZ for /images_nfs

Saw the same problem:

[root@virtlab605 ~]# ls -lZ /isos/
total 7138336
-rw-r--r--. 1 nobody nobody unconfined_u:object_r:unlabeled_t:s0 1747976192 Apr 25 02:49 Fedora-Workstation-Live-i386-28-1.1.iso
-rw-r--r--. 1 nobody nobody unconfined_u:object_r:unlabeled_t:s0 1865416704 Sep 19 22:45 Fedora-Workstation-Live-i386-29_Beta-1.5.iso
-rw-r--r--. 1 nobody nobody unconfined_u:object_r:unlabeled_t:s0 1787822080 Apr 25 02:45 Fedora-Workstation-Live-x86_64-28-1.1.iso
-rw-r--r--. 1 nobody nobody unconfined_u:object_r:unlabeled_t:s0 1908408320 Sep 19 22:45 Fedora-Workstation-Live-x86_64-29_Beta-1.5.iso
drwx------. 2 root   root   system_u:object_r:unlabeled_t:s0          16384 Oct  4 08:48 lost+found

Comment 10 Christophe de Dinechin 2018-10-29 12:25:59 UTC
Insterestingly, removing /isos/lost+found fixed it in my case. So apparently, it's a case of libvirt trying too hard to open subdirectories.

Comment 11 Cole Robinson 2018-11-12 14:39:39 UTC
Sounds like libvirt should ignore the error in this case and continue on, we do that for several other conditions for volume probing IIRC

Comment 12 Christian Kujau 2019-08-22 19:12:08 UTC
Still happening on Fedora 30:

$ rpm -qf `which virt-install`
virt-install-2.1.0-2.fc30.noarch

$ virt-install --name ubuntu0 --cdrom /mnt/nfs/data/incoming/ubuntu-19.04-server-amd64.iso
ERROR    Validating install media '/mnt/nfs/data/incoming/ubuntu-19.04-server-amd64.iso' failed: Could not start storage pool: cannot open volume '/mnt/nfs/data/incoming/.config': Transport endpoint is not connected

$ ls -lZd /mnt/{,nfs/{,data{,/incoming/{,.config,ubuntu-19.04-server-amd64.iso}}}}
drwxr-xr-x.  7 root root system_u:object_r:mnt_t:s0            63 Feb 11  2019 /mnt/
drwxr-xr-x.  4 root root unconfined_u:object_r:mnt_t:s0        31 Dec 19  2017 /mnt/nfs/
drwxr-xr-x. 12 root root system_u:object_r:nfs_t:s0          4096 Mar 20  2018 /mnt/nfs/data
drwxr-xr-x.  9  169  169 system_u:object_r:nfs_t:s0          4096 Aug 22 11:44 /mnt/nfs/data/incoming/
drwx--x---.  3  169  169 system_u:object_r:nfs_t:s0            41 Apr 10  2017 /mnt/nfs/data/incoming/.config
-rw-r--r--.  1  169  169 system_u:object_r:nfs_t:s0     790626304 Aug 22 11:43 /mnt/nfs/data/incoming/ubuntu-19.04-server-amd64.iso

As soon as any non-readable file or directory (here: ".config") in the /mnt/nfs/data/incoming/ is made accessible to the user, virt-install starts up OK. And it seems to be related to NFS as I couldn't reproduce it with a local directory.


Note You need to log in before you can comment on or make changes to this bug.