Bug 1223668 - [RFE] add a directory structure allowing windows automatically finding drivers
Summary: [RFE] add a directory structure allowing windows automatically finding drivers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: virtio-win
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Danilo Cesar Lemes de Paula
QA Contact: lijin
Yehuda Zimmerman
URL:
Whiteboard:
: 1217658 (view as bug list)
Depends On:
Blocks: 1721694 1725380
TreeView+ depends on / blocked
 
Reported: 2015-05-21 07:50 UTC by Gerd Hoffmann
Modified: 2019-09-23 20:02 UTC (History)
17 users (show)

Fixed In Version: virtio-win-1.9.8-2.el8
Doc Type: Enhancement
Doc Text:
.Windows automatically finds the needed virtio-win drivers Windows can now automatically find the virtio-win drivers it needs from the driver ISO without requiring the user to select the folder in which they are located.
Clone Of:
: 1721694 (view as bug list)
Environment:
Last Closed: 2019-07-30 14:22:06 UTC
Type: Feature Request
Target Upstream Version:


Attachments (Terms of Use)
post-processing script (1.12 KB, text/plain)
2015-05-21 07:52 UTC, Gerd Hoffmann
no flags Details
win8-32 can not load drivers automatically (69.41 KB, image/png)
2019-05-23 01:28 UTC, lijin
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:1997 None None None 2019-07-30 14:22:45 UTC

Description Gerd Hoffmann 2015-05-21 07:50:16 UTC
Description of problem:
Installing windows guest drivers from virtio-win.iso requires manually picking the correct driver.

Version-Release number of selected component (if applicable):
virtio-win-1.7.3-1.el7.noarch

How reproducible:
Install windows guest on virtio storage, using virtio-win.iso instead of the floppy images as source.

Actual results:
Windows asks for the driver location.

Expected results:
Windows finds the drivers on its own (like it is the case with the floppy drivers).

Additional info:

Comment 1 Gerd Hoffmann 2015-05-21 07:52:52 UTC
Created attachment 1028006 [details]
post-processing script

Script which takes the virtio-win.iso, unpacks it, adds the needed directories, links files, creates new iso.

Take care, hard coded paths in there, not to be used as-is.  But it's probably easier to describe the directory tree requirements this way ;)

Comment 4 Jeff Nelson 2015-10-15 19:12:18 UTC
The script in Comment #1 basically reorganizes the directory structure by removing the top directory (the driver name) and swapping the two subdirectories underneath (os, arch). For example:

NetKVM/w7/amd64/netkvm.inf  -->  amd64/w7/netkvm.inf

In addition, the original x86 directory is changed to i386. For example:

vioscsi/2k8/x86/vioscsi.inf --> i386/2k8/vioscsi.inf

The script creates the new directory structure and hard links to the files; there is no change to the existing structure and there is no size increase due to the fact that hard links are used.

Some design questions I have:

* Is there a need to maintain the original directory structure, or should we abandon it?

* The script only reorganizes the structure for some drivers, not all. Why are some processed and others are not? I'd like to understand the factors that go into making this determination. This also comes into play when a new driver shows up.

Thanks.

Comment 6 Gerd Hoffmann 2015-10-16 08:06:42 UTC
> * Is there a need to maintain the original directory structure, or should we
> abandon it?

I've decided to play safe and keep the existing structure, additionally to the new one.  I have no idea whenever there is any software which expects to find the current layout on the cdrom.

There have been some name clashes due to the driver directory level being gone.  IIRC the docs are not hard-linked in the new structure because of that.

Maybe the driver directory can be re-added, possibly windows does a full-tree search below $arch, don't remember whenever I've tried that or not.

> * The script only reorganizes the structure for some drivers, not all. Why
> are some processed and others are not? I'd like to understand the factors
> that go into making this determination. This also comes into play when a new
> driver shows up.

Tried to keep the new tree as small as possible.  Of course that point is moot when everything is migrated over into the new layout and the old is dropped.

Included network and storage drivers as these are required to successfully install windows.

Excluded old windows versions as they can't load drivers from cdrom anyway.

Comment 9 Gerd Hoffmann 2015-10-19 07:48:35 UTC
> Included network and storage drivers as these are required to successfully
> install windows.

... and those are also the drivers on the vdf images (same reason ;)

Comment 13 Cole Robinson 2016-02-17 19:15:10 UTC
Note there's some discussion in this direction with the shared packaging scripts:

https://www.redhat.com/archives/virt-tools-list/2016-February/msg00029.html
https://www.redhat.com/archives/libguestfs/2015-October/msg00325.html

And some RHEV discussion about moving off of .vfd images:

https://bugzilla.redhat.com/show_bug.cgi?id=1306586

I'm trying to spec out all the changes needed since there's several other related changes I want to make in this area. I'll triple check with relevant parties before pushing anything.

Comment 25 Cole Robinson 2018-08-08 23:17:03 UTC
*** Bug 1217658 has been marked as a duplicate of this bug. ***

Comment 28 ybendito 2019-02-18 09:35:14 UTC
Indeed, in order to improve drivers installation (viostor and vioscsi) during Windows Setup in case the user must have these drivers to let Windows find the local drive - it seems there is no other solution, i.e. the drivers must be in amd64/xx directories (or i386/xx directories).

Do we need also NetKVM drivers to be installed at this stage? Currently with only viostor and vioscsi the user has list of 18 drivers, with NetKVM it will be list of 27 drivers.

If the user needs to install drivers after the OS is up, due to OS specifier in each INF file the system will find proper drivers if the user points on the root of the CD (comment #23).
NetKVM, serial, balloon can be easily installed later, correct?

If we just add additional directories amd64 and i386 the root directory of the driver's ISO will look very non-intuitive:
It will contain:
Balloon NetKVM amd64 guest-agent i386 pvpanic qemufwcfg qemupciserial qxl smbus vioinput viorng vioscsi vioserial viostor

I would suggest to change the structure of the ISO to have in root directory: 'amd64', 'i386' and 'Drivers' with all the driver specific directories under 'Drivers'.
Is there some SW components that depends on exact structure, i.e. is it MUST to have backward-compatible structure?

Comment 29 Cole Robinson 2019-02-18 13:51:54 UTC
IMO we really can't remove anything from the ISO layout, not at least without a long transition period. There are definitely users programmatically accessing files on the ISO. libguestfs could historically do it, ovirtinstaller build process does AFAIK, but it's really the unknown number of external users that is the problem. But it should be fine to add additional amd64/i386/arm64. ISOs support hard links so it shouldn't increase size too much

Comment 30 ybendito 2019-02-20 09:59:26 UTC
We've discussed the next step and agreed on following:
- the layout shall stay backward-compatible
- amd64 and i386 are to be added like it is done in the script for viostor and vioscsi
- we do not see the need to do that for NetKVM (if somebody does, please respond)

We (virtio-win team) do not maintain the ISO build, who is going to take ownership on the BZ?

Comment 31 Gerd Hoffmann 2019-02-20 11:28:14 UTC
> - we do not see the need to do that for NetKVM (if somebody does, please
> respond)

FYI: IIRC I've added all drivers which are on the floppy images (assuming those are considered essential) to the amd64/i386 directories when writing the comment 1 script. No clue why NetKVM is included there though.  Maybe just for convinience.  Or for historical reasons (it is one of the oldest drivers after all).  Can't see any problems with installing NetKVM drivers later.

Comment 33 ybendito 2019-03-10 15:09:07 UTC
Submitted pull request
https://github.com/crobinso/virtio-win-pkg-scripts/pull/16

Comment 34 ybendito 2019-03-12 08:45:55 UTC
Merged upstream
ISO from build 164 created by Cole Robinson:
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.164-2/

Comment 37 Danilo Cesar Lemes de Paula 2019-03-15 01:48:30 UTC
It seems something I can import directly from Fedora, if I got this right.
So OK from my POV.

Comment 39 Danilo Cesar Lemes de Paula 2019-04-02 16:20:13 UTC
Yes!

But I'm not recognizing the product. Virtualization Tools.
Should it be regular RHEL?

Comment 45 ybendito 2019-04-10 16:10:36 UTC
This does not change the layout of zip, some folder added only when making the drivers ISO.

Comment 49 lijin 2019-05-23 01:27:03 UTC
Hi Danilo,

I tryied with virtio-win-1.9.8-0.el8, attach vitio-win iso and select virtio-win cdrom after load driver.
But it shows "No device drivers were found"(as attachment shows).
I still need to select to the specified path of scsi driver.

Could you help to confirm if my step is correct?

qemu cli:

/usr/libexec/qemu-kvm \
    -monitor stdio  \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -device pvpanic,ioport=0x505,id=idgfg5iI  \
    -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/test.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -m 30720  \
    -smp 32,cores=32  \
    -cpu 'EPYC',hv_stimer,hv_synic,hv_vpindex,hv_reset,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv-tlbflush,+kvm_pv_unhalt \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/usr/share/virtio-win/virtio-win.iso \
    -device ide-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :10  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -qmp tcp:localhost:4444,server,nowait \
    -device virtio-net-pci,mac=9a:f6:f7:f8:f9:fa,id=net0,netdev=hostnet0,bus=pci.0,addr=0x5,mq=on,vectors=66 \
    -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup,queues=32 \
    -cdrom /home/kvm_autotest_root/iso/ISO/Win8/en_windows_8_enterprise_x86_dvd_917587.iso

Comment 50 lijin 2019-05-23 01:28:36 UTC
Created attachment 1572209 [details]
win8-32 can not load drivers automatically

Comment 51 Danilo Cesar Lemes de Paula 2019-05-23 15:55:10 UTC
(In reply to lijin from comment #49)
> Hi Danilo,
> 
> I tryied with virtio-win-1.9.8-0.el8, attach vitio-win iso and select
> virtio-win cdrom after load driver.
> But it shows "No device drivers were found"(as attachment shows).
> I still need to select to the specified path of scsi driver.
> 
> Could you help to confirm if my step is correct?
> 
> qemu cli:
> 
> /usr/libexec/qemu-kvm \
>     -monitor stdio  \
>     -name 'avocado-vt-vm1' \
>     -machine pc  \
>     -nodefaults \
>     -device VGA,bus=pci.0,addr=0x2  \
>     -device pvpanic,ioport=0x505,id=idgfg5iI  \
>     -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
>     -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/home/test.qcow2 \
>     -device scsi-hd,id=image1,drive=drive_image1 \
>     -m 30720  \
>     -smp 32,cores=32  \
>     -cpu
> 'EPYC',hv_stimer,hv_synic,hv_vpindex,hv_reset,hv_relaxed,hv_spinlocks=0x1fff,
> hv_vapic,hv_time,hv-tlbflush,+kvm_pv_unhalt \
>     -drive
> id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/
> usr/share/virtio-win/virtio-win.iso \
>     -device ide-cd,id=cd1,drive=drive_cd1 \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
>     -vnc :10  \
>     -rtc base=localtime,clock=host,driftfix=slew  \
>     -boot order=cdn,once=c,menu=off,strict=off \
>     -enable-kvm \
>     -qmp tcp:localhost:4444,server,nowait \
>     -device
> virtio-net-pci,mac=9a:f6:f7:f8:f9:fa,id=net0,netdev=hostnet0,bus=pci.0,
> addr=0x5,mq=on,vectors=66 \
>     -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup,queues=32 \
>     -cdrom
> /home/kvm_autotest_root/iso/ISO/Win8/en_windows_8_enterprise_x86_dvd_917587.
> iso

I don't have a windows machine to test.

We're using all the upstrema bits. I will discuss this with Cole Robinson to find out what's wrong.

Comment 52 ybendito 2019-05-23 17:06:28 UTC
I did not test it with win8 (did the test with several other client and server systems).
Will check with win8.
Li Jin, can you please share more info:
- with which OS this directory structure was tested?
- with which other OS this does not work?

Comment 53 lijin 2019-05-24 02:15:02 UTC
(In reply to ybendito from comment #52)
> I did not test it with win8 (did the test with several other client and
> server systems).
> Will check with win8.
> Li Jin, can you please share more info:
> - with which OS this directory structure was tested?
> - with which other OS this does not work?

I tested on win7-32/64, win8-32/64, win10-32, win2012 and win2019, all of them do not work.

And I tried with ISO from build 164 created by Cole Robinson[1], it works better.
The difference betweenn Cole's iso and virtio-win-1.9.8-0.el8 is there are three extra folders: amd64, i386, ARM64
# ls amd64/
2k8  w10  w7  w8
# ls i386/
2k8  w10  w7  w8
# ls ARM64/
w10

Under these folders, scsi and blk drivers are included.
When I load driver during OS installation, I still need to select the path to E:\i386\w10, "E:" or "E:\i386\" do not work
Is this behavior your expectation?

btw, we need to add more windows guests for w8.1/2k8r2/2k12/2k12r2/2k19


[1]https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.164-2/

Comment 54 ybendito 2019-05-24 08:44:14 UTC
> The difference betweenn Cole's iso and virtio-win-1.9.8-0.el8 is there are
> three extra folders: amd64, i386, ARM64
Indeed, Fedora 164-2 build (by Col) has properly added drivers/folders for automatic installation (it is build by upstream scripts)

It looks like downstream build procedure is not changed:
virtio-win-1.9.8-0.el8 does not have these folders. In the spec file I see that it is built by just mkisofs without creation of links for automatic driver installation.

Comment 55 Danilo Cesar Lemes de Paula 2019-05-25 00:23:15 UTC
@Amnon, There's a difference between how Fedora and Downstream cook the drivers folder.
That's why even if we use the upstream script (with the fix), it won't work downstream.

I discussed this with Cole today and he was trying out some changes.
In worst case I run a script to manually create the required folders, but let me revisit this on monday.

Comment 56 Danilo Cesar Lemes de Paula 2019-05-25 00:23:52 UTC
Meanwhile I'm moving this back to assigned because it's not fixed downstream yet.

Comment 58 Danilo Cesar Lemes de Paula 2019-05-28 16:05:39 UTC
So, after the discussion with Cole here's what I did:

I updated the signed_win_drivers repository containing a script that generates the amd64 and i386 folders.
Then I re-ran the make-virtio-win-rpm-archive.py (from virtio-win-pkg-scripts) to include that folder in the virtio-win-1.9.8-bin-for-rpm.tar.gz
Then I regenerated the rpm (as a scratch build), which forces the iso to be reconstructed.

I've opened the iso and it now contains the same files from comment 53 (except the ARM folder).

Li Jin, can you please check https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=21865888 ?

Comment 59 Danilo Cesar Lemes de Paula 2019-05-29 15:54:12 UTC
Li Jin, can you pease test virtio-win-1.9.8-1.el8?

Comment 60 Danilo Cesar Lemes de Paula 2019-05-29 15:54:29 UTC
Li Jin, can you please test virtio-win-1.9.8-1.el8?

Comment 61 lijin 2019-05-31 01:16:23 UTC
(In reply to Danilo Cesar de Paula from comment #60)
> Li Jin, can you please test virtio-win-1.9.8-1.el8?

It works, but only w7,w8,w10 is included, how about other platforms?

Comment 62 ybendito 2019-05-31 08:40:47 UTC
Li Jin, can you please reply which platforms was tested and which ones do not work?

Comment 63 Danilo Cesar Lemes de Paula 2019-05-31 13:58:15 UTC
I'm doing this for the same platforms that upstream does (except arm).

@Yuri, as the one that made the upstream changes, is there any reason to do not include other platforms?

Comment 64 lijin 2019-06-03 00:19:19 UTC
(In reply to ybendito from comment #62)
> Li Jin, can you please reply which platforms was tested and which ones do
> not work?

I tested on w7,w8,w10 which are included in i386 and amd64 folder, and testing is passed.
Other OS are not tested as they are not included.

Comment 68 Danilo Cesar Lemes de Paula 2019-06-04 00:14:55 UTC
So it's agreed that adding more folders to the /drivers in the ISO is the way to go?
Which one? All?

I can update the create_synlink script to regenerate the package.

Comment 69 ybendito 2019-06-04 08:01:59 UTC
(In reply to Danilo Cesar de Paula from comment #68)
> So it's agreed that adding more folders to the /drivers in the ISO is the
> way to go?
> Which one? All?
> 
> I can update the create_synlink script to regenerate the package.

Yes, adding folders to amd64 and i386 (where applicable) at the root of the ISO, like that:
amd64/2k16 to contain links to all the files from viostor/2k16/amd64 and from vioscsi/2k16/amd64
i386/w8.1 to contain links to all the files from viostor/w8.1/x86 and from vioscsi/w8.1/x86 
For all the subfolders of vioscsi and viostor

Comment 71 Danilo Cesar Lemes de Paula 2019-06-07 19:28:54 UTC
I did that for the drivers we support:

'2k12'
'2k12R2'
'2k16'
'2k19'
'2k8R2'
'w10'
'w7'
'w8'
'w8.1'

2k8, 2k3 and xp were not included as they are not supported anymore.

Comment 72 lijin 2019-06-10 06:11:43 UTC
Check in virtio-win-1.9.8-2.el8.noarch, following folders are added, and drivers can be loaded correctly

# ls amd64/
2k12  2k12R2  2k16  2k19  2k8R2  w10  w7  w8  w8.1

# ls i386/
w10  w7  w8  w8.1

So change status to verified.

Comment 75 errata-xmlrpc 2019-07-30 14:22:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1997

Comment 76 Danilo Cesar Lemes de Paula 2019-08-13 14:49:52 UTC
Done. Please take a look if it fits your needs.

Comment 78 Danilo Cesar Lemes de Paula 2019-08-14 14:00:35 UTC
Looks good to me.


Note You need to log in before you can comment on or make changes to this bug.