Bug 2229017

Summary: anaconda Storage module times out when installing via virt-manager
Product: [Fedora] Fedora Reporter: Albert Stone <ahs3>
Component: anacondaAssignee: anaconda-maint
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 38CC: anaconda-maint, dan, vslavik, w
Target Milestone: ---   
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
tar file of logs from f36 installation attempt
none
tar file of logs from f38 installation attempt none

Description Albert Stone 2023-08-03 23:10:32 UTC
Libvirtd is running on an arm64 system (latest Fedora 38), with remote usage via ssh enabled.  From an x86_64 laptop running Fedora 38, I started virt-manager and tried to create a s390x VM from the F38 GA image from alt.fedoraproject.org.  There are a number of "access denied" messages from auditd, which seems problematic, but eventually the installer fails with the following:

anaconda 36.16.5-1.fc36 for Fedora 36 started.
 * installation log files are stored in /tmp during the installation
 * shell is available in second TMUX pane (ctrl+b, then press 2)
 * when reporting a bug add logs from /tmp as separate text/plain attachments
22:33:41 Service org.fedoraproject.Anaconda.Modules.Storage has failed to start: Failed to activate service 'org.fedoraproject.Anaconda.Modules.Storage': timed out (service_start_timeout=600000ms)

At that point, the installer terminates.  So, I'm kind of stuck; I can get access to an s390x machine, but I should also be able to install a VM from the ISO and I cannot.

Please note that while this cut'n'paste says "Fedora 36", this is the exact same message received with Fedora 38 (version numbers change, but that's it).  I used this text just to demonstrate that this problem seems to have been present for a while since I can replicate even with f36.

Reproducible: Always

Steps to Reproduce:
1.  Download the f38 GA image (or f36, even) for s390x.
2.  Use virt-manager to create a new VM using the downloaded ISO; note that I ask for 4GB 
of RAM, 2 CPUs, and 40GB of disk space when creating the VM, but use default NAT networking.
3.  Wait for the installer to start, then stop with the message in the description.
Actual Results:  
anaconda 36.16.5-1.fc36 for Fedora 36 started.
 * installation log files are stored in /tmp during the installation
 * shell is available in second TMUX pane (ctrl+b, then press 2)
 * when reporting a bug add logs from /tmp as separate text/plain attachments
22:33:41 Service org.fedoraproject.Anaconda.Modules.Storage has failed to start: Failed to activate service 'org.fedoraproject.Anaconda.Modules.Storage': timed out (service_start_timeout=600000ms)

Expected Results:  
A dialog with the anaconda installer so I can set all the necessary parameters for the install, followed by a successful installation to the VM.

Comment 1 Albert Stone 2023-08-03 23:16:33 UTC
Created attachment 1981529 [details]
tar file of logs from f36 installation attempt

Comment 2 Albert Stone 2023-08-03 23:19:00 UTC
(In reply to Albert Stone from comment #1)
> Created attachment 1981529 [details]
> tar file of logs from f36 installation attempt

I'll attach f38 logs (or someone will) as soon as I can retrieve them.

Comment 3 Albert Stone 2023-08-04 15:51:03 UTC
Created attachment 1981671 [details]
tar file of logs from f38 installation attempt

Comment 4 Dan Horák 2023-08-04 16:04:03 UTC
For the record I can reproduce the issue when installing s390x rawhide ISO on a ppc64le host, but not for F-36 image. The full system emulation is used by people wanting access other platforms for development/debugging.

https://github.com/rhinstaller/anaconda/blob/master/data/dbus/anaconda-bus.conf#L115 is the location of the timeout definition

Comment 5 Dan Horák 2023-08-04 16:07:25 UTC
Not sure we want to change the default, but a way to override (eg. via some inst.foo parameter) it would be great.

Comment 6 Dan Horák 2023-08-04 16:41:04 UTC
a workaround could be using an updates image, please try adding inst.updates=http://fedora.danny.cz/s390/timeout.img to the kernel parameters when booting the installer ISO. The image contain an updated /usr/share/anaconda/dbus/anaconda-bus.conf file.

Comment 7 Albert Stone 2023-08-07 16:03:31 UTC
(In reply to Dan Horák from comment #6)
> a workaround could be using an updates image, please try adding
> inst.updates=http://fedora.danny.cz/s390/timeout.img to the kernel
> parameters when booting the installer ISO. The image contain an updated
> /usr/share/anaconda/dbus/anaconda-bus.conf file.

Forgive my ignorance here ... how does one do this?  Virt-manager is not providing access to a grub menu nor do I see anywhere to add it to boot options; do I need to extract the kernel and initrd from the ISO and then add the parameter so I can direct boot a kernel in virt-manager?  Or, just point me at a man page :).

Thanks.

Comment 8 Dan Horák 2023-08-07 16:32:17 UTC
Good question :-) And I am not sure there is an easy way with virt manager, but manually editing the XML definition before the installation start might allow it (there is a "review options" check box on the last screen of the wizard). Definitely it's possible when using virt-install (the command line tool) and its --extra-args.

something like: virt-install --name f38 --memory 4096 --vcpus 2 --disk size=10 --graphics none --location https://kojipkgs.fedoraproject.org/compose/38/latest-Fedora-38/compose/Everything/s390x/os/ --extra-args="inst.updates=http://fedora.danny.cz/s390/timeout.img"

it will need the proper "connection string" to your remote host

Comment 9 Dan Horák 2023-08-07 16:46:59 UTC
I believe direct boot with kernel/initrd extracted from the ISO (or downloaded) would be equivalent to what the virt-install is doing ...

Comment 10 Albert Stone 2023-08-07 21:56:58 UTC
(In reply to Dan Horák from comment #8)
> Good question :-) And I am not sure there is an easy way with virt manager,
> but manually editing the XML definition before the installation start might
> allow it (there is a "review options" check box on the last screen of the
> wizard). Definitely it's possible when using virt-install (the command line
> tool) and its --extra-args.
> 
> something like: virt-install --name f38 --memory 4096 --vcpus 2 --disk
> size=10 --graphics none --location
> https://kojipkgs.fedoraproject.org/compose/38/latest-Fedora-38/compose/
> Everything/s390x/os/
> --extra-args="inst.updates=http://fedora.danny.cz/s390/timeout.img"
> 
> it will need the proper "connection string" to your remote host

Confirmed.  The longer timeout lets the installer start properly.  Not
sure why it's so slow, but the workaround does work.