Steps to reproduce continued:

4. Click on the Finish button anyway,
 I get a dialog asking if I want to start the network now
5. Click on yes
 I get a new dialog containing

Could not start virtual network 'default': internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper) unexpected exit status 1: 
dnsmasq: unknown user or group: dnsmasq


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/device/netlist.py", line 310, in _check_network_is_running
    netobj.start()
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/network.py", line 75, in start
    self._backend.create()
  File "/usr/lib64/python3.7/site-packages/libvirt.py", line 3068, in create
    if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self)
libvirt.libvirtError: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper) unexpected exit status 1: 
dnsmasq: unknown user or group: dnsmasq

Created attachment 1666149 [details]
journal

first instance is during startup:


[    9.094867] fmac.local libvirtd[970]: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper) unexpected exit status 1:

And then later when I try to create the machine

[ 8116.089262] fmac.local dnsmasq[3678]: unknown user or group: dnsmasq
[ 8116.089383] fmac.local dnsmasq[3678]: FAILED to start up
[ 8116.090149] fmac.local libvirtd[3526]: internal error: Child process (VIR_BRIDGE_NAME=virbr0 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper) unexpected exit status 1: 
                                          dnsmasq: unknown user or group: dnsmasq
[ 8116.090620] fmac.local avahi-daemon[818]: Interface virbr0.IPv4 no longer relevant for mDNS.

Created attachment 1666153 [details]
libvirtd.log

Just in case it's not already obvious ...

Edit /etc/libvirt/libvirtd.conf and set

    log_filters="1:firewall 1:network 1:iptables"
    log_outputs="1:file:/var/log/libvirt/libvirtd.log"

Proposed as a Blocker for 32-beta by Fedora user chrismurphy using the blocker tracking app because:

 Beta: The release must be able host virtual guest instances of the same release. 
https://fedoraproject.org/wiki/Fedora_32_Beta_Release_Criteria#Self_hosting_virtualization

Can't host a virtual instance if I can't create a machine to host in it.

/etc/passwd does not contain dnsmasq user, should it? A Rawhide system clean installed before branch does contain it, but I don't know what date.

dnsmasq-2.80-11.fc32.x86_64

F32
$ id dnsmasq
id: ‘dnsmasq’: no such user

F31
$ id dnsmasq
uid=987(dnsmasq) gid=987(dnsmasq) groups=987(dnsmasq)

Maybe related to this?
https://fedoraproject.org/wiki/Changes/Adopting_sysusers.d_format

I don't think this is related to the Change in any way, because sysusers
uses systemd-sysusers directly. dnsmasq has (and has had for a while) the
following %preinstall scriptlet:

systemd-sysusers --replace=/usr/lib/sysusers.d/dnsmasq.conf - <<SYSTEMD_INLINE_EOF >/dev/null 2>&1 || : 
u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq 
SYSTEMD_INLINE_EOF

And on my system, the user gets created properly. And even if it wasn't
created properly, it should get created after reboot. Can you run
'sudo SYSTEMD_LOG_LEVEL=debug systemd-sysusers'?

Created attachment 1666627 [details]
journalctl -b -o short-monotonic, with log_level debug

$ sudo SYSTEMD_LOG_LEVEL=debug systemd-sysusers
Successfully loaded SELinux database in 1.692ms, size on heap is 337K.
Reading config file "/usr/lib/sysusers.d/basic.conf"…
Reading config file "/usr/lib/sysusers.d/dbus.conf"…
Reading config file "/usr/lib/sysusers.d/dnsmasq.conf"…
Reading config file "/usr/lib/sysusers.d/systemd.conf"…
Group adm already exists.
Group wheel already exists.
Group kmem already exists.
Group tty already exists.
Group utmp already exists.
Group audio already exists.
Group cdrom already exists.
Group dialout already exists.
Group disk already exists.
Group input already exists.
Group kvm already exists.
Group lp already exists.
Group render already exists.
Group tape already exists.
Group video already exists.
Group users already exists.
Group systemd-journal already exists.
Group root already exists.
User root already exists.
Group nobody already exists.
User nobody already exists.
Group dbus already exists.
User dbus already exists.
Creating group dnsmasq with gid 976.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 976 and gid 976.
Group systemd-network already exists.
User systemd-network already exists.
Group systemd-resolve already exists.
User systemd-resolve already exists.
Group systemd-timesync already exists.
User systemd-timesync already exists.
Group systemd-coredump already exists.
User systemd-coredump already exists.
$ id dnsmasq
uid=976(dnsmasq) gid=976(dnsmasq) groups=976(dnsmasq)

This bug is reproducible on baremetal and VM with a clean install of Fedora-Workstation-Live-x86_64-32-20200226.n.0.iso.

Running 'systemd-sysusers' post-install does persistently fix the problem (dnsmasq user doesn't exist), and resolves the side effect (libvirt won't start VMs).

RE: the blocker bug question

GNOME Boxes uses libvirt too, however it doesn't use NAT: '-netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=' and gets an IP assigned by local AP's DHCP service. And therefore it works fine, so its separate final release criterion (default application functionality) isn't triggered.

I'm not certain this is a blocker bug. virt-manager isn't installed by default, and the beta release criteria's "Virtualization requirements" only says that virtualization must work. Since GNOME Boxes does work, virtualization is working. Could be a gray area.

(In reply to Chris Murphy from comment #10)
> I'm not certain this is a blocker bug. virt-manager isn't installed by
> default, and the beta release criteria's "Virtualization requirements" only
> says that virtualization must work. Since GNOME Boxes does work,
> virtualization is working. Could be a gray area.

My vote is -1 blocker, since Boxes is our virtualization app, and if Boxes works that means virtualization is working....

Discussed during the 2020-03-02 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following criterion:

"The release must be able host virtual guest instances of the same release...This criterion applies only to the recommended Fedora virtualization tools - the qemu/kvm - libvirt - virt-manager stack."

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-03-02/f32-blocker-review.2020-03-02-17.00.txt

Moving to dnsmasq for further triage, which is responsible for adding the user at package install time.

There's been no material changes to dnsmasq since the f31 branch, so nothing is suspect there.

dnsmasq provides a /usr/lib/sysusers.d/dnsmasq.conf file which is offloading things to systemd to
take care of:

$ cat /usr/lib/sysusers.d/dnsmasq.conf
u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq

I'm not really familiar with sysusers.d though

> Failed to check if group dnsmasq already exists: No such process

Seems to be some issue with systemd-sysusers or nss.

https://github.com/systemd/systemd/pull/15002

Fixed in rawhide now.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #16)
> Fixed in rawhide now.

You are so quick! nice work.
Note this is marked as an f32 beta blocker so please pull into f32 too

FEDORA-2020-185b9d6417 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-185b9d6417

systemd-245~rc1-4.fc32 has been pushed to the Fedora 32 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-185b9d6417

systemd-245~rc1-4.fc32 has been pushed to the Fedora 32 stable repository. If problems still persist, please make note of it in this bug report.