openQA upgrade tests to Rawhide are failing ATM. The immediate error is this:

Jan 04 05:59:31 localhost.localdomain dnf[625]: Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-rawhide-x86_64 [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-rawhide-x86_64]
Jan 04 05:59:31 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-system-upgrade comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jan 04 05:59:31 localhost.localdomain kernel: kauditd_printk_skb: 70 callbacks suppressed
Jan 04 05:59:31 localhost.localdomain kernel: audit: type=1130 audit(1515074371.834:81): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-system-upgrade comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jan 04 05:59:31 localhost.localdomain systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 05:59:31 localhost.localdomain systemd[1]: Failed to start System Upgrade using DNF.

It can't find the file because it doesn't exist. The tests use '--releasever=rawhide', so dnf looks for a file called RPM-GPG-KEY-fedora-rawhide-x86_64 - the 'rawhide' there is taken from the 'releasever' value.

I can fiddle around with the tests so they know what Rawhide's release number is and pass that as 'releasever' instead, but the thing is, I shouldn't need to. The tests also pass '--nogpgcheck', so dnf shouldn't really be looking for GPG keys at all. This has worked in the past. The actual commands the tests run are:

dnf -y --releasever=rawhide --nogpgcheck system-upgrade download
dnf -y --releasever=rawhide --nogpgcheck system-upgrade reboot

and when they do that, the above problem happens. In the past, the upgrade worked OK, despite the GPG key file not existing, which seems to indicate that DNF either did not previously look for it, or didn't care if it didn't exist.

One thing I've noticed here is there actually seems to be a sort of collision between packagekit-offline-update.service and dnf-system-upgrade.service , here, as we can see in the logs:

Jan 04 05:59:24 localhost.localdomain systemd[1]: Starting System Upgrade using DNF...
Jan 04 05:59:24 localhost.localdomain systemd[1]: Listening on D-Bus System Message Bus Socket.
Jan 04 05:59:24 localhost.localdomain systemd[1]: Starting Update the operating system whilst offline...
Jan 04 05:59:24 localhost.localdomain pk-offline-update[626]: another framework set up the trigger
Jan 04 05:59:24 localhost.localdomain systemd[1]: Started Update the operating system whilst offline.

"System Upgrade using DNF" is dnf-system-upgrade.service , while "Update the operating system whilst offline" is packagekit-offline-update.service . As you can see, *both* services get started. I'm not sure if that is relevant here, or not, but it is interesting. Both services are wanted by system-update.target.