Bug 2209924

Summary: D-Bus: OCI deployments fail with "Generating update variant: Invalid refspec"
Product: [Fedora] Fedora Reporter: Martin Pitt <mpitt>
Component: rpm-ostreeAssignee: Colin Walters <walters>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 38CC: amurdaca, dustymabe, jmarrero, jonathan, lucab, miabbott, philip.wyett, travier, walters
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
python GDBus GI reproducer none

Description Martin Pitt 2023-05-25 08:51:01 UTC 
I am currently working on making https://github.com/cockpit-project/cockpit-ostree/ fit for OCI based deployments. I noticed that I run into a lot of "Invalid refspec" errors when trying to deploy a registry image with a tag name, such as quay.io/fedora/fedora-coreos:stable . This works fine for "untagged" (well, implied "latest") OCI names, like I use on my personal laptop with 
ghcr.io/martinpitt/workstation-ostree-config.

This doesn't seem to happen with the rpm-ostree CLI, just through the D-Bus API. rpm-ostreed is rather hard to talk to via D-Bus, this is impossible to show with busctl/gdbus, due to the private D-Bus transaction. So I wrote a standalone Python reproducer using GDBus through GI (dbus-python cannot really do this either).

Reproducible: Always

Steps to Reproduce:
1. rpm-ostree rebase ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
2. reboot
3. rpm-ostree install -A python3-gobject-base gobject-introspection
4. python3 rpm-ostree-check-update.py

Actual Results:  
unix:path=/run/rpm-ostree-transaction.sock
Start: True
transaction signal org.projectatomic.rpmostree1.Transaction.Message ('Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable',)
transaction signal org.projectatomic.rpmostree1.Transaction.ProgressEnd ()
transaction signal org.projectatomic.rpmostree1.Transaction.Message ('No upgrade available.',)
transaction signal org.projectatomic.rpmostree1.Transaction.Finished (false, 'Generating update variant: Invalid refspec ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable')


Expected Results:  
On my personal laptop which uses a container image name without a tag (i.e. "latest"), this works fine:

booted: /org/projectatomic/rpmostree1/fedora
unix:path=/run/rpm-ostree-transaction.sock
Start: True
transaction signal org.projectatomic.rpmostree1.Transaction.Message ('Pulling manifest: ostree-unverified-image:docker://ghcr.io/martinpitt/workstation-ostree-config',)
transaction signal org.projectatomic.rpmostree1.Transaction.ProgressEnd ()
transaction signal org.projectatomic.rpmostree1.Transaction.Message ('No upgrade available.',)
transaction signal org.projectatomic.rpmostree1.Transaction.Finished (true, '')
Comment 1 Martin Pitt 2023-05-25 08:51:54 UTC 
Created attachment 1966814 [details]
python GDBus GI reproducer
Comment 2 Martin Pitt 2023-05-25 09:37:31 UTC 
Hmm, apparently it's not the tag name. I changed our integration tests to use "latest", and I'm still getting

    Generating update variant: Invalid refspec ostree-unverified-registry:localhost:5000/ostree-oci

Deployments:
● ostree-unverified-registry:localhost:5000/ostree-oci
                   Digest: sha256:da2503574d72f4cd0982060416ef1d47f3ce8db23821e660029bbce03c0d5353
                  Version: cockpit-base.1 (2023-05-25T09:13:30Z)

`rpm-ostree upgrade --download-only` works fine, though. Adjusting the title accordingly.
Comment 3 Martin Pitt 2023-05-25 16:21:46 UTC 
> Steps to Reproduce

Sorry, I forgot an important step to enable GDBus through GI. Full reproducer:

1. rpm-ostree rebase ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
2. reboot
3. rpm-ostree install -A python3-gobject-base gobject-introspection
4. python3 rpm-ostree-check-update.py
Comment 4 Colin Walters 2023-06-20 12:29:41 UTC 
This may relate to https://github.com/coreos/rpm-ostree/issues/4176
Comment 5 Martin Pitt 2023-07-24 05:16:13 UTC 
I'd really appreciate a quick check if we are using the API wrong, or if this is an actual bug in rpm-ostree. I noticed https://github.com/coreos/rpm-ostree/issues/4176 as well, but it has a different symptom. May be the same root cause, of course.

Thanks!
Comment 6 Colin Walters 2023-07-25 16:02:23 UTC 
I'm pretty sure it's just a dup of 4176 with the same root cause.  We have https://github.com/coreos/rpm-ostree/pull/4486 inbound to attempt to fix it.
Comment 7 Martin Pitt 2023-07-25 17:10:25 UTC 
Thanks Colin! I subscribed to that PR.