Nevermind, just going to move this to 8.8

I think what might have happened is I did the build too soon after the 8.8 branch opening, and OSCI wasn't fully wired up or something.  (And I think all the OSCI stuff is message and not controller based, i.e. it doesn't attempt to reconcile continuously)

It might help to just do a dummy change and rebuild.

I should note I already tagged this build to ship in OCP/RHCOS 4.12 in https://issues.redhat.com/browse/ART-4777

This will be used once https://github.com/openshift/machine-config-operator/pull/3317/ merges - so I think we could just mark verified once the jobs there are passing.

(But, it's also fine to do verification based on the upstream unit test https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/destructive/container-update-inplace )

> (But, it's also fine to do verification based on the upstream unit test
> https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/destructive/
> container-update-inplace )

Hi Colin, start with 412.86.202209170127-0 (with fixed rpm-ostree-2022.10.94.g89f58028-2.el8.x86_64) using cosa according to the steps in container-update-inplace, get error, is there any config I missed?

Test steps (skip https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/destructive/container-update-inplace#L21-L35):
# rpm -q skopeo rpm-ostree
skopeo-1.8.0-1.rhaos4.12.el8.x86_64
rpm-ostree-2022.10.94.g89f58028-2.el8.x86_64
# checksum=$(rpm-ostree status --json | jq -r '.deployments[0].checksum')
# ostree container encapsulate --repo=/ostree/repo ${checksum} containers-storage:localhost/rhcos-test
error: skopeo failed: time="2022-09-19T01:46:47Z" level=fatal msg="Error loading trust policy: invalid policy in \"/etc/containers/policy.json\": Unknown key \"keyPaths\""

# cat /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
	    "registry.access.redhat.com": [
		{
		    "type": "signedBy",
		    "keyType": "GPGKeys",
		    "keyPaths": ["/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release", "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"]
		}
	    ],
	    "registry.redhat.io": [
		{
		    "type": "signedBy",
		    "keyType": "GPGKeys",
		    "keyPaths": ["/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release", "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"]
		}
	    ]
	},
        "docker-daemon": {
	    "": [
		{
		    "type": "insecureAcceptAnything"
		}
	    ]
	}
    }
}

That's a generic skopeo failure; does a plain e.g. `skopeo inspect docker://quay.io/fedora/fedora:latest` work? 

Did you make a custom 4.12 build using https://coreos.github.io/rpm-ostree/administrator-handbook/#using-overrides-and-usroverlay ?

My guess is that you somehow got a wrong version of skopeo in your custom build.  

I'd try doing a per-machine live override (https://coreos.github.io/rpm-ostree/administrator-handbook/#using-overrides-and-usroverlay) first.

I run `skopeo inspect docker://quay.io/fedora/fedora:latest` this works, just start rhcos-412.86.202209170127-0-qemu.x86_64.qcow2 using cosa, not quite sure about the skopeo version, will try to find, thanks!

This reproduces the failure outside of the ostree stack:

```
$ rpm -q skopeo containers-common
skopeo-1.8.0-1.rhaos4.12.el8.x86_64
containers-common-1-27.rhaos4.12.el8.x86_64
$ skopeo copy docker://quay.io/fedora/fedora:latest containers-storage:localhost/fedora
FATA[0000] Error loading trust policy: invalid policy in "/etc/containers/policy.json": Unknown key "keyPaths"
$
```

I think something in skopeo regressed here; it's not related to the new rpm-ostree.

This only seems to happen when pulling to `containers-storage:` - which means it doesn't affect `rpm-ostree rebase` - phew.

It also only affects skopeo - `podman run --rm -ti registry.access.redhat.com/ubi8/ubi:latest` works (as you'd expect as a failure there would have tripped hopefully several CI pipelines).

Are you doing the `container-encapsulate` to get an rhcos container image to test rebasing?  Let's use the official image:

$ oc adm release info --image-for=rhel-coreos-8 quay.io/openshift-release-dev/ocp-release:4.12.0-ec.3-x86_64
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b5087a3d1f3ac26db6cb8b07ff21d4364d29ad673c8fbe9374f8118d61e8f21c

Thanks Colin for the reply! Start with rhcos 412.86.202209152103-0 image using cosa, rebase the ostree image from 412.86.202209170127-0 works, reboot and check OS is upgraded, are my steps expected?


$ cosa run rhcos-412.86.202209152103-0-qemu.x86_64.qcow2 -m 2048
After login rhcos,
# mkdir -p /run/ostree
# touch /run/ostree/auth.json
# podman run --authfile /run/ostree/auth.json --privileged --pid=host --net=host --rm -v /:/run/host quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c36859b2519a1ab69faffee63cb9cca7fb02cfc63573cfcd71f3892de18134f1 rpm-ostree ex deploy-from-self /run/host
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c36859b2519a1ab69faffee63cb9cca7fb02cfc63573cfcd71f3892de18134f1...
Getting image source signatures
......
Copying config 596e945685 done  
Writing manifest to image destination
Storing signatures
NOTICE: Experimental commands are subject to change.
Imported: 21bf98492dbc5de6524f201f352b137dbba6d854a2d3d57fd83e13ca4b7a83a5
Staging deployment...done
Upgraded:
  coreos-installer 0.15.0-2.rhaos4.12.el8 -> 0.16.0-1.rhaos4.12.el8
  coreos-installer-bootinfra 0.15.0-2.rhaos4.12.el8 -> 0.16.0-1.rhaos4.12.el8
Changes queued for next boot. Run "systemctl reboot" to start a reboot


[root@cosa-devsh ~]# podman images -a
REPOSITORY                                      TAG         IMAGE ID      CREATED             SIZE
quay.io/openshift-release-dev/ocp-v4.0-art-dev  <none>      596e94568584  About a minute ago  2.64 GB

[root@cosa-devsh ~]# podman inspect 596e94568584 | grep version | head -1
                    "version": "412.86.202209170127-0"


reboot and check
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
● 21bf98492dbc5de6524f201f352b137dbba6d854a2d3d57fd83e13ca4b7a83a5
                  Version: 412.86.202209170127-0 (2022-09-17T01:30:25Z)

  968723c9159a2596abbb616105808ffb6c4c1332d1c7d062dc886ef8b0091ef9
                  Version: 412.86.202209152103-0 (2022-09-15T21:06:11Z)
[core@cosa-devsh ~]$ rpm-ostree db diff
ostree diff commit from: rollback deployment (968723c9159a2596abbb616105808ffb6c4c1332d1c7d062dc886ef8b0091ef9)
ostree diff commit to:   booted deployment (21bf98492dbc5de6524f201f352b137dbba6d854a2d3d57fd83e13ca4b7a83a5)
Upgraded:
  coreos-installer 0.15.0-2.rhaos4.12.el8 -> 0.16.0-1.rhaos4.12.el8
  coreos-installer-bootinfra 0.15.0-2.rhaos4.12.el8 -> 0.16.0-1.rhaos4.12.el8

Additional info for Comment 16
[core@cosa-devsh ~]$ rpm -q rpm-ostree
rpm-ostree-2022.10.94.g89f58028-2.el8.x86_64

Yep, that looks good to me, I believe we can mark this VERIFIED.  (Also because we got working code in https://github.com/openshift/machine-config-operator/pull/3317 )

Thanks Colin for the confirmation, change status to verified

Verify passed with rpm-ostree-2022.10.97.gade6df33-2.el8.x86_64 according to steps in comment #16

$ rpm-ostree status
State: idle
Deployments:
● 6f7fc4df61b039708373a8258f32461698bb23bdcdb02210c5128f823baae2b6
                  Version: 412.86.202210072111-0 (2022-10-07T21:14:25Z)
           LocalOverrides: rpm-ostree-libs rpm-ostree 2022.10.94.g89f58028-2.el8 -> 2022.10.97.gade6df33-2.el8

  69c24476c6073a37da9f89809d00ff21adb6b34a2cb4f03964f18ccdc44c2193
                  Version: 412.86.202210061014-0 (2022-10-06T10:17:40Z)
           LocalOverrides: rpm-ostree-libs rpm-ostree 2022.10.94.g89f58028-2.el8 -> 2022.10.97.gade6df33-2.el8

(In reply to Colin Walters from comment #10)
> I should note I already tagged this build to ship in OCP/RHCOS 4.12 in
> https://issues.redhat.com/browse/ART-4777
> 
> This will be used once
> https://github.com/openshift/machine-config-operator/pull/3317/ merges - so
> I think we could just mark verified once the jobs there are passing.
> 
> (But, it's also fine to do verification based on the upstream unit test
> https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/destructive/
> container-update-inplace )

thanks!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rpm-ostree bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2759