Hi, https://fedoraproject.org/wiki/Changes/OstreeNativeContainer has been steadily progressing and I think it's time to discuss Anaconda support. Today we have the `ostreesetup` kickstart verb (https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#ostreesetup). We could extend that, or (my somewhat weak preference) add a new one. Let's say the kickstart looks just like this: ``` ostreecontainer --no-signature-verification quay.io/examplecorp/example:latest ``` In the code today, around here https://github.com/rhinstaller/anaconda/blob/6b0cd37247ff7ba4420753acb39c59957193d180/pyanaconda/modules/payloads/payload/rpm_ostree/installation.py#L516 we'd add a conditional like: ``` if kickstart.container_image is not None: ``` And here we just invoke `ostree container image deploy` as is done today in e.g. coreos-assembler: https://github.com/coreos/coreos-assembler/blob/e4f39c89bff800ef9ea14de4f5e2ab4a7035384f/src/create_disk.sh#L344 And...I think that's basically it. The end story here is that for example we have quay.io/fedora/silverblue:latest - and Anaconda installs that instead of using ostree-on-the-wire, and invocations of `rpm-ostree upgrade` pull that container. (What's more compelling though is that users can create their own custom *derived* container images, and use `ostreecontainer` on that...)
Wouldn't that require network access? How should we set that up for the case where we ship everything in the ISO?
No, the default images would be embedded in the ISO as e.g. oci: directories, the same way we embed ostree repositories.
xref https://github.com/openshift/machine-config-operator/issues/3137 where I have some partially-baked thoughts that could actually lead us to a world where we also support installing RHEL CoreOS (and OpenShift) via Anaconda using this.
Hi Colin, that sounds doable and somewhat easy. What do you think about adding new parameter `--container-url` next to current --url with higher priority? Another option which come to mi mind is to use `--url=oci+<url>`, I know I already saw this somewhere but not sure if it doesn't conflict with something else.
For reference, there's a new followup Change up: https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable Yeah, I'm totally fine extending the `ostreesetup` kickstart verb. --container-url seems fine to me. It does feel a bit less elegant than a clean new verb, but OTOH we can easily add a new verb later too.
As of recently https://github.com/containers/bootc exists, and it may make sense to "brand" things that way. i.e. the kickstart verb is: ``` bootc --no-signature-verification quay.io/examplecorp/example:latest ```
Currently the KS command looks like: ostreesetup --osname OSNAME [--remote REMOTE] --url URL --ref REF [--nogpg] If I understand it correctly than --ref and --url is combined in the new container URL and --nogpg is basically proposed --no-signature-verification. However, it seems that --remote would be still required because we need to be able to set remote when installing from local installation source on DVD (current SilverBlue way). --osname seems to be still required?
I see that the `--remote`` parameter is already resolved at comment 2, when there are no embed image on container it means we don't have to care about the remote. Even thought, we already had this requirement from elsewhere so I guess we would have to add that in the future if we go with a new verb.
(In reply to Jiri Konecny from comment #8) > I see that the `--remote`` parameter is already resolved at comment 2, when > there are no embed image on container it means we don't have to care about > the remote. Even thought, we already had this requirement from elsewhere so > I guess we would have to add that in the future if we go with a new verb. My bad, seems that I can't read. So if there will be the baked images on the DVD than we need also this `--remote`` parameter. Which means that we would almost copy the parameters of the current kickstart command?
I had a discussion with Colin and this is the proposal for the Kickstart command: bootc [--osname OSNAME] [--remote REMOTE] [--no-signature-verification] quay.io/examplecorp/example:latest
The osname parameter would be optional based on: https://github.com/ostreedev/ostree/issues/2794
(In reply to Jiri Konecny from comment #10) > I had a discussion with Colin and this is the proposal for the Kickstart > command: > > bootc [--osname OSNAME] [--remote REMOTE] [--no-signature-verification] > quay.io/examplecorp/example:latest I see that bootc is a new project, but I'm not clear as to what it actually does in the context of a new installation. Also, can it be used without ostreesetup? Are the parameters the same (eg. same --osname and/or --remote?) as an included ostreesetup or can they be different? If they have to match ostreesetup and all this does is tell anaconda to run `bootc something` then I'd rather it was added to ostreesetup. If it can act independently, with different parameters, then a new command does look reasonable.
At the current time, bootc itself does not expose verbs to install. However, https://github.com/containers/bootc/issues/1 is relevant in this context (but again, we're going to support being installed via Anaconda too). The actual mechanics involved here are basically: - Pull container image and set up the ostree stuff backing it - Install bootloader stuff The first part is very similar to the "plain ostree" stuff already implemented by anaconda, see: https://github.com/coreos/coreos-assembler/blob/e4f39c89bff800ef9ea14de4f5e2ab4a7035384f/src/create_disk.sh#L344 To say this another way, where ostree can be used, `ostree container` can be used too. But, I'd like to rebrand "ostree container" => bootc. It may indeed make sense to wedge this stuff into the existing `ostreesetup` for now and circle back to adding bootc as a nicer frontend later.
Thanks, since that's the case it looks to me like ostreesetup just needs a --bootc argument for now. Once that's working and if it needs more we can add a new command for it. In kickstart we'd need to switch what arguments are required/optional based on the presence of --bootc and maybe alias --no-signature-verification to --nogpg.
Hmm. I am oscillating a bit between simply wanting to get the functionality out in any form versus the reality of needing to support the kickstart syntax for years to come =) The thing I'm most uncertain about is attempting to conflate --no-signature-verification and --nogpg. A major part of the whole concept here is that we can now reuse signature mechanisms shipped in the podman/skopeo (containers/image) stack - not just "simple signing" aka GPG but the new sigstore/cosign. But...eh. So to flesh out your proposal a bit, it'd look like: ostreesetup --bootc quay.io/acmecorp/exampleos:latest --nogpg which would be the same as what `bootc switch --no-signature-verification quay.io/acmecorp/exampleos:latest` does? If so...yeah, I'm fine with that.
Hi Brian, we discussed this with Vendy and from the point of view that (based on my information) this could in the future replace the original way of deployment and also that more specific commands to this could be added in the future, we would rather prefer the new `bootc` command solution as described at comment 10. Doing this now would improve our situation in the future if we need to extend it or deprecate the current command. Also it's solution in a similar way of what we have now for DNF (cdrom, harddrive, url, nfs...) Would that be fine with you? If yes, than my question is if I should handle the pykickstart part or you want to do that?
If bootc was fully functional I'd be more inclined to go in that direction, but it looks like the current state is that it's still being worked on and is incomplete. It might even go away if things don't work out, right? So I don't want to add a new command (or really even new arguments) until they are actually needed for new install methods. I'm fine doing the work, I just want to make sure it's as focused as possible. @walters I was thinking it would be like: ostreesetup --bootc --url quay.io/acmecorp/exampleos:latest --no-signature-verification since we already have a --url that parsed in the current method. --no-signature-verification is a bit, um, verbose :) but I do like it better than --nogpg since it implies it covers more than just gpg verification and internally it can just be an alias to nogpg.
Implementation wise, bootc is currently a *very* layer on top of all the same "ostree container" bits that are used by rpm-ostree. You're right though, it it very young and indeed things may change there...but I'm hoping not.
Seems that we have a decision to go with the `ostreesetup` version from Brian. In that case I'll start working on the Anaconda part when Pykickstart is available. Thanks Brian and Colin for your help!
Also maybe you want to make the --osname optional with this changeset as pointed at comment 11.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
Pykickstart PR: https://github.com/pykickstart/pykickstart/pull/434 Anaconda PR: https://github.com/rhinstaller/anaconda/pull/4561
> * it takes really long time with no progress info (could be confusing to users). I think we talked about this one before; worst case right now, Anaconda could scrape out the last line of text output from the command and render it, though this is obviously suboptimal. The real fix is going to require adding some sort of explicit API that exports progress information in a way that can be rendered in a GUI. This relates to https://github.com/containers/bootc/issues/4 too. > * the resulting system seems to be working fine, but it boots to command line not GDM, is that on purpose? This seems like a bug for sure; probably something is overriding the systemd default.target? I do see # ls -al /usr/lib/systemd/system/default.target lrwxrwxrwx. 3 root root 16 Jan 1 1970 /usr/lib/systemd/system/default.target -> graphical.target in the image. > Also do you agree with needs_network implementation that --transport=registry == it require network? (edited) Yep
(In reply to Colin Walters from comment #24) > > * it takes really long time with no progress info (could be confusing to users). > > I think we talked about this one before; worst case right now, Anaconda > could scrape out the last line of text output from the command and render > it, though this is obviously suboptimal. > > The real fix is going to require adding some sort of explicit API that > exports progress information in a way that can be rendered in a GUI. This > relates to https://github.com/containers/bootc/issues/4 too. I tested https://pagure.io/releng/issue/11047#comment-846010 and it seems to be much faster to me for deployment. We can do that but let's do that in the next version and not complicate the current PR. > > > * the resulting system seems to be working fine, but it boots to command line not GDM, is that on purpose? > > This seems like a bug for sure; probably something is overriding the systemd > default.target? I do see > > # ls -al /usr/lib/systemd/system/default.target > lrwxrwxrwx. 3 root root 16 Jan 1 1970 > /usr/lib/systemd/system/default.target -> graphical.target > > in the image. After the installation I see the same the `graphical.target` is the default. gdm.service seems to be enabled and loaded but not started when I do `systemctl start gdm` it will correctly show GDM. Not sure what is happening there. > > > Also do you agree with needs_network implementation that --transport=registry == it require network? (edited) > > Yep That is great. Thanks! However, I'm seeing a lot of GIO errors during the deployment (at the end): ``` 12:41:49,506 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program: 12:41:49,506 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:(process:2578): GLib-GIO-CRITICAL **: 12:35:31.416: GFileInfo created without standard::size 12:41:49,506 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program: 12:41:49,506 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:(process:2578): GLib-GIO-CRITICAL **: 12:35:31.416: file ../gio/gfileinfo.c: line 1800 (g_file_info_get_size): should not be reached ``` Tested on Rawhide and it seems to be coming from ostree container tool. Attaching the logs, do you have an idea why this happens Colin?
Created attachment 1950613 [details] syslog - rawhide
Ok, the issue was in kickstart file and maybe not related issue in Anaconda. Anyway seems like everything is working correctly. The Anaconda changes should be prepared for merge. I'll backport my PR to F38 after review.
FEDORA-2023-0cd1293bf8 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-0cd1293bf8
FEDORA-2023-0cd1293bf8 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
Not done the PR is not yet merged.
FEDORA-2023-6671e3a283 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-6671e3a283
FEDORA-2023-6671e3a283 has been pushed to the Fedora 38 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-6671e3a283 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-6671e3a283 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.