Bug 1808428

Giuseppe,

It looks like to did some related changes in upstream.
fedora 30 still has atomic utility which support ostree but latest upgrade of skopeo broke it.
How do you plan to fix it in fedora 30?

there should not be any atomic tool on Fedora 30.  The last build I can see on Bodhi is for Fedora 28.

Can you please check from where the atomic tool is coming from?

(In reply to Giuseppe Scrivano from comment #2)
> there should not be any atomic tool on Fedora 30.  The last build I can see
> on Bodhi is for Fedora 28.
> 
> Can you please check from where the atomic tool is coming from?

Yes I can.
sh# podman run -ti --rm docker.io/fedora:30 bash
[root@e8c909734d28 /]# dnf repolist
repo id                                   repo name
fedora                                    Fedora 30 - x86_64
fedora-modular                            Fedora Modular 30 - x86_64
updates                                   Fedora 30 - x86_64 - Updates
updates-modular                           Fedora Modular 30 - x86_64 - Updates
[root@e8c909734d28 /]# dnf -e0 -d0 info atomic
Available Packages
Name         : atomic
Version      : 1.22.1
Release      : 28.gitb507039.fc30
Architecture : x86_64
Size         : 1.0 M
Source       : atomic-1.22.1-28.gitb507039.fc30.src.rpm
Repository   : fedora
Summary      : Tool for managing ProjectAtomic systems and containers
URL          : https://github.com/projectatomic/atomic
License      : LGPLv2+
Description  : The goal of Atomic is to provide a high level, coherent entrypoint to the
             : system, and fill in gaps.
             :
             : atomic can make it easier to interact with container runtimes for different
             : kinds of containers, such as super-privileged and system containers.
             :
             : The atomic host subcommand wraps rpm-ostree providing unified management.

[root@e8c909734d28 /]# cat /etc/os-release 
NAME=Fedora
VERSION="30 (Container Image)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

I can see it also in koji
https://koji.fedoraproject.org/koji/buildinfo?buildID=1184437
and f30 branch is not empty either
https://src.fedoraproject.org/rpms/atomic/tree/f30

atomic utility was removed just from f31+

Dan, should we revert back to the previous build for Skopeo?

Yes I think so.

Giuseppe,
may I know what was the reason for removing support for ostree from skopeo in upstream?
Colin[1] (can) use ostree format[2] to do simple labels/filesystem checks for containers in openshift.
Which drop same capabilities by default
     "CapDrop": ["KILL", "MKNOD", "SETGID", "SETUID"]

I know you prefer to do lot of things with rootless podman,
but rootless != capabilityless.

Correct me is I am wrong but SETGID", "SETUID" are required even for rootless podman
die to ambient capabilities. And thus we cannot use it in openshift with default scc.

The colin uses just a single feature from atomic utility "atomic mount" which I could easily
reimplement and remove dependency on atomic utility. But skopeo would need to support ostree.
So we will need to either stuck with fedora 30 (even after EOL or to move to centos7 + install everything
from pip due to requirement on python3.6).

How difficult would be to support ostree even in skopeo >= 0.1.41-1?

[1] https://github.com/user-cont/colin
[2] https://github.com/user-cont/colin/blob/master/colin/core/target.py#L437

The main reason from dropping support for ostree is that atomic is the only known user of it, and we are not maintaining atomic anymore.

The ostree backend is tailored for a very specific use case and required a lot of logic into the atomic tool.

Without a user namespace, you can extract (which is really an extraction) the image but all the files are owned by the same user.

That can be achieved also in an environment without SETUID/SETGID, as long as you are able to create a user namespace.  I'd expect `podman mount` to work also without capabilities, and the vfs backend to work without having access to /dev/fuse.

You can also use umoci to extract an OCI image: https://github.com/openSUSE/umoci

(In reply to Giuseppe Scrivano from comment #8)
> The main reason from dropping support for ostree is that atomic is the only
> known user of it, and we are not maintaining atomic anymore.
> 
> The ostree backend is tailored for a very specific use case and required a
> lot of logic into the atomic tool.
> 
> Without a user namespace, you can extract (which is really an extraction)
> the image but all the files are owned by the same user.
>

I knot but that is not a problem for my use-case. That was also the reason for
for ostree --mode=bare-user (in the description of BZ)
 
> That can be achieved also in an environment without SETUID/SETGID, as long
> as you are able to create a user namespace.

The openshift cluster is not maintained by me :-)
How can I test whether I can use user namespaces there?
The cluster seems to be deployed on el7 based on umame
3.10.0-957.21.2.el7.x86_64

Cause I would be glad to use podman directly.

Anyway, ostree support should stay in f30 :-)

(In reply to Giuseppe Scrivano from comment #8)
> You can also use umoci to extract an OCI image:
> https://github.com/openSUSE/umoci

That is quite interesting just have a question about difference in  skopeo inspect

bash-5.0$ export IMAGE_NAME=docker.io/fedora/apache:latest
bash-5.0$ /usr/bin/skopeo copy docker://$IMAGE_NAME oci:/var/tmp/oci:$IMAGE_NAME
Getting image source signatures
Copying blob 0fc456f626d7 done
Copying blob 8eea4f8b1da3 done
Copying blob 40da690b3498 done
Copying blob 4e81794d88f1 done
Copying blob ebb42f0b0e1a done
Copying config aeee6d164d done
Writing manifest to image destination
Storing signatures

bash-5.0$ ostree init --repo=/var/tmp/os3/ --mode=bare-user
bash-5.0$ /usr/bin/skopeo copy docker://$IMAGE_NAME ostree:$IMAGE_NAME@/var/tmp/os3
Getting image source signatures
Copying blob 0fc456f626d7 done
Copying blob 8eea4f8b1da3 done
Copying blob 40da690b3498 done
Copying blob 4e81794d88f1 done
Copying blob ebb42f0b0e1a done
Copying config c786010769 done
Writing manifest to image destination
Storing signatures

bash-5.0$ diff -u <(skopeo inspect oci:/var/tmp/oci:$IMAGE_NAME) <(skopeo inspect ostree:$IMAGE_NAME@/var/tmp/os3)
--- /dev/fd/63  2020-03-20 12:01:14.463930483 +0000
+++ /dev/fd/62  2020-03-20 12:01:14.463930483 +0000
@@ -1,8 +1,8 @@
 {
-    "Digest": "sha256:811764eb3b54db5650e0c1e55dd54f499bcc051b0a92efef58b448124c586b72",
+    "Digest": "sha256:8531786520bb57b155bbb39d3c670dceab554b9c4ccdb556ccfbe89b23df414c",
     "RepoTags": [],
     "Created": "2017-02-08T23:42:37.72165736Z",
-    "DockerVersion": "",
+    "DockerVersion": "1.12.6-cs6",
     "Labels": {
         "RUN": "docker run -d -p 80:80 $IMAGE"
     },

Is the change in digest expected or bug in skopeo?
(sorry for misusing BZ but I do not want to create unnecessary bugs)

probably a bug in skopeo

You can check whether user namespaces are supported by running "podman unshare" as unprivileged user, or alternatively run directly "unshare -r COMMAND"

(In reply to Giuseppe Scrivano from comment #5)
> Dan, should we revert back to the previous build for Skopeo?

(In reply to Daniel Walsh from comment #6)
> Yes I think so.

Giusseppe,
when do you plan to fix it?

Could we revert the version of skopeo on Fedora 30?

This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

FEDORA-2020-948a6a30da has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-948a6a30da

FEDORA-2020-948a6a30da has been pushed to the Fedora 30 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-948a6a30da`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-948a6a30da

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-948a6a30da has been pushed to the Fedora 30 stable repository.
If problem still persists, please make note of it in this bug report.