Bug 1765476

Summary: creating an empty pod needs working internet connection
Product: Red Hat Enterprise Linux 8 Reporter: Christoph Karl <christoph.karl>
Component: podmanAssignee: Daniel Walsh <dwalsh>
Status: CLOSED CURRENTRELEASE QA Contact: Yuhui Jiang <yujiang>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.3CC: bbaude, dornelas, dwalsh, gscrivan, jligon, jnovy, lsm5, mheon, mmezynsk, mschibli, subhat, tsweeney, umohnani, vrothber, ypu
Target Milestone: rcKeywords: Reopened
Target Release: 8.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-11 15:50:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1853455    
Bug Blocks: 1186913    

Description Christoph Karl 2019-10-25 08:04:28 UTC
Description of problem:
Creating an empty pod tries to download the container k8s.gcr.io/pause from the internet.

Version-Release number of selected component (if applicable):
>podman --version
podman version 1.4.4

How reproducible:
Always, if there is no internet connection


Steps to Reproduce:
1.> podman pod create

Actual results:
>podman pod create
ERRO[0060] Error pulling image ref //k8s.gcr.io/pause:3.1: Error initializing source docker://k8s.gcr.io/pause:3.1: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp 74.125.140.82:443: i/o timeout 
ERRO[0071] Error freeing pod lock after failed creation: no such file or directory 
Error: unable to create pod: error adding Infra Container: unable to pull k8s.gcr.io/pause:3.1: unable to pull image: Error initializing source docker://k8s.gcr.io/pause:3.1: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp 74.125.140.82:443: i/o timeout


Expected results:
"podman pod create" works, even if there is no internet connection.


Additional info:
>podman images
REPOSITORY                        TAG      IMAGE ID       CREATED         SIZE
k8s.gcr.io/pause                  3.1      da86e6ba6ca1   22 months ago   747 kB

This "pause" image is only 757kB.
It should be possible to put this somewhere into the configuration files.

Comment 2 Brent Baude 2019-10-25 14:08:21 UTC
This is not a bug.  The image is required for pods by design.

Comment 3 Daniel Walsh 2019-10-25 14:42:52 UTC
Why not just create a image tar ball that we ship under /usr/share/containers and then we can just pull this in to the tool if it does not exists?

Comment 4 Daniel Walsh 2019-10-25 14:43:38 UTC
I think this is a bug, disconnected systems are definitely something we want to support.

Comment 5 Giuseppe Scrivano 2019-10-25 16:08:03 UTC
this is the same issue that would be encountered with an offline CRI-O installation.

I think we need to document the issue and let people use a different `pause_image=` in `/etc/containers/libpod.conf`.

Another alternative would be to create the pod with `--infra=no`

Comment 6 Daniel Walsh 2019-10-25 16:16:57 UTC
What do you think about just having the image in a tar ball and we load it from their rather then pulling it from the internet?

Comment 7 Giuseppe Scrivano 2019-10-25 16:32:09 UTC
carrying an additional binary blob that will be used in very rare cases seem costly and error prone.

We could probably fall back to creating a container using the first image available in the storage and setting the command to "tail -f /dev/null".

What do you think?

Comment 8 Daniel Walsh 2019-10-25 17:03:14 UTC
That would require a container containing tail.  We could also just include the pause executable, and build a container on the fly.
from scratch
copy pause /pause
entrypoint /pause

Comment 9 Giuseppe Scrivano 2019-10-25 17:27:57 UTC
where do we get /pause?  I am fine with adding a new pause executable to podman, that would also be useful/used with rootless containers instead of re-execing Podman.

Comment 11 Valentin Rothberg 2019-10-28 09:13:32 UTC
(In reply to Giuseppe Scrivano from comment #5)
> this is the same issue that would be encountered with an offline CRI-O
> installation.
> 
> I think we need to document the issue and let people use a different
> `pause_image=` in `/etc/containers/libpod.conf`.
> 
> Another alternative would be to create the pod with `--infra=no`

I am supportive of Giuseppe's proposal. Disconnected deployments need to get the images in any case from somewhere. The pause image would just be one of many.

Documenting this scenario, adding a `pause_image` and being aligned with CRI-O seems worth approaching to me.

Comment 12 Matthew Heon 2019-10-28 13:14:36 UTC
# Default infra (pause) image name for pod infra containers
infra_image = "k8s.gcr.io/pause:3.1"

# Default command to run the infra container
infra_command = "/pause"

Present in the default libpod.conf today. More visibility in the docs could be good, but from a config standpoint, we're completely set.

Comment 13 Daniel Walsh 2019-10-28 16:20:33 UTC
Valentin, isn't this something where we could take advantage of mirroring? also?

Comment 14 Valentin Rothberg 2019-11-06 14:49:53 UTC
(In reply to Daniel Walsh from comment #13)
> Valentin, isn't this something where we could take advantage of mirroring?
> also?

I'm not sure mirroring would help here, unless a user wants to specify an air-gapped mirror for "k8s.gcr.io".

As Matt mentions above, libpod.conf already supports setting the infra image and its command, so I don't think there is much left we can do besides improving documentation.

Note that it's possible to let the `infra_image` point to a local image (e.g., "oci:/home/valentin/infra").

Comment 19 Giuseppe Scrivano 2020-01-13 14:19:28 UTC
There are two possible workarounds:

1) if they are using an internal registry, they can use skopeo to copy the pause image there:

$ skopeo copy docker://k8s.gcr.io/pause:3.1 docker://internal.registry/pause:3.1

Then change the infra_image setting in /etc/containers/libpod.conf to point to the new location:

#infra_image = "k8s.gcr.io/pause:3.1"
infra_image = "internal.registry/pause:3.1"


2) it is possible to create the image locally (the first method is better though), using tail.

Create and launch this script:

-------------------------------------------
#!/bin/sh

IMAGE=fedora

printf "#!/bin/sh\ntail -f /dev/null" > pause
chmod +x pause
cat > Dockerfile <<EOF
FROM $IMAGE
COPY pause /pause
ENTRYPOINT /pause
EOF

podman build -t infra-pause .
-------------------------------------------

You still need to have a base image, please update IMAGE= accordingly in the script.  I've used fedora for my test.

Once that is done, you still need to update infra_image in the /etc/containers/libpod.conf file:

#infra_image = "k8s.gcr.io/pause:3.1"
infra_image = "infra-image"



The same method applies for rootless containers: just make sure to use ~/.config/containers/libpod.conf instead of /etc/containers/libpod.conf

Comment 20 Daniel Walsh 2020-01-13 16:37:01 UTC
I have a feeling this is more about getting two containers to talk to each other.  Podman 1.6.4  or Podman 1.7 has dns support to make this happen, I believe.

Comment 23 Daniel Walsh 2020-06-03 14:34:08 UTC
You need to change containers.conf now, libpod.conf is no longer supported.

Comment 24 Daniel Walsh 2020-06-03 14:36:28 UTC
Is this just a documentation issue?

Comment 25 Tom Sweeney 2020-06-08 21:39:31 UTC
Matt, Giuseppe any update on this one?

Comment 26 Derrick Ornelas 2020-08-25 18:20:40 UTC
I believe the suggested workarounds require Bug 1853455 to be fixed first

Comment 28 Daniel Walsh 2021-01-28 11:57:26 UTC
Fixes will be in podman 3.0 to make the workaround possible.

Comment 32 Daniel Walsh 2021-01-29 16:23:26 UTC
Perhaps here from containers/common

```
common (format) $ git show 7f460be90af569956699d003fbd47fd4cd44e23b
commit 7f460be90af569956699d003fbd47fd4cd44e23b
Author: Matthew Heon <mheon>
Date:   Thu Nov 19 15:43:36 2020 -0500

    We should not be setting a default infra command.
    
    We should be sourcing from the image CMD/ENTRYPOINT by default.
    Having a default prevents us from doing that - we should only be
    using this for user-configured values that do not come from the
    image.
    
    Signed-off-by: Matthew Heon <mheon>

diff --git a/pkg/config/default.go b/pkg/config/default.go
index 5f8f499..29eb3c3 100644
--- a/pkg/config/default.go
+++ b/pkg/config/default.go
@@ -46,8 +46,6 @@ var (
        DefaultInitPath = "/usr/libexec/podman/catatonit"
        // DefaultInfraImage to use for infra container
        DefaultInfraImage = "k8s.gcr.io/pause:3.2"
-       // DefaultInfraCommand to be run in an infra container
-       DefaultInfraCommand = "/pause"
        // DefaultRootlessSHMLockPath is the default path for rootless SHM locks
        DefaultRootlessSHMLockPath = "/libpod_rootless_lock"
        // DefaultDetachKeys is the default keys sequence for detaching a
@@ -308,7 +306,6 @@ func defaultConfigFromMemory() (*EngineConfig, error) {
        c.InitPath = DefaultInitPath
        c.NoPivotRoot = false
 
-       c.InfraCommand = DefaultInfraCommand
        c.InfraImage = DefaultInfraImage
        c.EnablePortReservation = true
        c.NumLocks = 2048
```

Comment 33 Matthew Heon 2021-01-29 17:06:35 UTC
99% sure that is completely separate. Was the fix here not that we now build the pause image ourselves?

If the request is that we should not require a non-local pause image at all, this is not resolved.

Comment 36 RHEL Program Management 2021-02-01 07:05:34 UTC
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.

Comment 44 Daniel Walsh 2021-06-11 15:50:41 UTC
Ok well we have suggested fixes for it, so I am going to close.

Comment 45 Christoph Karl 2021-06-14 04:00:36 UTC
Suggested fix accepted.