Bug 1719993

Summary: toolbox appears broken by latest podman update
Product: [Fedora] Fedora Reporter: Gerard Ryan <fedora>
Component: toolboxAssignee: Debarshi Ray <debarshir>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: debarshir, tagoh
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-29 09:33:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gerard Ryan 2019-06-12 21:49:36 UTC
Description of problem:
I'm on Fedora Silverblue and rely on toolbox a lot. Today I pulled in an update to podman, and now I can't get into my toolbox container with "toolbox enter". I'm not 100% sure what other packages were updated, but this seems like the most likely candidate, right?

Version-Release number of selected component (if applicable):
 
podman-1.4.0-2.fc30.x86_64
toolbox-0.0.10-1.fc30.noarch

(see the "Additional info" section below for the entire set of changed package versions)

How reproducible:
I can 100% not get into the container with the new version of podman by using toolbox enter. If I `rpm-ostree rollback` (Silverblue++) I get back to the older version of podman, and it works fine again. 

Steps to Reproduce:
1. Get to the same versions of podman and toolbox as I have
2. Notice that you can't "toolbox enter" any longer

Actual results:

$ toolbox -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.4.0 is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is fedora-toolbox-30
toolbox: checking if container fedora-toolbox-30 exists
toolbox: container fedora-toolbox-30 not found
toolbox: container fedora-toolbox-gryan-30 found
toolbox: trying to start container fedora-toolbox-gryan-30
toolbox: looking for /etc/profile.d/toolbox.sh in container fedora-toolbox-gryan-30
Error: exit status 1
toolbox: copying /etc/profile.d/toolbox.sh to container fedora-toolbox-gryan-30
Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying
toolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-gryan-30


Expected results:
toolbox enter works as before

Additional info:

In case it's useful, here is the not working rpm-ostree deployment info:

● ostree://fedora-workstation:fedora/30/x86_64/silverblue
                   Version: 30.20190612.1 (2019-06-12T14:40:39Z)
                BaseCommit: d0bf1e2c98fb385f8a7176c9f176e8355dc0f767443bffe910cb4eedc18f2d3d
                            ├─ repo-0 ((invalid timestamp))
                            ├─ repo-1 ((invalid timestamp))
                            └─ repo-2 ((invalid timestamp))
                    Commit: 508ee255dce7d165a68b27c09e80387d9a584ac2a86570bfbeb5dce4d80e0da4
                            ├─ updates (2019-06-12T14:42:48Z)
                            ├─ fedora (2019-04-25T23:49:41Z)
                            └─ google-chrome (2019-06-11T15:31:30Z)
                 StateRoot: fedora-workstation
              GPGSignature: 1 signature
                            Signature made Wed 12 Jun 2019 03:40:46 PM IST using RSA key ID EF3C111FCFC659B9
                            Good signature from "Fedora <fedora-30-primary>"
           LayeredPackages: compat-openssl10 fedora-workstation-repositories golang-bin krb5-workstation mozilla-fira-mono-fonts powerline qemu-kvm solaar syncthing virt-manager
             LocalPackages: libvirt-daemon-config-nwfilter-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-scsi-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-nwfilter-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-sheepdog-5.1.0-8.fc30.x86_64 libvirt-daemon-config-network-5.1.0-8.fc30.x86_64 libvirt-devel-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-mpath-5.1.0-8.fc30.x86_64 google-chrome-stable-75.0.3770.80-1.x86_64 libvirt-daemon-driver-storage-logical-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-iscsi-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-gluster-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-core-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-disk-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-zfs-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-iscsi-direct-5.1.0-8.fc30.x86_64
                            libvirt-libs-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-nodedev-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-rbd-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-5.1.0-8.fc30.x86_64 libvirt-5.1.0-8.fc30.x86_64 libvirt-daemon-kvm-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-qemu-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-interface-5.1.0-8.fc30.x86_64 libvirt-bash-completion-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-secret-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-lxc-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-vbox-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-network-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-libxl-5.1.0-8.fc30.x86_64 libvirt-daemon-5.1.0-8.fc30.x86_64 libvirt-client-5.1.0-8.fc30.x86_64

and here's the rolled-back deployment that does work (the "Downgraded" field appears to be showing all the package differences):

ostree://fedora-workstation:fedora/30/x86_64/silverblue
                   Version: 30.20190611.0 (2019-06-11T01:51:26Z)
                BaseCommit: 1f3be0e0b8cd4c0af0b9271a627e2e698d6884c55429588d1dba06d9f4452540
                            ├─ repo-0 ((invalid timestamp))
                            ├─ repo-1 ((invalid timestamp))
                            └─ repo-2 ((invalid timestamp))
                    Commit: ca53100c3ad4d1a55b4afc6cb47288ab2436c34c543800342ddeec42602a5b1b
                            ├─ updates (2019-06-11T01:53:31Z)
                            ├─ fedora (2019-04-25T23:49:41Z)
                            └─ google-chrome (2019-06-06T18:05:57Z)
                    Staged: no
                 StateRoot: fedora-workstation
              GPGSignature: 1 signature
                            Signature made Tue 11 Jun 2019 02:51:32 AM IST using RSA key ID EF3C111FCFC659B9
                            Good signature from "Fedora <fedora-30-primary>"
                Downgraded: containernetworking-plugins 0.7.5-1.fc30 -> 0.7.4-2.fc30
                            crypto-policies 20190527-1.git0b3add8.fc30 -> 20190211-2.gite3eacfc.fc30
                            gnome-settings-daemon 3.32.0-2.fc30 -> 3.32.0-1.fc30
                            kernel 5.1.8-300.fc30 -> 5.1.7-300.fc30
                            kernel-core 5.1.8-300.fc30 -> 5.1.7-300.fc30
                            kernel-devel 5.1.8-300.fc30 -> 5.1.7-300.fc30
                            kernel-headers 5.1.8-300.fc30 -> 5.1.7-300.fc30
                            kernel-modules 5.1.8-300.fc30 -> 5.1.7-300.fc30
                            kernel-modules-extra 5.1.8-300.fc30 -> 5.1.7-300.fc30
                            podman 2:1.4.0-2.fc30 -> 2:1.3.1-1.git7210727.fc30
                            podman-manpages 2:1.4.0-2.fc30 -> 2:1.3.1-1.git7210727.fc30
                            python3-urllib3 1.24.3-1.fc30 -> 1.24.2-1.fc30
                            vim-minimal 2:8.1.1517-1.fc30 -> 2:8.1.1471-1.fc30
           LayeredPackages: compat-openssl10 fedora-workstation-repositories golang-bin krb5-workstation mozilla-fira-mono-fonts powerline qemu-kvm solaar syncthing virt-manager
             LocalPackages: libvirt-daemon-config-nwfilter-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-scsi-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-nwfilter-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-sheepdog-5.1.0-8.fc30.x86_64 libvirt-daemon-config-network-5.1.0-8.fc30.x86_64 libvirt-devel-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-mpath-5.1.0-8.fc30.x86_64 google-chrome-stable-75.0.3770.80-1.x86_64 libvirt-daemon-driver-storage-logical-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-iscsi-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-gluster-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-core-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-disk-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-zfs-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-iscsi-direct-5.1.0-8.fc30.x86_64
                            libvirt-libs-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-nodedev-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-storage-rbd-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-storage-5.1.0-8.fc30.x86_64 libvirt-5.1.0-8.fc30.x86_64 libvirt-daemon-kvm-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-qemu-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-interface-5.1.0-8.fc30.x86_64 libvirt-bash-completion-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-secret-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-lxc-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-vbox-5.1.0-8.fc30.x86_64 libvirt-daemon-driver-network-5.1.0-8.fc30.x86_64
                            libvirt-daemon-driver-libxl-5.1.0-8.fc30.x86_64 libvirt-daemon-5.1.0-8.fc30.x86_64 libvirt-client-5.1.0-8.fc30.x86_64

Comment 1 Jens Petersen 2019-06-14 10:13:27 UTC
FWIW I can reproduce this in an up-to-date F30 Workstation too, with a newly created toolbox:

$ toolbox create -r 29
$ toolbox enter -r 29
toolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-29

Comment 2 Akira TAGOH 2019-06-14 11:05:09 UTC
There seems two issues here.

1) Unable to copy a file into rootless container. needing --pause=false as podman proposes:

$ podman cp /etc/profile.d/toolbox.sh fedora-toolbox-30:/etc/profile.d
Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying

2) podman seems trying to create a directory with the source name. but it already exists in container as a file. then fail.

$ podman cp --pause=false /etc/profile.d/toolbox.sh fedora-toolbox-30:/etc/profile.d
Error: error copying "/etc/profile.d/toolbox.sh" to "/var/home/tagoh/.local/share/containers/storage/overlay/f8bd6ca0826f345921a1aa542177d549d473fc78033b5d29db9d71d95301067d/merged/etc/profile.d/toolbox.sh/toolbox.sh": mkdir /var/home/tagoh/.local/share/containers/storage/overlay/f8bd6ca0826f345921a1aa542177d549d473fc78033b5d29db9d71d95301067d/merged/etc/profile.d/toolbox.sh: not a directory

Comment 3 Jens Petersen 2019-07-09 07:37:18 UTC
I believe this issue could be closed now.

Comment 4 Debarshi Ray 2019-10-01 20:41:11 UTC
Yes, we stopped using 'podman cp' altogether in toolbox-0.0.12 onwards.