Created attachment 1542129 [details] detailed console output for build/push Description of problem: Errors while pushing images to quay (or even docker hub). Copying blob a824b1174a5f [======================================] 256.3MiB / 256.3MiB Patch https://quay.io/v2/jdiaz/hive/blobs/uploads/c009eb0d-2211-4a52-b68c-98846773d994: open /home/jdiaz/.local/share/containers/storage/overlay/a95178b59b265412ae4c8736798de6746958940f0e30402a0b6118031d51d672/merged/etc/yum.repos.d/redhat.repo: no such file or directory ERRO[0010] exit status 1 Version-Release number of selected component (if applicable): buildah-1.7-1.git873f001.fc29.x86_64 How reproducible: 100% Steps to Reproduce: 1. buildah build an image 2. buildah push the image 3. Actual results: [jdiaz@minigoomba hive (remoteingress $%)]$ buildah push quay.io/jdiaz/hive:latest Getting image source signatures Copying blob 91e4cddb1d38 [======================================] 203.8MiB / 203.8MiB Copying blob 04d580b33d5d [======================================] 10.0KiB / 10.0KiB Copying blob fb322172482d [======================================] 6.5KiB / 6.5KiB Copying blob ef91d31e8e6a [======================================] 20.1MiB / 20.1MiB Copying blob a824b1174a5f [======================================] 256.3MiB / 256.3MiB Patch https://quay.io/v2/jdiaz/hive/blobs/uploads/c009eb0d-2211-4a52-b68c-98846773d994: open /home/jdiaz/.local/share/containers/storage/overlay/a95178b59b265412ae4c8736798de6746958940f0e30402a0b6118031d51d672/merged/etc/yum.repos.d/redhat.repo: no such file or directory ERRO[0010] exit status 1 Expected results: Successful push of image. Additional info: After much trial an error, I was able to get the push to work after nuking ~/.config/containers ~/.local/share/containers and modifying the default ~/.config/containers/storage.conf away from overlay to vfs driver. My home dir is on ext4.
Created attachment 1542130 [details] console output of build/push working with vfs
Are you running as root or non root? I have a feeling this is a rootless/fuse-overlay problem?
what package are you using for fuse-overlayfs? It looks like an issue that was fixed recently there. Also, could you share your Dockerfile?
I can reproduce with: https://github.com/openshift/hive
PR here: https://github.com/containers/fuse-overlayfs/pull/53 I will keep playing with it and cut a release next week.
Sorry...long lunch :) So, you already found the repo https://github.com/openshift/hive I am running as non-root. And as I just discovered, I can push different containers without a problem. For example this worked: [jdiaz@minigoomba docker]$ cat Dockerfile FROM fedora RUN touch /foo [jdiaz@minigoomba docker]$ Then buildah bud and buildah push worked against that simple container. And for completeness, this version of fuse-overlay: fuse-overlayfs-0.3-4.dev.gitea72572.fc29.x86_64
would it be possible for you to confirm that the patch solves the problem you have seen?
It errors while trying to build (using 'buildah bud -f ./Dockerfile.static .'). Running custom install script '/build/libfuse/util/install_helper.sh /etc /usr/bin /usr/lib/udev/rules.d' mknod: /dev/fuse: Operation not permitted FAILED: meson-install /usr/bin/meson install --no-rebuild ninja: build stopped: subcommand failed. error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DISTTAG=f29container FGC=f29 FBR=f29] Command:run Args:[git clone https://github.com/libfuse/libfuse && cd libfuse && mkdir build && cd build && LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. && ninja && ninja install] Flags:[] Attrs:map[] Message:RUN git clone https://github.com/libfuse/libfuse && cd libfuse && mkdir build && cd build && LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. && ninja && ninja install Original:RUN git clone https://github.com/libfuse/libfuse && cd libfuse && mkdir build && cd build && LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. && ninja && ninja install}: error while running runtime: exit status 1 ERRO[0177] exit status 1 [jdiaz@minigoomba fuse-overlayfs (fix-lookup-if-whiteout-lower-layer=)]$
Nevermind...i got it built the classic ./configure way. Using the patched /usr/bin/fuse-overlayfs does get things working again with "overlayfs" storage driver.
thanks for verifying it
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f0f271f325
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b8667f8d1c
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b8667f8d1c
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f0f271f325
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.