Bug 1686889

Summary: buildah push failing when storage.conf driver is set to "overlay" (works with "vfs")
Product: [Fedora] Fedora Reporter: Joel Diaz <jdiaz>
Component: fuse-overlayfsAssignee: Giuseppe Scrivano <gscrivan>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: amurdaca, dwalsh, fkluknav, gscrivan, jdiaz, lsm5, nalin
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-12 22:18:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
detailed console output for build/push
none
console output of build/push working with vfs none

Description Joel Diaz 2019-03-08 15:33:12 UTC
Created attachment 1542129 [details]
detailed console output for build/push

Description of problem:
Errors while pushing images to quay (or even docker hub).
                                                                         
Copying blob a824b1174a5f [======================================] 256.3MiB / 256.3MiB                                                                          
Patch https://quay.io/v2/jdiaz/hive/blobs/uploads/c009eb0d-2211-4a52-b68c-98846773d994: open /home/jdiaz/.local/share/containers/storage/overlay/a95178b59b265412ae4c8736798de6746958940f0e30402a0b6118031d51d672/merged/etc/yum.repos.d/redhat.repo: no such file or directory                                                  
ERRO[0010] exit status 1


Version-Release number of selected component (if applicable):
buildah-1.7-1.git873f001.fc29.x86_64

How reproducible:
100%

Steps to Reproduce:
1. buildah build an image
2. buildah push the image
3.

Actual results:

[jdiaz@minigoomba hive (remoteingress $%)]$ buildah push quay.io/jdiaz/hive:latest                                                                              
Getting image source signatures
Copying blob 91e4cddb1d38 [======================================] 203.8MiB / 203.8MiB                                                                          
Copying blob 04d580b33d5d [======================================] 10.0KiB / 10.0KiB                                                                            
Copying blob fb322172482d [======================================] 6.5KiB / 6.5KiB                                                                              
Copying blob ef91d31e8e6a [======================================] 20.1MiB / 20.1MiB                                                                            
Copying blob a824b1174a5f [======================================] 256.3MiB / 256.3MiB                                                                          
Patch https://quay.io/v2/jdiaz/hive/blobs/uploads/c009eb0d-2211-4a52-b68c-98846773d994: open /home/jdiaz/.local/share/containers/storage/overlay/a95178b59b265412ae4c8736798de6746958940f0e30402a0b6118031d51d672/merged/etc/yum.repos.d/redhat.repo: no such file or directory                                                  
ERRO[0010] exit status 1



Expected results:
Successful push of image.

Additional info:

After much trial an error, I was able to get the push to work after nuking ~/.config/containers ~/.local/share/containers and modifying the default ~/.config/containers/storage.conf away from overlay to vfs driver.

My home dir is on ext4.

Comment 1 Joel Diaz 2019-03-08 15:33:54 UTC
Created attachment 1542130 [details]
console output of build/push working with vfs

Comment 2 Daniel Walsh 2019-03-08 16:11:59 UTC
Are you running as root or non root?
I have a feeling this is a rootless/fuse-overlay problem?

Comment 3 Giuseppe Scrivano 2019-03-08 16:32:17 UTC
what package are you using for fuse-overlayfs?  It looks like an issue that was fixed recently there.

Also, could you share your Dockerfile?

Comment 4 Giuseppe Scrivano 2019-03-08 16:46:46 UTC
I can reproduce with: https://github.com/openshift/hive

Comment 5 Giuseppe Scrivano 2019-03-08 17:18:52 UTC
PR here: https://github.com/containers/fuse-overlayfs/pull/53

I will keep playing with it and cut a release next week.

Comment 6 Joel Diaz 2019-03-08 18:07:54 UTC
Sorry...long lunch :)

So, you already found the repo https://github.com/openshift/hive

I am running as non-root. And as I just discovered, I can push different containers without a problem. For example this worked:

[jdiaz@minigoomba docker]$ cat Dockerfile 
FROM fedora
RUN touch /foo
[jdiaz@minigoomba docker]$

Then buildah bud and buildah push worked against that simple container.

And for completeness, this version of fuse-overlay: fuse-overlayfs-0.3-4.dev.gitea72572.fc29.x86_64

Comment 7 Giuseppe Scrivano 2019-03-08 18:13:09 UTC
would it be possible for you to confirm that the patch solves the problem you have seen?

Comment 8 Joel Diaz 2019-03-08 18:43:02 UTC
It errors while trying to build (using 'buildah bud -f ./Dockerfile.static .').

Running custom install script '/build/libfuse/util/install_helper.sh /etc /usr/bin /usr/lib/udev/rules.d'
mknod: /dev/fuse: Operation not permitted
FAILED: meson-install 
/usr/bin/meson install --no-rebuild
ninja: build stopped: subcommand failed.
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DISTTAG=f29container FGC=f29 FBR=f29] Command:run Args:[git clone https://github.com/libfuse/libfuse &&     cd libfuse &&     mkdir build &&     cd build &&     LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. &&     ninja &&     ninja install] Flags:[] Attrs:map[] Message:RUN git clone https://github.com/libfuse/libfuse &&     cd libfuse &&     mkdir build &&     cd build &&     LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. &&     ninja &&     ninja install Original:RUN git clone https://github.com/libfuse/libfuse &&     cd libfuse &&     mkdir build &&     cd build &&     LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. &&     ninja &&     ninja install}: error while running runtime: exit status 1
ERRO[0177] exit status 1                                
[jdiaz@minigoomba fuse-overlayfs (fix-lookup-if-whiteout-lower-layer=)]$

Comment 9 Joel Diaz 2019-03-08 18:58:55 UTC
Nevermind...i got it built the classic ./configure way.

Using the patched /usr/bin/fuse-overlayfs does get things working again with "overlayfs" storage driver.

Comment 10 Giuseppe Scrivano 2019-03-08 19:06:25 UTC
thanks for verifying it

Comment 11 Fedora Update System 2019-03-10 09:50:43 UTC
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f0f271f325

Comment 12 Fedora Update System 2019-03-10 09:51:01 UTC
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b8667f8d1c

Comment 13 Fedora Update System 2019-03-10 18:27:59 UTC
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b8667f8d1c

Comment 14 Fedora Update System 2019-03-10 19:13:58 UTC
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f0f271f325

Comment 15 Fedora Update System 2019-03-12 22:18:48 UTC
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2019-03-19 05:19:12 UTC
fuse-overlayfs-0.3-5.dev.git8ec68ae.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.