Bug 1810768

Summary: Podman pushes a v1 manifest when pushing to the same repo but a different tag
Product: [Fedora] Fedora Reporter: Matt Prahl <mprahl>
Component: podmanAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: bbaude, debarshir, dwalsh, jnovy, lsm5, lucarval, mheon, mhofmann, mitr, rh.container.bot, santiago, tsweeney, vrothber
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman 3.1.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-25 15:45:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matt Prahl 2020-03-05 20:58:16 UTC
Description of problem:

I created a new organization and repository in quay.io. I pushed a local image to that repository, and podman pushed a v2 manifest as expected. When I pushed that image again to the same repository but using a different tag, podman pushes it as a v1 manifest, which is not expected.

See the commands I ran with verbose output in the "Actual results" section.


Version-Release number of selected component (if applicable):
podman-1.7.0-3.fc30.x86_64

How reproducible:
Reproducible

Steps to Reproduce:
1. Create a new organization and repository in quay.io
2. Push a local image to that repository
3. Push the same local image to that repository but specifying a different tag

Actual results:

First Push
----------

$ podman push --authfile temp.json --log-level=debug iib-build:69-amd64 docker://quay.io/temp-iib/iib:test2
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1002` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/iib-worker/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/iib-worker/.local/share/containers/storage 
DEBU[0000] Using run root /var/tmp/run-1002/containers  
DEBU[0000] Using static dir /home/iib-worker/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/tmp/run-1002/libpod/tmp   
DEBU[0000] Using volume path /home/iib-worker/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1002` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/iib-worker/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/iib-worker/.local/share/containers/storage 
DEBU[0000] Using run root /var/tmp/run-1002/containers  
DEBU[0000] Using static dir /home/iib-worker/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/tmp/run-1002/libpod/tmp   
DEBU[0000] Using volume path /home/iib-worker/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] Failed to add podman to systemd sandbox cgroup: exec: "dbus-launch": executable file not found in $PATH 
INFO[0000] running as rootless                          
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1002` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/iib-worker/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/iib-worker/.local/share/containers/storage 
DEBU[0000] Using run root /var/tmp/run-1002/containers  
DEBU[0000] Using static dir /home/iib-worker/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/tmp/run-1002/libpod/tmp   
DEBU[0000] Using volume path /home/iib-worker/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] parsed reference into "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/iib-build:69-amd64" 
DEBU[0000] reference "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/iib-build:69-amd64" does not resolve to an image ID 
DEBU[0000] parsed reference into "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]localhost/iib-build:69-amd64" 
DEBU[0000] parsed reference into "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0000] Returning credentials from temp.json         
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000] Using blob info cache at /home/iib-worker/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/iib-worker/.local/share/containers/storage]@27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:6d9f9156f0c159370f4350070ba425cee4ff8ba2ccf4f16780d118814537f817 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:c5fd858e68a1c419b3015e1ca4fbf6fd4924c1ed1cfe6795c4c606d681e8bb97 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:955fd64b607d560dbfd9f03335a0d31b4d94a67b1b31d7d953ba0fdbdbdc634c 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:0bc90931771724b917cdc2dff9b2ed6dbf4071c158acf2466ef754f7ac8a0af0 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:32de5a5806e49b57399d012a4991b0cba03b6eb94774b18b5d152e6379918259 
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:6d9f9156f0c159370f4350070ba425cee4ff8ba2ccf4f16780d118814537f817 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:c5fd858e68a1c419b3015e1ca4fbf6fd4924c1ed1cfe6795c4c606d681e8bb97 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:32de5a5806e49b57399d012a4991b0cba03b6eb94774b18b5d152e6379918259 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:0bc90931771724b917cdc2dff9b2ed6dbf4071c158acf2466ef754f7ac8a0af0 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:955fd64b607d560dbfd9f03335a0d31b4d94a67b1b31d7d953ba0fdbdbdc634c 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad in quay.io/rh-iib-stage/iib 
DEBU[0000] Checking /v2/rh-iib-stage/iib/blobs/sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Arh-iib-stage%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd in quay.io/rh-iib-stage/iib 
DEBU[0000] Checking /v2/rh-iib-stage/iib/blobs/sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Arh-iib-stage%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] exporting filesystem layer "7f7ce55250e6b2d585c5442558715dbf39c40b18118cc2ecf1c66399169983ba" without compression for blob "sha256:6d9f9156f0c159370f4350070ba425cee4ff8ba2ccf4f16780d118814537f817" 
DEBU[0000] overlay: mount_data=lowerdir=/home/iib-worker/.local/share/containers/storage/overlay/l/QUEZU4HSSJBOEDGZFHFOXLZVAD:/home/iib-worker/.local/share/containers/storage/overlay/l/GGHMA32SNIJNO23PX2NNXLXJDL:/home/iib-worker/.local/share/containers/storage/overlay/l/4WHPANUIJBGSNH5M7WN6RVLRUF:/home/iib-worker/.local/share/containers/storage/overlay/l/PIOGBWFOAM5UHH6W5FOFJ2CJTY:/home/iib-worker/.local/share/containers/storage/overlay/l/J42EFYGTJHT5CE54REE5GE5I6A,upperdir=/home/iib-worker/.local/share/containers/storage/overlay/7f7ce55250e6b2d585c5442558715dbf39c40b18118cc2ecf1c66399169983ba/diff,workdir=/home/iib-worker/.local/share/containers/storage/overlay/7f7ce55250e6b2d585c5442558715dbf39c40b18118cc2ecf1c66399169983ba/work 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 in quay.io/rh-iib-stage/iib 
DEBU[0000] Checking /v2/rh-iib-stage/iib/blobs/sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Arh-iib-stage%2Fiib%3Apull&service=quay.io 
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/temp-iib/iib/blobs/uploads/    
DEBU[0000] POST https://quay.io/v2/temp-iib/iib/blobs/uploads/ 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:b3df4eef66106f91cf317f82f8304a80587ed6c5cbeb69d4d60c426cc9085a8c in quay.io/rh-iib-stage/iib 
DEBU[0000] Checking /v2/rh-iib-stage/iib/blobs/sha256:b3df4eef66106f91cf317f82f8304a80587ed6c5cbeb69d4d60c426cc9085a8c 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Arh-iib-stage%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:5c2f92cb66c07d1ed7e48745f036cfae56f5983a0d5d61a6a134cd4893678b49 in quay.io/rh-iib-stage/iib 
DEBU[0000] Checking /v2/rh-iib-stage/iib/blobs/sha256:5c2f92cb66c07d1ed7e48745f036cfae56f5983a0d5d61a6a134cd4893678b49 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Arh-iib-stage%2Fiib%3Apull&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/rh-iib-stage/iib/blobs/sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd 
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/rh-iib-stage/iib/blobs/sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/rh-iib-stage/iib/blobs/sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/rh-iib-stage/iib/blobs/sha256:b3df4eef66106f91cf317f82f8304a80587ed6c5cbeb69d4d60c426cc9085a8c 
DEBU[0000] ... not authorized                           
DEBU[0000] ... Failed: Error checking whether a blob sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd exists in quay.io/rh-iib-stage/iib: unauthorized: authentication required 
DEBU[0000] exporting filesystem layer "efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d" without compression for blob "sha256:efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d" 
DEBU[0000] overlay: mount_data=lowerdir=/home/iib-worker/.local/share/containers/storage/overlay/efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d/empty,upperdir=/home/iib-worker/.local/share/containers/storage/overlay/efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d/diff,workdir=/home/iib-worker/.local/share/containers/storage/overlay/efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d/work 
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/temp-iib/iib/blobs/uploads/    
DEBU[0000] POST https://quay.io/v2/temp-iib/iib/blobs/uploads/ 
DEBU[0000] ... not authorized                           
DEBU[0000] ... Failed: Error checking whether a blob sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 exists in quay.io/rh-iib-stage/iib: unauthorized: authentication required 
DEBU[0000] exporting filesystem layer "81b46f240a2e791c9592d383a15a2e04564dbd1b7f6e8058becbba19c94eff05" without compression for blob "sha256:c5fd858e68a1c419b3015e1ca4fbf6fd4924c1ed1cfe6795c4c606d681e8bb97" 
DEBU[0000] overlay: mount_data=lowerdir=/home/iib-worker/.local/share/containers/storage/overlay/l/J42EFYGTJHT5CE54REE5GE5I6A,upperdir=/home/iib-worker/.local/share/containers/storage/overlay/81b46f240a2e791c9592d383a15a2e04564dbd1b7f6e8058becbba19c94eff05/diff,workdir=/home/iib-worker/.local/share/containers/storage/overlay
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob efe2c0ceb24b [--------------------------------------] 8.0b / 205.2MiB
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/temp-iib/iib/blobs/uploads/    
DEBU[0000] POST https://quay.io/v2/temp-iib/iib/blobs/uploads/ 
DEBU[0000] ... not authorized                           
DEBU[0000] ... Failed: Error checking whether a blob sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad exists in quay.io/rh-iib-stage/iib: unauthorized: authentication required 
DEBU[0000] exporting filesystem layer "07aef82d692f476c25f9412db2c2de15795c8b10d5a457addba763c5cf985cfb" without compression for blob "sha256:0bc90931771724b917cdc2dff9b2ed6dbf4071c158acf2466ef754f7ac8a0af0" 
DEBU[0000] overlay: mount_data=lowerdir=/home/iib-worker/.local/share/containers/storage/overlay/l/PIOGBWFOAM5UHH6W5FOFJ2CJTY:/home/iib-worker/.local/share/containers/storage/overlay/l/J42EFYGTJHT5CE54REE5GE5I6A,upperdir=/home/iib-worker/.local/share/containers/storage/overlay/07aef82d692f476c25f9412db2c2de15795c8b10d5a457addba763c5cf985cfb/diff,workdir=/home/iib-worker/.local/share/containers/storage/overlay/07aef82d692f476c25f9412db2c2de15795c8b10d5a457addba763c5cf985cfb/work 
DEBU[0000] ... not authorized                           
DEBU[0000] ... Failed: Error checking whether a blob sha256:b3df4eef66106f91cf317f82f8304a80587ed6c5cbeb69d4d60c426cc9085a8c exists in quay.io/rh-iib-stage/iib: unauthorized: authentication required 
DEBU[0000] exporting filesystem layer "cedf6f9871af41f250728eaa13615ecf34f3ed211d1e0496dfb5d34a503c5943" without compression for blob "sha256:32de5a5806e49b57399d012a4991b0cba03b6eb94774b18b5d152e6379918259" 
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/temp-iib/iib/blobs/uploads/    
DEBU[0000] POST https://quay.io/v2/temp-iib/iib/blobs/uploads/ 
DEBU[0000] overlay: mount_data=lowerdir=/home/iib-worker/.local/share/containers/storage/overlay/l/GGHMA32SNIJNO23PX2NNXLXJDL:/home/iib-worker/.local/share/containers/storage/overlay/l/4WHPANUIJBGSNH5M7WN6RVLRUF:/home/iib-worker/.local/share/containers/storage/overlay/l/PIOGBWFOAM5UHH6W5FOFJ2CJTY:/home/iib-worker/.local/share/containers/storage/overlay/l/J42EFYGTJHT5CE54REE5GE5I6A,upperdir=/home/iib-worker/.local/share/containers/storage/overlay/cedf6f9871af41f250728eaa13615ecf34f3ed211d1e0496dfb5d34a503c5943/diff,workdir=/home/iib-worker/.local/share/containers/storage/overlay/cedf6f9871af41f250728eaa13615ecf34f3ed211d1e0496dfb5d34a503c5943/work 
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/temp-iib/iib/blobs/uploads/    
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob efe2c0ceb24b [--------------------------------------] 8.0b / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob efe2c0ceb24b [====>---------------------------------] 25.0MiB / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 [===================================>--] 188.6MiB / 200.4MiB
DEBU[0002] ... not authorized                           
DEBU[0002] ... Failed: Error checking whether a blob sha256:5c2f92cb66c07d1ed7e48745f036cfae56f5983a0d5d61a6a134cd4893678b49 exists in quay.io/rh-iib-stage/iib: unauthorized: authentication required 
DEBU[0002] exporting filesystem layer "64da43ab3fc81210114b1e68282c13c878744492b3048343e18b7e26276f9f1d" without compression for blob "sha256:955fd64b607d560dbfd9f
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob efe2c0ceb24b [====>---------------------------------] 28.6MiB / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
DEBU[0002] No compression detected                      
DEBU[0002] Compressing blob on the fly                  
DEBU[0002] Uploading /v2/temp-iib/iib/blobs/uploads/    
DEBU[0002] POST https://quay.io/v2/temp-iib/iib/blobs/uploads/ 
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob efe2c0ceb24b [========>-----------------------------] 49.8MiB / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob efe2c0ceb24b [==========>---------------------------] 61.3MiB / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 6d9f9156f0c1 [--------------------------------------] 8.0b / 118.0KiB
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b [==============>-----------------------] 83.6MiB / 205.2MiB
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b [==================>-------------------] 101.5MiB / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b [======================>---------------] 125.8MiB / 205.2MiB
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b [========================>-------------] 137.0MiB / 205.2MiB
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b done  
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b done  
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b done  
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b done  
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 6d9f9156f0c1 done  
Copying blob efe2c0ceb24b done  
Copying blob c5fd858e68a1 done  
Copying blob 0bc909317717 done  
Copying blob 32de5a5806e4 done  
Copying blob 955fd64b607d done  
DEBU[0010] exporting opaque data as blob "sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0010] No compression detected                      
DEBU[0010] Using original blob without modification     
DEBU[0010] Checking /v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0010] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0010] ... not present                              
DEBU[0010] Uploading /v2/temp-iib/iib/blobs/uploads/    
DEBU[0010] POST https://quay.io/v2/temp-iib/iib/blobs/uploads/ 
Copying config 27d3ea8d6f [--------------------------------------] 0.0b / 3.6KiB
Copying config 27d3ea8d6f done  
DEBU[0011] PUT https://quay.io/v2/temp-iib/iib/blobs/uploads/577299c3-ee22-49ee-856a-d7bf041d922c?digest=sha256%3A27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f
Copying config 27d3ea8d6f done  
Copying config 27d3ea8d6f done  
Writing manifest to image destination
DEBU[0011] PUT https://quay.io/v2/temp-iib/iib/manifests/test2 
DEBU[0011] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest: Error uploading manifest test2 to quay.io/temp-iib/iib: manifest invalid: manifest invalid 
DEBU[0011] Trying to use manifest type application/vnd.docker.distribution.manifest.v2+json… 
DEBU[0011] exporting opaque data as blob "sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0011] No compression detected                      
DEBU[0011] Using original blob without modification     
DEBU[0011] Checking /v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0011] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0011] ... already exists                           
Writing manifest to image destination
DEBU[0011] PUT https://quay.io/v2/temp-iib/iib/manifests/test2 
Storing signatures
DEBU[0013] Successfully pushed docker://quay.io/temp-iib/iib:test2 with digest sha256:22b6a12f6dfd7374cb75a1090e31e930808318ebd8374ef659f323f3877a7b48 


Second Push
-----------
$ podman push --authfile temp.json --log-level=debug iib-build:69-amd64 docker://quay.io/temp-iib/iib:test3
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1002` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/iib-worker/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/iib-worker/.local/share/containers/storage 
DEBU[0000] Using run root /var/tmp/run-1002/containers  
DEBU[0000] Using static dir /home/iib-worker/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/tmp/run-1002/libpod/tmp   
DEBU[0000] Using volume path /home/iib-worker/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1002` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/iib-worker/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/iib-worker/.local/share/containers/storage 
DEBU[0000] Using run root /var/tmp/run-1002/containers  
DEBU[0000] Using static dir /home/iib-worker/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/tmp/run-1002/libpod/tmp   
DEBU[0000] Using volume path /home/iib-worker/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] Failed to add podman to systemd sandbox cgroup: exec: "dbus-launch": executable file not found in $PATH 
INFO[0000] running as rootless                          
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1002` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/iib-worker/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/iib-worker/.local/share/containers/storage 
DEBU[0000] Using run root /var/tmp/run-1002/containers  
DEBU[0000] Using static dir /home/iib-worker/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/tmp/run-1002/libpod/tmp   
DEBU[0000] Using volume path /home/iib-worker/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] parsed reference into "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/iib-build:69-amd64" 
DEBU[0000] reference "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/iib-build:69-amd64" does not resolve to an image ID 
DEBU[0000] parsed reference into "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]localhost/iib-build:69-amd64" 
DEBU[0000] parsed reference into "[overlay@/home/iib-worker/.local/share/containers/storage+/var/tmp/run-1002/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0000] Returning credentials from temp.json         
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000] Using blob info cache at /home/iib-worker/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/iib-worker/.local/share/containers/storage]@27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:6d9f9156f0c159370f4350070ba425cee4ff8ba2ccf4f16780d118814537f817 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:c5fd858e68a1c419b3015e1ca4fbf6fd4924c1ed1cfe6795c4c606d681e8bb97 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:955fd64b607d560dbfd9f03335a0d31b4d94a67b1b31d7d953ba0fdbdbdc634c 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:0bc90931771724b917cdc2dff9b2ed6dbf4071c158acf2466ef754f7ac8a0af0 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:32de5a5806e49b57399d012a4991b0cba03b6eb94774b18b5d152e6379918259 
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:32de5a5806e49b57399d012a4991b0cba03b6eb94774b18b5d152e6379918259 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:955fd64b607d560dbfd9f03335a0d31b4d94a67b1b31d7d953ba0fdbdbdc634c 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:6d9f9156f0c159370f4350070ba425cee4ff8ba2ccf4f16780d118814537f817 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:c5fd858e68a1c419b3015e1ca4fbf6fd4924c1ed1cfe6795c4c606d681e8bb97 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:0bc90931771724b917cdc2dff9b2ed6dbf4071c158acf2466ef754f7ac8a0af0 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:5c2f92cb66c07d1ed7e48745f036cfae56f5983a0d5d61a6a134cd4893678b49 in quay.io/temp-iib/iib 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:5c2f92cb66c07d1ed7e48745f036cfae56f5983a0d5d61a6a134cd4893678b49 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Atemp-iib%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd in quay.io/temp-iib/iib 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Atemp-iib%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:ba4f5f11713df9b1dcf7ebf5e1f29e4b21994d6a3b2a63199be20d0953ddbef6 in quay.io/temp-iib/iib 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:ba4f5f11713df9b1dcf7ebf5e1f29e4b21994d6a3b2a63199be20d0953ddbef6 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Atemp-iib%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad in quay.io/temp-iib/iib 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Atemp-iib%2Fiib%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 in quay.io/temp-iib/iib 
DEBU[0000] Checking /v2/temp-iib/iib/blobs/sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 
DEBU[0000] GET https://quay.io/v2/auth?account=temp-iib%2Biib&scope=repository%3Atemp-iib%2Fiib%3Apull%2Cpush&scope=repository%3Atemp-iib%2Fiib%3Apull&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:4fb3f774cece8cd660186c7d7246fedea8eead20ef13d22d323a99690e14f8fd 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:5c2f92cb66c07d1ed7e48745f036cfae56f5983a0d5d61a6a134cd4893678b49 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:806fce4b820395f159e7b67b378b3a06cfee851331bf5a7468eca1b14c5c4cad 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:ba4f5f11713df9b1dcf7ebf5e1f29e4b21994d6a3b2a63199be20d0953ddbef6 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:38a2bcbfbc279b983bb006ab43692aabc61689af3e21db0adb25a8a223c616f5 
DEBU[0000] ... already exists                           
DEBU[0000] Skipping blob sha256:efe2c0ceb24b399543a450a041fe293288c398d90c8405f7a5b493e78f2cc68d (already present): 
Copying blob efe2c0ceb24b skipped: already exists  
DEBU[0000] ... already exists                           
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob 955fd64b607d skipped: already exists  
DEBU[0000] ... already exists                           
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob 955fd64b607d skipped: already exists  
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob 955fd64b607d skipped: already exists  
Copying blob 0bc909317717 skipped: already exists  
Copying blob 6d9f9156f0c1 skipped: already exists  
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob efe2c0ceb24b skipped: already exists  
Copying blob 955fd64b607d skipped: already exists  
Copying blob 0bc909317717 skipped: already exists  
Copying blob 6d9f9156f0c1 skipped: already exists  
Copying blob c5fd858e68a1 skipped: already exists  
Copying blob 32de5a5806e4 [--------------------------------------] 0.0b / 0.0b
DEBU[0001] exporting opaque data as blob "sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0001] No compression detected                      
DEBU[0001] Using original blob without modification     
DEBU[0001] Checking /v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0001] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0001] ... already exists                           
Writing manifest to image destination
DEBU[0001] PUT https://quay.io/v2/temp-iib/iib/manifests/test3 
DEBU[0001] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest: Error uploading manifest test3 to quay.io/temp-iib/iib: manifest invalid: manifest invalid 
DEBU[0001] Trying to use manifest type application/vnd.docker.distribution.manifest.v2+json… 
DEBU[0001] exporting opaque data as blob "sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0001] No compression detected                      
DEBU[0001] Using original blob without modification     
DEBU[0001] Checking /v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0001] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001 
DEBU[0002] ... already exists                           
Writing manifest to image destination
DEBU[0002] PUT https://quay.io/v2/temp-iib/iib/manifests/test3 
DEBU[0002] Upload of manifest type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest: Error uploading manifest test3 to quay.io/temp-iib/iib: manifest invalid: manifest invalid 
DEBU[0002] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws… 
DEBU[0002] exporting opaque data as blob "sha256:27d3ea8d6f64359e22425b0f88b252f207e0867792dc1b20f05040c5f8b0f001" 
DEBU[0002] Uploading empty layer during conversion to schema 1 
DEBU[0002] Checking /v2/temp-iib/iib/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0002] HEAD https://quay.io/v2/temp-iib/iib/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0002] ... already exists                           
Writing manifest to image destination
DEBU[0002] PUT https://quay.io/v2/temp-iib/iib/manifests/test3 
Storing signatures
DEBU[0003] Successfully pushed docker://quay.io/temp-iib/iib:test3 with digest sha256:f9e3a4b2dea7f42d2225c95db2fe07180906e0b15705bffbe7bc99176a883c99

Expected results:

The second push should product a v2 manifest

Additional info:

My current workaround is to run:
skopeo copy --format v2s2 $destination $destination

Comment 1 Lokesh Mandvekar 2020-03-19 14:29:07 UTC
Tom, could you PTAL?

Comment 2 Tom Sweeney 2020-03-19 21:52:46 UTC
Matt,

I'm not seeing this problem with upstream and you're a few versions behind latest.  Any chance you could upgrade Podman and try again?

Also is there a difference in behavior if you first tag the image `podman tag iib-build:69-amd64 quay.io/temp-iib/iib:test3` and then do `podman push quay.io/temp-iib/iib:test3`?

Matt Heon does this sound at all familiar?

Comment 3 Matt Prahl 2020-03-20 13:08:01 UTC
Hi Tom,
The same issue is present on the latest in Fedora 30 (podman-2:1.8.0-4.fc30.x86_64).

Could you provide a scratch build for Fedora 30 or RHEL 8 of the latest podman and crun? The VM I'm seeing this on runs on RHEL 8 and I can't install F31+ RPMs on it due to the switch to zstd compression on RPMs in F31.

Comment 4 Tom Sweeney 2020-03-20 20:52:45 UTC
Hi Matt,

The smoothest way would be to either have you git clone the Libpod repo and build Podman (https://podman.io/getting-started/installation) or secondarily you could pull the container image from quay.io at quay.io/podman/upstream:latest and see if either of those work.

Can you try one of those routes?

Comment 5 Matt Prahl 2020-03-24 19:58:41 UTC
Hi Tom,
I tried podman 1.8.2 from the following repository, but I get the same issue:
https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo

Here is the output:


Build the Image
---------------
$ podman -v
podman version 1.8.2
$ cat Dockerfile 
FROM centos:8

RUN echo hello

$ podman build -t reproducer:1 -f Dockerfile .
STEP 1: FROM centos:8
STEP 2: RUN echo hello
hello
STEP 3: COMMIT reproducer:1
--> 8dde1584cff
8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2



First Push
----------

$ podman push --authfile=./auth.json --log-level=debug reproducer:1 quay.io/temp/reproducer:1
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-runtime: no valid executable found for OCI runtime kata-runtime: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-qemu: no valid executable found for OCI runtime kata-qemu: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-fc: no valid executable found for OCI runtime kata-fc: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0000] Default CNI network name podman is unchangeable 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/reproducer:1" 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/reproducer:1" does not resolve to an image ID 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]localhost/reproducer:1" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0000] Returning credentials from ./auth.json       
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]@8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:43c079536031ea0173ca8f3076aebb101685ac8da3fbd6eba7cf42a558dce87c 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd 
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:43c079536031ea0173ca8f3076aebb101685ac8da3fbd6eba7cf42a558dce87c 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 in quay.io/temp/reproducer 
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&scope=repository%3Atemp%2Freproducer%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] exporting filesystem layer "26aae65626fe7464fbd25ceeb715dcde030c8276076f0e5db1385e9ce8cb0e6f" without compression for blob "sha256:43c079536031ea0173ca8f3076aebb101685ac8da3fbd6eba7cf42a558dce87c" 
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/2EO4HLZ63OSTE6RCVMYZXSKLAC,upperdir=/var/lib/containers/storage/overlay/26aae65626fe7464fbd25ceeb715dcde030c8276076f0e5db1385e9ce8cb0e6f/diff,workdir=/var/lib/containers/storage/overlay/26aae65626fe7464fbd25ceeb715dcde030c8276076f0e5db1385e9ce8cb0e6f/work 
DEBU[0000] No compression detected                      
DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/temp/reproducer/blobs/uploads/ 
DEBU[0000] POST https://quay.io/v2/temp/reproducer/blobs/uploads/ 
Copying blob 43c079536031 [--------------------------------------] 8.0b / 2.5KiB
Copying blob 43c079536031 done  
DEBU[0000] Increasing token expiration to: 60 seconds   
Copying blob 43c079536031 done  
DEBU[0001] ... not present                              
DEBU[0001] Trying to reuse cached location sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 in quay.io/temp/centos 
DEBU[0001] Checking /v2/temp/centos/blobs/sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 
DEBU[0001] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&scope=repository%3Atemp%2Fcentos%3ApuCopying blob 43c079536031 done  
DEBU[0001] PUT https://quay.io/v2/temp/reproducer/blobs/uploads/c8048da1-a8fd-4dab-99bb-4f8d4c2b1c76?digest=sha256%3A2d84362881e26b2da037e1a5eCopying blob 43c079536031 done  
DEBU[0001] Increasing token expiration to: 60 seconds   
Copying blob 43c079536031 done  
DEBU[0001] ... not present                              
DEBU[0001] exporting filesystem layer "aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd" without compression for blob "sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd" 
DEBU[0001] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd/empty,upperdir=/var/lib/containers/storage/overlay/aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd/diff,workdir=/var/lib/containers/storage/overlay/aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd/work 
DEBU[0001] No compression detected                      
DEBU[0001] Compressing blob on the fly                  
DEBU[0001] Uploading /v2/temp/reproducer/blobs/uploads/ 
Copying blob 43c079536031 done  
Copying blob 43c079536031 done  
Copying blob 43c079536031 done  
Copying blob aff9833d4641 done  
Copying blob 43c079536031 done  
Copying blob aff9833d4641 done  
DEBU[0008] Upload of layer sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 complete 
DEBU[0009] exporting opaque data as blob "sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0009] No compression detected                      
DEBU[0009] Using original blob without modification     
DEBU[0009] Checking /v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0009] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0009] ... not present                              
DEBU[0009] Uploading /v2/temp/reproducer/blobs/uploads/ 
DEBU[0009] POST https://quay.io/v2/temp/reproducer/blobs/uploads/ 
Copying config 8dde1584cf [--------------------------------------] 0.0b / 1.2KiB
Copying config 8dde1584cf done  
DEBU[0009] PUT https://quay.io/v2/temp/reproducer/blobs/uploads/db07c76f-70de-4fd1-b9d1-3f5b1b477b90?digest=sha256%3A8dde1584cff50d2bbc2d7337eCopying config 8dde1584cf done  
DEBU[0010] Upload of layer sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 complete 
Writing manifest to image destination
DEBU[0010] PUT https://quay.io/v2/temp/reproducer/manifests/1 
DEBU[0010] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest: Error uploading manifest 1 to quay.io/temp/reproducer: manifest invalid: manifest invalid 
DEBU[0010] Trying to use manifest type application/vnd.docker.distribution.manifest.v2+json… 
DEBU[0010] exporting opaque data as blob "sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0010] No compression detected                      
DEBU[0010] Using original blob without modification     
DEBU[0010] Checking /v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0010] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0010] ... already exists                           
Writing manifest to image destination
DEBU[0010] PUT https://quay.io/v2/temp/reproducer/manifests/1 
Storing signatures
DEBU[0010] Successfully pushed docker://quay.io/temp/reproducer:1 with digest sha256:39c44cb791119b1e405467c1323e1611513c5a07ecb7d01cd680f81c409eea8a


Second Push
-----------
$ podman push --authfile=./auth.json --log-level=debug reproducer:1 quay.io/temp/reproducer:2
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
WARN[0000] Error initializing configured OCI runtime kata-fc: no valid executable found for OCI runtime kata-fc: invalid argument 
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-runtime: no valid executable found for OCI runtime kata-runtime: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-qemu: no valid executable found for OCI runtime kata-qemu: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0000] Default CNI network name podman is unchangeable 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/reproducer:1" 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/reproducer:1" does not resolve to an image ID 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]localhost/reproducer:1" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0000] Returning credentials from ./auth.json       
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]@8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:43c079536031ea0173ca8f3076aebb101685ac8da3fbd6eba7cf42a558dce87c 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd 
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&service=quay.io 
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:43c079536031ea0173ca8f3076aebb101685ac8da3fbd6eba7cf42a558dce87c 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:2d84362881e26b2da037e1a5edc7b89003b549ac015265f12b75390eccffb982 in quay.io/temp/reproducer 
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:2d84362881e26b2da037e1a5edc7b89003b549ac015265f12b75390eccffb982 
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&scope=repository%3Atemp%2Freproducer%3Apull&service=quay.io 
DEBU[0000] ... not present                              
DEBU[0000] Trying to reuse cached location sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 in quay.io/temp/reproducer 
DEBU[0000] Checking /v2/temp/reproducer/blobs/sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 
DEBU[0000] GET https://quay.io/v2/auth?account=mprahl&scope=repository%3Atemp%2Freproducer%3Apull%2Cpush&scope=repository%3Atemp%2Freproducer%3Apull&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:2d84362881e26b2da037e1a5edc7b89003b549ac015265f12b75390eccffb982 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4 
DEBU[0001] ... already exists                           
DEBU[0001] Skipping blob sha256:43c079536031ea0173ca8f3076aebb101685ac8da3fbd6eba7cf42a558dce87c (already present): 
Copying blob 43c079536031 skipped: already exists  
DEBU[0001] ... already exists                           
Copying blob 43c079536031 skipped: already exists  
Copying blob aff9833d4641 [--------------------------------------] 0.0b / 0.0b
DEBU[0001] exporting opaque data as blob "sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0001] No compression detected                      
DEBU[0001] Using original blob without modification     
DEBU[0001] Checking /v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0001] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0001] ... already exists                           
Writing manifest to image destination
DEBU[0001] PUT https://quay.io/v2/temp/reproducer/manifests/2 
DEBU[0001] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest: Error uploading manifest 2 to quay.io/temp/reproducer: manifest invalid: manifest invalid 
DEBU[0001] Trying to use manifest type application/vnd.docker.distribution.manifest.v2+json… 
DEBU[0001] exporting opaque data as blob "sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0001] No compression detected                      
DEBU[0001] Using original blob without modification     
DEBU[0001] Checking /v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0001] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2 
DEBU[0001] ... already exists                           
Writing manifest to image destination
DEBU[0001] PUT https://quay.io/v2/temp/reproducer/manifests/2 
DEBU[0001] Upload of manifest type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest: Error uploading manifest 2 to quay.io/temp/reproducer: manifest invalid: manifest invalid 
DEBU[0001] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws… 
DEBU[0001] exporting opaque data as blob "sha256:8dde1584cff50d2bbc2d7337e42e0e4eb86113d3dedf6ec2ea7687ad174be2f2" 
DEBU[0001] Uploading empty layer during conversion to schema 1 
DEBU[0001] Checking /v2/temp/reproducer/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0001] HEAD https://quay.io/v2/temp/reproducer/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0001] ... already exists                           
Writing manifest to image destination
DEBU[0001] PUT https://quay.io/v2/temp/reproducer/manifests/2 
Storing signatures
DEBU[0001] Successfully pushed docker://quay.io/temp/reproducer:2 with digest sha256:7e46b69403820d80b9cd0e26df4fad60992f389ee0cc53fb1e9f754ee824cd27

Comment 6 Tom Sweeney 2020-04-29 21:57:51 UTC
Miloslav, any thoughts on why a second push would change the manifest from v2 to v1?

Comment 7 Miloslav Trmač 2020-04-29 22:03:57 UTC
I’m fairly sure this is https://github.com/containers/image/issues/733 . Quay rejects the schema2 manifest with non-matching MIME types, causing the push to fall back to schema1, and that succeeds because schema1 does not contain MIME types.

Comment 8 Ben Cotton 2020-04-30 20:10:45 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 9 Matt Prahl 2020-05-18 17:22:13 UTC
Hello,
Is there any update on this? It'd be nice to get this fixed.

Comment 10 Tom Sweeney 2020-05-18 17:51:20 UTC
Miloslav (or Valentin), were you able to confirm that the PR's mentioned in https://github.com/containers/image/issues/733 will fix this?  (https://github.com/containers/image/pull/904 and https://github.com/containers/image/pull/909)

If that's the case, it should be in c/image 5.4.4 which should land in Podman v2.0.  It doesn't look to have been included in the Pdoman v1.9 release branch at this time though.

Comment 11 Miloslav Trmač 2020-05-18 18:55:40 UTC
There is no PR for https://github.com/containers/image/issues/733 yet.

Comment 12 Ben Cotton 2020-11-03 16:26:19 UTC
This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 13 Daniel Walsh 2021-01-29 10:14:42 UTC
I believe this is fixed in podman 3.0, at least by following the linked issues above.

I will mark this bug as modified and fixed in podman 3.0

Reopen if I am mistaken.

Comment 14 Jindrich Novy 2021-01-29 20:29:49 UTC
Assigning to Lokesh as he handles Fedora.

Comment 15 Fedora Program Management 2021-04-29 16:13:32 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Ben Cotton 2021-05-25 15:45:20 UTC
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 18 Luiz Carvalho 2021-05-25 17:12:44 UTC
I can no longer reproduce the issue on Fedora 33 with podman 3.1.2. Thank you!

Comment 19 Red Hat Bugzilla 2023-09-15 00:30:07 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days