Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2000195

Summary:	OCP update failed - error setting value of extended attribute "user.Librepo.checksum.mtime"
Product:	OpenShift Container Platform	Reporter:	Yurii Prokulevych <yprokule>
Component:	RHCOS	Assignee:	RHCOS Bug Triage <rhcos-triage>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Michael Nguyen <mnguyen>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.7	CC:	achernet, agurenko, bbaude, dornelas, dwalsh, iheim, jerzhang, jligon, jnovy, lsm5, mcornea, mheon, miabbott, mrussell, nstielau, pthomas, smalleni, travier, tsweeney, umohnani, walters, wking
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2000743 (view as bug list)		Environment:
Last Closed:	2021-09-02 19:29:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2000743, 2000746

Comment 6 Colin Walters 2021-09-01 21:34:28 UTC

OK so one thing we don't know/understand here is if this is somehow a regression.  Under what scenarios are we falling back to podman for the pull?  Why do we still have two paths in the MCO?

Need to check if the xattrs are actually fatal or not.  Also https://github.com/coreos/coreos-assembler/issues/2401 for that.

The "broken pipe" error is odd.  I suspect there's a real underlying error here we're missing.

Comment 7 Yurii Prokulevych 2021-09-02 13:00:23 UTC

Got same error while doing 4.7.25 -> 4.7.28 update

Comment 8 Colin Walters 2021-09-02 13:03:18 UTC

I don't think is this exactly a bug in podman, unless we can pin it down as a regression.  And even if it was I think we'd need to ship some sort of workaround in OCP relatively quickly.
Moving back to RHCOS.

Comment 11 Colin Walters 2021-09-02 17:59:26 UTC

OK yeah trying with a random older `podman-1.9.3-2.rhaos4.6.el8.x86_64`:

```
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
* ostree://5d65bddfb072101a84501cd87b8abc650beb8dc0aa2bfeff022fc750cde52f1d
                   Version: 46.82.202009222340-0 (2020-09-22T23:44:32Z)
[root@cosa-devsh ~]# rpm -q podman
podman-1.9.3-2.rhaos4.6.el8.x86_64

[root@cosa-devsh ~]# podman create quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f6ab2e31c789c92aa6cb8ca1bad72c7eab7ef9e53fdd6126b02a1b8abe9b13
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f6ab2e31c789c92aa6cb8ca1bad72c7eab7ef9e53fdd6126b02a1b8abe9b13...
Getting image source signatures
Copying blob dfd8c625d022 done  
Copying blob b124782ed007 done  
Copying blob 6990ea5645c7 done  
Copying config 45be8c0eed done  
Writing manifest to image destination
Storing signatures
5bbd2c594459b541966e627876aa78b2e911c18f73834bf93919b2a1a0e05dfd
[root@cosa-devsh ~]# podman cp 5bbd2c594459b541966e627876aa78b2e911c18f73834bf93919b2a1a0e05dfd:/ /run/oscontent
[root@cosa-devsh ~]#
```

It looks like the old `podman cp` implementation defaulted to ignoring the inability to set xattrs, whereas with podman 3 it's a fatal error.

Comment 12 Colin Walters 2021-09-02 19:20:00 UTC

One thing I want to highlight here is https://github.com/coreos/coreos-assembler/issues/2401#issuecomment-911673692

A discovery I made here is that in the major extensions rework done in #2028 we stopped writing these user.* xattrs.

So the machine-os-content in the 4.8+ stream does not have this problem, but earlier versions i.e. ≤ 4.7 will.

Comment 13 Micah Abbott 2021-09-02 19:25:49 UTC

For paperwork purposes, we'll need to make multiple clones of this in order to track the fix in 4.7.z

This BZ (and the 4.8.z) clone will likely get closed as CURRENTRELEASE and we'll use the 4.7.z clone to accurately track the fix.

Comment 14 Micah Abbott 2021-09-02 19:31:20 UTC

See https://bugzilla.redhat.com/show_bug.cgi?id=2000746 for tracking the fix to 4.7.z