Created attachment 1735556 [details] mirror log Description of problem: oc adm release mirror' command on a Windows 10 machine fails with below error ''' error: unable to push quay.io/openshift-release-dev/ocp-v4.0-art-dev: failed to upload blob sha256:5eb0659f735f684a1127b091754f3b7c0067517cbbe33d91cf93ddbc826612b5: rename H:\mirror\v2\openshift\release\blobs\sha256:5eb0659f735f684a1127b091754f3b7c0067517cbbe33d91cf93ddbc826612b5.download H:\mirror\v2\openshift\release\blobs\sha256:5eb0659f735f684a1127b091754f3b7c0067517cbbe33d91cf93ddbc826612b5: The filename, directory name, or volume label syntax is incorrect. ''' tried H:/mirror and H:\mirror. I've added the --to-file file://openshift/release and tried leaving that out. Command H:\> .\oc.exe adm release mirror -a pull-secret.txt --from=quay.io/openshift-release-dev/ocp-release:4.6.5-x86_64 --to-dir=H:\mirror
Given the error message and how a mirror looks on RHEL I suspect the issue is related to invalid characters in the file names. Windows doesn't tolerate things like : or . in the file name. If you run the oc mirror command on Linux you get files like mirror/v2/openshift/release/blobs/sha256:03c950f086b7eea09a6e4fe6d820a4336a5fc78c75a612701c41a0c481b035d0 Unless the oc command is adjusting these file names for Windows, I think it will throw an error.
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.
Just tried this again a few days ago and it doesn't work
The LifecycleStale keyword was removed because the bug got commented on recently. The bug assignee was notified.
I'll be looking into this in the upcoming sprint, as stated above, oc is probably not handling file names for Windows correctly.
Any updates on this issue?
other bugs & features took priority, will be looking into this in the upcoming sprint.
other bugs & features took priority, will be looking into fixing oc handling of file names for Windows in the upcoming sprint.
@maciej I haven't had the chance to look into this Windows bug, I'm re-setting the assignee for this.
The ability to use oc mirror from a Windows machine is critical to my customer - please can this be looked at again.
@Anthony: Sure will syncup with the product engineering team and revert back to you on this bugzilla.
This is also impacting my ability to support customers evaluate and use OpenShift
Note that the issues with oc on Windows extend beyond just mirroring to disk. You litererally cannot use a windows machine as a bastion to mirror images at all. This bug covers mirroring to disk and how NTFS doesn't handle the file names. I worked with another customer that wanted to mirror images from the internet straigh through to their registry on their restricted network. That resulted in a partial mirror that didn't have all the images needed to install. As soon as we switched over to a Linux bastion the problem was resolved. TL;DR cannot use oc mirroring on windows at all.
@danclark Could you create a bug (or bugs) for the issues you're experiencing with mirroring on Windows? That would help bring awareness to the issues with oc on Windows and get them fixed. We want to bring oc on Windows up to parity with Linux, but we need detailed bug reports so we can schedule the work. Thanks.
@rpeoples This bug contains all the details that I have on mirroring to disk failing on Windows systems. The error appears to be related to file names on an NTFS system. I will create another bug for mirroring through a Windows system to another registry (Use case #2) I suspect the problems are related to each other but probably good to have separate bugs.
Hello, Is there any update on this? Apart from creating a different bug for #2, is there a initiative to use oc client on windows for mirroring images?
@suchaud I did not create the second BZ because I could not recreate the issue that I saw when helping a customer. That issue was a Windows bastion as a bridge mirroring from registry.redhat.io, through the windows bastion, to the endpoint registry running somewhere else. Since I couldn't recreate the issue I didn't file a bug. This bug is about being unable to mirror openshift images to disk on Windows, most likely because certain characters in the image names are not allowed on an NTFS partition. I personally have not seen many customers with this use case. I would suggest that if we do not intend to support it that we at the very least update the documentation to indicate the mirroring to disk is not supported on Windows machines and, if possible, update the oc command to error out when attempting. Another thing to note would be that while NTFS is most commonly used on Windows system, Linux systems can have NTFS partitions, and I assume that the error would occur there since it is the filesystem that does not allow the characters and not really the operating system itself.
Reproduced with oc 4.6.6 : $ ./oc.exe version --client Client Version: 4.6.6 $ ./Downloads/oc.exe adm release mirror -a pullsecret--Sm82grIvD0 --from=registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-06-183041 --to-dir=C:\mirror466 info: Mirroring 161 images to file://openshift/release ... <dir> openshift/release blobs: registry.ci.openshift.org/ocp/release sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3 1.747KiB ..... error: unable to copy layer sha256:d430a6a906ad518ed754ffe32553fa902be6ddd090b0130d821bb8caf73e4003 to file://openshift/release: rename C:mirror466\v2\openshift\release\blobs\sha256:d430a6a906ad518ed754ffe32553fa902be6ddd090b0130d821bb8caf73e4003.download C:mirror466\v2\openshift\release\blobs\sha256:d430a6a906ad518ed754ffe32553fa902be6ddd090b0130d821bb8caf73e4003: The parameter is incorrect. info: Mirroring completed in 1m10.21s (0B/s) error: one or more errors occurred while uploading images While 4.9.0 works well: $ ./oc.exe version --client Client Version: 4.9.0 $ ./oc.exe adm release mirror -a pullsecret--Sm82grIvD0 --from=registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-06-183041 --to-dir=C:\mirror info: Mirroring 161 images to file://openshift/release ... <dir> openshift/release blobs: registry.ci.openshift.org/ocp/release sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3 1.747KiB .... sha256:1f2bc989bc2b95c2d6890adb1c1bcbd12721875ce9f0dfac5c3af3c0fe08004d file://openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64-csi-driver-nfs sha256:aa9a530b15a68ccbd11cc579e3f25430e9833d6e58ba2e6e0b4335aa8a52024f file://openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64-cluster-kube-scheduler-operator info: Mirroring completed in 1m2.23s (203.4MB/s) Success Update image: openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64 To upload local images to a registry, run: oc image mirror --from-dir=C:mirror 'file://openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64*' REGISTRY/REPOSITORY Configmap signature file C:mirror\config\signature-sha256-702cce13aea51b75.yaml created
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.9.13 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0029