Bug 1903545 - Mirroring OpenShift images on Windows Fails
Summary: Mirroring OpenShift images on Windows Fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.6
Hardware: Unspecified
OS: Windows
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Ross Peoples
QA Contact: zhou ying
Olivia Payne
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-02 10:41 UTC by kedar
Modified: 2022-01-10 08:51 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Windows file paths don't allow the colon character Consequence: Panic when mirroring images on Windows machines Fix: Replace colon with dash on Windows machines Result: Mirroring on Windows machines no longer panics
Clone Of:
Environment:
Last Closed: 2022-01-10 08:50:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
mirror log (174.29 KB, text/plain)
2020-12-02 10:41 UTC, kedar
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 897 0 None None None 2021-08-10 19:33:03 UTC
Red Hat Product Errata RHBA-2022:0029 0 None None None 2022-01-10 08:51:15 UTC

Description kedar 2020-12-02 10:41:12 UTC
Created attachment 1735556 [details]
mirror log

Description of problem:


oc adm release mirror' command on a Windows 10 machine fails with below error 

'''
error: unable to push quay.io/openshift-release-dev/ocp-v4.0-art-dev: failed to upload blob sha256:5eb0659f735f684a1127b091754f3b7c0067517cbbe33d91cf93ddbc826612b5: rename H:\mirror\v2\openshift\release\blobs\sha256:5eb0659f735f684a1127b091754f3b7c0067517cbbe33d91cf93ddbc826612b5.download H:\mirror\v2\openshift\release\blobs\sha256:5eb0659f735f684a1127b091754f3b7c0067517cbbe33d91cf93ddbc826612b5: The filename, directory name, or volume label syntax is incorrect.
'''

tried H:/mirror and H:\mirror. I've added the --to-file file://openshift/release and tried leaving that out. 

Command

H:\> .\oc.exe adm release mirror -a pull-secret.txt --from=quay.io/openshift-release-dev/ocp-release:4.6.5-x86_64 --to-dir=H:\mirror

Comment 1 Dan Clark 2020-12-02 13:46:31 UTC
Given the error message and how a mirror looks on RHEL I suspect the issue is related to invalid characters in the file names. Windows doesn't tolerate things like : or . in the file name. If you run the oc mirror command on Linux you get files like 

mirror/v2/openshift/release/blobs/sha256:03c950f086b7eea09a6e4fe6d820a4336a5fc78c75a612701c41a0c481b035d0

Unless the oc command is adjusting these file names for Windows, I think it will throw an error.

Comment 2 Michal Fojtik 2021-01-01 13:58:21 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 3 Dan Clark 2021-01-01 20:16:33 UTC
Just tried this again a few days ago and it doesn't work

Comment 4 Michal Fojtik 2021-01-01 20:58:28 UTC
The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 5 Sally 2021-01-04 20:28:35 UTC
I'll be looking into this in the upcoming sprint, as stated above, oc is probably not handling file names for Windows correctly.

Comment 6 Michal Fojtik 2021-02-03 20:45:59 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 7 Dan Clark 2021-02-03 22:10:11 UTC
Any updates on this issue?

Comment 8 Michal Fojtik 2021-02-03 22:46:12 UTC
The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 9 Sally 2021-02-05 04:56:46 UTC
other bugs & features took priority, will be looking into this in the upcoming sprint.

Comment 10 Sally 2021-02-27 14:08:49 UTC
other bugs & features took priority, will be looking into fixing oc handling of file names for Windows in the upcoming sprint.

Comment 11 Sally 2021-03-21 23:59:26 UTC
@maciej I haven't had the chance to look into this Windows bug, I'm re-setting the assignee for this.

Comment 12 Anthony Kesterton 2021-08-04 22:34:37 UTC
The ability to use oc mirror from a Windows machine is critical to my customer - please can this be looked at again.

Comment 13 kedar 2021-08-06 16:58:13 UTC
@Anthony: Sure will syncup with the product engineering team and revert back to you on this bugzilla.

Comment 14 John Call 2021-09-15 02:02:49 UTC
This is also impacting my ability to support customers evaluate and use OpenShift

Comment 15 Dan Clark 2021-09-15 02:06:23 UTC
Note that the issues with oc on Windows extend beyond just mirroring to disk. You litererally cannot use a windows machine as a bastion to mirror images at all. This bug covers mirroring to disk and how NTFS doesn't handle the file names. I worked with another customer that wanted to mirror images from the internet straigh through to their registry on their restricted network. That resulted in a partial mirror that didn't have all the images needed to install. As soon as we switched over to a Linux bastion the problem was resolved.

TL;DR cannot use oc mirroring on windows at all.

Comment 16 Ross Peoples 2021-09-21 18:44:09 UTC
@danclark Could you create a bug (or bugs) for the issues you're experiencing with mirroring on Windows? That would help bring awareness to the issues with oc on Windows and get them fixed. We want to bring oc on Windows up to parity with Linux, but we need detailed bug reports so we can schedule the work. Thanks.

Comment 17 Dan Clark 2021-09-22 00:34:19 UTC
@rpeoples This bug contains all the details that I have on mirroring to disk failing on Windows systems. The error appears to be related to file names on an NTFS system. I will create another bug for mirroring through a Windows system to another registry (Use case #2) I suspect the problems are related to each other but probably good to have separate bugs.

Comment 18 Sudarshan Chaudhari 2021-10-12 17:36:53 UTC
Hello, 

Is there any update on this?

Apart from creating a different bug for #2, is there a initiative to use oc  client on windows for mirroring images?

Comment 19 Dan Clark 2021-10-12 21:25:57 UTC
@suchaud  I did not create the second BZ because I could not recreate the issue that I saw when helping a customer. That issue was a Windows bastion as a bridge mirroring from registry.redhat.io, through the windows bastion, to the endpoint registry running somewhere else. Since I couldn't recreate the issue I didn't file a bug.

This bug is about being unable to mirror openshift images to disk on Windows, most likely because certain characters in the image names are not allowed on an NTFS partition. I personally have not seen many customers with this use case. I would suggest that if we do not intend to support it that we at the very least update the documentation to indicate the mirroring to disk is not supported on Windows machines and, if possible, update the oc command to error out when attempting.

Another thing to note would be that while NTFS is most commonly used on Windows system, Linux systems can have NTFS partitions, and I assume that the error would occur there since it is the filesystem that does not allow the characters and not really the operating system itself.

Comment 22 zhou ying 2022-01-07 06:34:08 UTC
Reproduced with oc 4.6.6 :

$ ./oc.exe version --client
Client Version: 4.6.6

$ ./Downloads/oc.exe adm release mirror -a  pullsecret--Sm82grIvD0  --from=registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-06-183041  --to-dir=C:\mirror466
info: Mirroring 161 images to file://openshift/release ...
<dir>
  openshift/release
    blobs:
      registry.ci.openshift.org/ocp/release sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3 1.747KiB
.....

error: unable to copy layer sha256:d430a6a906ad518ed754ffe32553fa902be6ddd090b0130d821bb8caf73e4003 to file://openshift/release: rename C:mirror466\v2\openshift\release\blobs\sha256:d430a6a906ad518ed754ffe32553fa902be6ddd090b0130d821bb8caf73e4003.download C:mirror466\v2\openshift\release\blobs\sha256:d430a6a906ad518ed754ffe32553fa902be6ddd090b0130d821bb8caf73e4003: The parameter is incorrect.
info: Mirroring completed in 1m10.21s (0B/s)
error: one or more errors occurred while uploading images


While 4.9.0 works well:
$ ./oc.exe version --client
Client Version: 4.9.0

$ ./oc.exe adm release mirror -a pullsecret--Sm82grIvD0  --from=registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2022-01-06-183041  --to-dir=C:\mirror
info: Mirroring 161 images to file://openshift/release ...
<dir>
  openshift/release
    blobs:
      registry.ci.openshift.org/ocp/release sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3 1.747KiB
....
sha256:1f2bc989bc2b95c2d6890adb1c1bcbd12721875ce9f0dfac5c3af3c0fe08004d file://openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64-csi-driver-nfs
sha256:aa9a530b15a68ccbd11cc579e3f25430e9833d6e58ba2e6e0b4335aa8a52024f file://openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64-cluster-kube-scheduler-operator
info: Mirroring completed in 1m2.23s (203.4MB/s)

Success
Update image:  openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64

To upload local images to a registry, run:

    oc image mirror --from-dir=C:mirror 'file://openshift/release:4.10.0-0.nightly-2022-01-06-183041-x86_64*' REGISTRY/REPOSITORY

Configmap signature file C:mirror\config\signature-sha256-702cce13aea51b75.yaml created

Comment 25 errata-xmlrpc 2022-01-10 08:50:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.13 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0029


Note You need to log in before you can comment on or make changes to this bug.