Bug 1971745 - Windows pod with a Projected Volume is stuck at ContainerCreating
Summary: Windows pod with a Projected Volume is stuck at ContainerCreating
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Windows Containers
Version: 4.7
Hardware: x86_64
OS: Windows
Target Milestone: ---
: 4.9.0
Assignee: Aravindh Puthiyaparambil
QA Contact: gaoshang
Depends On:
Blocks: 1973580
TreeView+ depends on / blocked
Reported: 2021-06-14 17:17 UTC by Aravindh Puthiyaparambil
Modified: 2021-08-03 20:29 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Windows Pods with a Projected Volume and with a RunAsUser present in the Pod's SecurityContext results in os.Chown() being executed for entities being projected. Consequence: os.Chown() is not implemented for Windows and results in an error being thrown and the Pod is stuck in ContainerCreating. Fix: Don't execute os.Chown() if the kubelet is running on Windows Result: Pods go to running
Clone Of:
Last Closed: 2021-08-03 20:29:16 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 804 0 None open Bug 1971745: only chown if non-windows machine 2021-06-17 19:07:52 UTC
Github openshift windows-machine-config-operator pull 475 0 None open Bug 1971745: [submodule][kubelet] Update to eb86af7b4ac 2021-06-21 19:31:23 UTC
Red Hat Product Errata RHSA-2021:3001 0 None None None 2021-08-03 20:29:49 UTC

Description Aravindh Puthiyaparambil 2021-06-14 17:17:39 UTC
Description of problem:

When a Windows Pod is created with a Projected Volume, the pod is stuck in ContainerCreating

How reproducible: Always

Steps to Reproduce:
1. Create a Windows Pod with a Projected Volume

Actual results: Pod is stuck in ContainerCreating

Expected results: Pod should go to Running

Additional info:

The KEP [0] and implementation [1] for setting proper file permission for projected volumes did not take Windows into account. The introduction of os.Chown(), which is not supported in Windows, caused this issue to happen.

[0] https://github.com/kubernetes/enhancements/pull/1598/commits/73188e85c48dce815ddae6cb109ecf17fa72a687#
[1] https://github.com/kubernetes/kubernetes/commit/3db7275b549559696c42c0b5f51c9a2397e9571d#diff-f4218ebfe0ced84087844da9d87b126752414e347e727c0455687d69e8e3d912

Comment 1 Aravindh Puthiyaparambil 2021-06-14 17:18:51 UTC
This is a blocker bug for WMCO 3.0.0 release

Comment 3 Aravindh Puthiyaparambil 2021-06-15 17:15:59 UTC
Upstream issue: https://github.com/kubernetes/kubernetes/issues/102849
Upstream WIP PR: https://github.com/kubernetes/kubernetes/pull/102868

Comment 4 Aravindh Puthiyaparambil 2021-06-16 21:30:47 UTC
In Windows there does not seem to be a clear equivalent of chown. From my reading, the SetNamedSecurityInfoA call for a file needs a list of entries where you specify a user and permission like GENERIC_READ, GENERIC_WRITE. This almost seems like a chown+chomod. I am not able to figure out if there is a way to call SetNamedSecurityInfoA with a GRANT_ACCESS entry without specifying permissions.

Docs I have read so far: SetEntriesInAclW [0], SetNamedSecurityInfoW [1], ACCESS_MODE enumeration[2], Generic Access Rights[3].

I have reached out to MSFT folks on sig-windows on direction and waiting to hear back. The upstream PR has been updated to account for RunAsUsername being specified in a Windows Pod and I have added a comment to the spot where we have to do the Windows equivalent of chown().

[0] https://docs.microsoft.com/en-us/windows/win32/api/aclapi/nf-aclapi-setentriesinaclw
[1] https://docs.microsoft.com/en-us/windows/win32/api/aclapi/nf-aclapi-setnamedsecurityinfow
[2] https://docs.microsoft.com/en-us/windows/win32/api/accctrl/ne-accctrl-access_mode
[3] https://docs.microsoft.com/en-us/windows/win32/secauthz/generic-access-rights

Comment 5 gaoshang 2021-06-22 09:49:07 UTC
This bug has been verified on OCP 4.9.0-0.nightly-2021-06-21-191858 and passed, thanks.

Version-Release number of selected component (if applicable):
WMCO commit: 2043f8e518dd2f73498bde9f044ecda97f2f853e
OCP build: 4.9.0-0.nightly-2021-06-21-191858

1, Build WMCO from master branch and install the operator on OCP 4.9
2, Bootstrap Windows node
3, Create the Secrets for Projected Volume
oc new-project winc-test
echo -n "admin" > ./username.txt
echo -n "1f2d1e2e67df" > ./password.txt
oc create secret generic user --from-file=./username.txt
oc create secret generic pass --from-file=./password.txt

4, Create Windows Pod with Projected Volume
oc create -f https://raw.githubusercontent.com/sgaoshang/winc-test/master/data/WinWebServer_Projected_Volume.yaml

5, Check creating Windows pod with a Projected Volume succeed and the projected-volume directory contains your projected sources

# oc get pod -owide
NAME                            READY   STATUS    RESTARTS   AGE   IP           NODE                                         NOMINATED NODE   READINESS GATES
win-webserver-796c99c67-kgzs6   1/1     Running   0          21m   ip-10-0-147-116.us-east-2.compute.internal   <none>           <none>

# oc exec -it win-webserver-796c99c67-kgzs6 powershell
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved. 

PS C:\> ls .\projected-volume\ 

    Directory: C:\projected-volume

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        6/22/2021   9:25 AM                ..2021_06_22_09_25_06.072795943
d----l        6/22/2021   9:25 AM                ..data
-a---l        6/22/2021   9:25 AM              0 password.txt
-a---l        6/22/2021   9:25 AM              0 username.txt

PS C:\> cat .\projected-volume\username.txt 
PS C:\> cat .\projected-volume\password.txt 

Comment 8 errata-xmlrpc 2021-08-03 20:29:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Container Platform for Windows Containers 3.0.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.