Bug 1971745 - Windows pod with a Projected Volume is stuck at ContainerCreating
Summary: Windows pod with a Projected Volume is stuck at ContainerCreating
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Windows Containers
Version: 4.7
Hardware: x86_64
OS: Windows
urgent
urgent
Target Milestone: ---
: 4.9.0
Assignee: Aravindh Puthiyaparambil
QA Contact: gaoshang
URL:
Whiteboard:
Depends On:
Blocks: 1973580
TreeView+ depends on / blocked
 
Reported: 2021-06-14 17:17 UTC by Aravindh Puthiyaparambil
Modified: 2021-08-03 20:29 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Windows Pods with a Projected Volume and with a RunAsUser present in the Pod's SecurityContext results in os.Chown() being executed for entities being projected. Consequence: os.Chown() is not implemented for Windows and results in an error being thrown and the Pod is stuck in ContainerCreating. Fix: Don't execute os.Chown() if the kubelet is running on Windows Result: Pods go to running
Clone Of:
Environment:
Last Closed: 2021-08-03 20:29:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 804 0 None open Bug 1971745: only chown if non-windows machine 2021-06-17 19:07:52 UTC
Github openshift windows-machine-config-operator pull 475 0 None open Bug 1971745: [submodule][kubelet] Update to eb86af7b4ac 2021-06-21 19:31:23 UTC
Red Hat Product Errata RHSA-2021:3001 0 None None None 2021-08-03 20:29:49 UTC

Description Aravindh Puthiyaparambil 2021-06-14 17:17:39 UTC
Description of problem:

When a Windows Pod is created with a Projected Volume, the pod is stuck in ContainerCreating


How reproducible: Always


Steps to Reproduce:
1. Create a Windows Pod with a Projected Volume

Actual results: Pod is stuck in ContainerCreating

Expected results: Pod should go to Running

Additional info:

The KEP [0] and implementation [1] for setting proper file permission for projected volumes did not take Windows into account. The introduction of os.Chown(), which is not supported in Windows, caused this issue to happen.

[0] https://github.com/kubernetes/enhancements/pull/1598/commits/73188e85c48dce815ddae6cb109ecf17fa72a687#
[1] https://github.com/kubernetes/kubernetes/commit/3db7275b549559696c42c0b5f51c9a2397e9571d#diff-f4218ebfe0ced84087844da9d87b126752414e347e727c0455687d69e8e3d912

Comment 1 Aravindh Puthiyaparambil 2021-06-14 17:18:51 UTC
This is a blocker bug for WMCO 3.0.0 release

Comment 3 Aravindh Puthiyaparambil 2021-06-15 17:15:59 UTC
Upstream issue: https://github.com/kubernetes/kubernetes/issues/102849
Upstream WIP PR: https://github.com/kubernetes/kubernetes/pull/102868

Comment 4 Aravindh Puthiyaparambil 2021-06-16 21:30:47 UTC
In Windows there does not seem to be a clear equivalent of chown. From my reading, the SetNamedSecurityInfoA call for a file needs a list of entries where you specify a user and permission like GENERIC_READ, GENERIC_WRITE. This almost seems like a chown+chomod. I am not able to figure out if there is a way to call SetNamedSecurityInfoA with a GRANT_ACCESS entry without specifying permissions.

Docs I have read so far: SetEntriesInAclW [0], SetNamedSecurityInfoW [1], ACCESS_MODE enumeration[2], Generic Access Rights[3].

I have reached out to MSFT folks on sig-windows on direction and waiting to hear back. The upstream PR has been updated to account for RunAsUsername being specified in a Windows Pod and I have added a comment to the spot where we have to do the Windows equivalent of chown().

[0] https://docs.microsoft.com/en-us/windows/win32/api/aclapi/nf-aclapi-setentriesinaclw
[1] https://docs.microsoft.com/en-us/windows/win32/api/aclapi/nf-aclapi-setnamedsecurityinfow
[2] https://docs.microsoft.com/en-us/windows/win32/api/accctrl/ne-accctrl-access_mode
[3] https://docs.microsoft.com/en-us/windows/win32/secauthz/generic-access-rights

Comment 5 gaoshang 2021-06-22 09:49:07 UTC
This bug has been verified on OCP 4.9.0-0.nightly-2021-06-21-191858 and passed, thanks.

Version-Release number of selected component (if applicable):
WMCO commit: 2043f8e518dd2f73498bde9f044ecda97f2f853e
OCP build: 4.9.0-0.nightly-2021-06-21-191858

Steps:
1, Build WMCO from master branch and install the operator on OCP 4.9
2, Bootstrap Windows node
3, Create the Secrets for Projected Volume
oc new-project winc-test
echo -n "admin" > ./username.txt
echo -n "1f2d1e2e67df" > ./password.txt
oc create secret generic user --from-file=./username.txt
oc create secret generic pass --from-file=./password.txt

4, Create Windows Pod with Projected Volume
e.g.
oc create -f https://raw.githubusercontent.com/sgaoshang/winc-test/master/data/WinWebServer_Projected_Volume.yaml

5, Check creating Windows pod with a Projected Volume succeed and the projected-volume directory contains your projected sources

# oc get pod -owide
NAME                            READY   STATUS    RESTARTS   AGE   IP           NODE                                         NOMINATED NODE   READINESS GATES
win-webserver-796c99c67-kgzs6   1/1     Running   0          21m   10.132.0.2   ip-10-0-147-116.us-east-2.compute.internal   <none>           <none>

# oc exec -it win-webserver-796c99c67-kgzs6 powershell
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved. 

PS C:\> ls .\projected-volume\ 

    Directory: C:\projected-volume

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        6/22/2021   9:25 AM                ..2021_06_22_09_25_06.072795943
d----l        6/22/2021   9:25 AM                ..data
-a---l        6/22/2021   9:25 AM              0 password.txt
-a---l        6/22/2021   9:25 AM              0 username.txt


PS C:\> cat .\projected-volume\username.txt 
admin 
PS C:\> cat .\projected-volume\password.txt 
1f2d1e2e67df

Comment 8 errata-xmlrpc 2021-08-03 20:29:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Container Platform for Windows Containers 3.0.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3001


Note You need to log in before you can comment on or make changes to this bug.