Bug 1768858 - Failed to create pod on windows node when project is not "default"
Summary: Failed to create pod on windows node when project is not "default"
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Windows Containers
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Sebastian Soto
QA Contact: gaoshang
URL:
Whiteboard:
: 1785787 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-05 12:13 UTC by gaoshang
Modified: 2020-02-14 08:59 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-03 03:02:17 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description gaoshang 2019-11-05 12:13:04 UTC
Description of problem:

According to dev preview doc, create pod on windows node succeeded when oc project is "default", but failed to create pod in a new project, get error:

Warning  FailedCreatePodSandBox  2m10s (x13 over 2m23s)  kubelet, winworker-tobxu  Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-8648d6f7b8-cbvxl": Error response from daemon: security option not supported: label

Version-Release number of selected component (if applicable):
OCP version: 4.2 GA
# ./openshift-install version
./openshift-install v4.2.0
built from commit 90ccb37ac1f85ae811c50a29f9bb7e779c5045fb
release image quay.io/openshift-release-dev/ocp-release@sha256:c5337afd85b94c93ec513f21c8545e3f9e36a227f55d41bc1dfb8fcc3f2be129

windows-machine-config-operator version:
# git tag
0.1

windows instance:
MicrosoftWindowsServer:WindowsServer:2019-Datacenter-with-Containers:latest

How reproducible:
Always

Steps to Reproduce:
1. Install OCP 4.2 GA and bootstrap windows node on it
2. Create pod on windows node in a new project

# oc new-project prosgao

# wget https://gist.githubusercontent.com/suhanime/683ee7b5a2f55c11e3a26a4223170582/raw/86376218c26eadc0e709607b9a3354f275c52132/WinWebServer.yaml

# oc create -f WinWebServer.yaml

# oc get pod
NAME                             READY   STATUS              RESTARTS   AGE
win-webserver-8648d6f7b8-cbvxl   0/1     ContainerCreating   0          118s

# oc describe pod win-webserver-8648d6f7b8-cbvxl
Name:               win-webserver-8648d6f7b8-cbvxl
Namespace:          prosgao
Priority:           0
PriorityClassName:  <none>
Node:               winworker-tobxu/10.0.32.7
Start Time:         Tue, 05 Nov 2019 19:22:37 +0800
Labels:             app=win-webserver
                    pod-template-hash=8648d6f7b8
Annotations:        openshift.io/scc: restricted
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/win-webserver-8648d6f7b8
Containers:
  windowswebserver:
    Container ID:  
    Image:         mcr.microsoft.com/windows/servercore:ltsc2019
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      powershell.exe
      -command
      <#code used from https://gist.github.com/wagnerandrade/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ;  ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus)  } ; 
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-w5cd5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-w5cd5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-w5cd5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                     From                      Message
  ----     ------                  ----                    ----                      -------
  Normal   Scheduled               2m24s                   default-scheduler         Successfully assigned prosgao/win-webserver-8648d6f7b8-cbvxl to winworker-tobxu
  Warning  FailedCreatePodSandBox  2m10s (x13 over 2m23s)  kubelet, winworker-tobxu  Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-8648d6f7b8-cbvxl": Error response from daemon: security option not supported: label
  Normal   SandboxChanged          2m10s (x12 over 2m22s)  kubelet, winworker-tobxu  Pod sandbox changed, it will be killed and re-created.


Actual results:
Create pod should succeed

Expected results:
Create pod failed

Additional info:

Kubelet log in windows node:
PS C:\k> Get-Content .\kubelet.log -tail 1 -wait
...
I1105 12:09:03.249173    1888 volume_manager.go:350] Waiting for volumes to attach and mount for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:03.249173    1888 volume_manager.go:383] All volumes are attached and mounted for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:03.249173    1888 kuberuntime_manager.go:427] No ready sandbox for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)" can be found. Need to start a new one
I1105 12:09:03.249173    1888 kuberuntime_manager.go:617] computePodActions got {KillPod:true CreateSandbox:true SandboxID:9f43b8901758f221564b0653005a7c9b0d4e6e365a9722266ac15a168c362d96 Attempt:6 NextInitContainerToStart:nil ContainersToStart:[0] ContainersToKill:map[]} for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:03.249173    1888 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"prosgao", Name:"win-webserver-8648d6f7b8-8wx7p", UID:"0503f429-ffc5-11e9-b09e-000d3a8a4ef0", APIVersion:"v1", ResourceVersion:"423503", FieldPath:""}): type: 'Normal' reason: 'SandboxChanged' Pod sandbox changed, it will be killed and re-created.
I1105 12:09:03.268189    1888 docker_service.go:474] Setting cgroup parent to: ""
E1105 12:09:03.312201    1888 azure_dd.go:172] failed to list vm sizes in GetVolumeLimits, plugin.host: winnode, location: eastus
I1105 12:09:03.446943    1888 secret.go:186] Setting up volume default-token-w5cd5 for pod 0503f429-ffc5-11e9-b09e-000d3a8a4ef0 at c:\var\lib\kubelet\pods\0503f429-ffc5-11e9-b09e-000d3a8a4ef0\volumes\kubernetes.io~secret\default-token-w5cd5
I1105 12:09:03.446943    1888 secret.go:210] Received secret prosgao/default-token-w5cd5 containing (4) pieces of data, 11233 total bytes
I1105 12:09:03.448378    1888 empty_dir.go:260] pod 0503f429-ffc5-11e9-b09e-000d3a8a4ef0: mounting tmpfs for volume wrapped_default-token-w5cd5
I1105 12:09:03.448378    1888 mount_windows.go:61] mounting source ("tmpfs"), target ("c:\\var\\lib\\kubelet\\pods\\0503f429-ffc5-11e9-b09e-000d3a8a4ef0\\volumes\\kubernetes.io~secret\\default-token-w5cd5"), with options ([])
I1105 12:09:03.863563    1888 eviction_manager.go:230] eviction manager: synchronize housekeeping
I1105 12:09:03.932526    1888 helpers.go:822] eviction manager: observations: signal=memory.available, available: 7634224Ki, capacity: 8388148Ki, time: 2019-11-05 12:09:03.2271795 +0000 GMT m=+531.597566101
I1105 12:09:03.932526    1888 helpers.go:822] eviction manager: observations: signal=nodefs.available, available: 14512340Ki, capacity: 30943228Ki, time: 2019-11-05 12:09:03.2271795 +0000 GMT m=+531.597566101
I1105 12:09:03.932526    1888 helpers.go:822] eviction manager: observations: signal=imagefs.available, available: 14512340Ki, capacity: 30943228Ki, time: 2019-11-05 12:09:03.9195113 +0000 GMT
I1105 12:09:03.932526    1888 eviction_manager.go:321] eviction manager: no resources are starved
E1105 12:09:04.908838    1888 remote_runtime.go:109] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-8648d6f7b8-8wx7p": Error response from daemon: security option not supported: label
E1105 12:09:04.908838    1888 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)" failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-8648d6f7b8-8wx7p": Error response from daemon: security option not supported: label
E1105 12:09:04.908838    1888 kuberuntime_manager.go:693] createPodSandbox for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)" failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-8648d6f7b8-8wx7p": Error response from daemon: security option not supported: label
E1105 12:09:04.908838    1888 pod_workers.go:190] Error syncing pod 0503f429-ffc5-11e9-b09e-000d3a8a4ef0 ("win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"), skipping: failed to "CreatePodSandbox" for "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)" with CreatePodSandboxError: "CreatePodSandbox for pod \"win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)\" failed: rpc error: code = Unknown desc = failed to start sandbox container for pod \"win-webserver-8648d6f7b8-8wx7p\": Error response from daemon: security option not supported: label"
I1105 12:09:04.909828    1888 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"prosgao", Name:"win-webserver-8648d6f7b8-8wx7p", UID:"0503f429-ffc5-11e9-b09e-000d3a8a4ef0", APIVersion:"v1", ResourceVersion:"423503", FieldPath:""}): type: 'Warning' reason: 'FailedCreatePodSandBox' Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "win-webserver-8648d6f7b8-8wx7p": Error response from daemon: security option not supported: label
I1105 12:09:04.920840    1888 kubelet.go:1930] SyncLoop (PLEG): "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)", event: &pleg.PodLifecycleEvent{ID:"0503f429-ffc5-11e9-b09e-000d3a8a4ef0", Type:"ContainerDied", Data:"c860f81272a9c2da14802c973a7c2eb3f5326f3811e6ccfcea63ba123d35aa32"}
I1105 12:09:04.920840    1888 kubelet_pods.go:1334] Generating status for "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
W1105 12:09:04.920840    1888 pod_container_deletor.go:75] Container "c860f81272a9c2da14802c973a7c2eb3f5326f3811e6ccfcea63ba123d35aa32" not found in pod's containers
I1105 12:09:05.934194    1888 kubelet_pods.go:1334] Generating status for "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:05.934194    1888 status_manager.go:364] Ignoring same status for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)", status: {Phase:Pending Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-05 12:08:46 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-05 12:08:46 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [windowswebserver]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-05 12:08:46 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [windowswebserver]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-11-05 12:08:46 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.32.8 PodIP: StartTime:2019-11-05 12:08:46 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:windowswebserver State:{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:mcr.microsoft.com/windows/servercore:ltsc2019 ImageID: ContainerID:}] QOSClass:BestEffort}
I1105 12:09:05.934194    1888 volume_manager.go:350] Waiting for volumes to attach and mount for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:05.934194    1888 volume_manager.go:383] All volumes are attached and mounted for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:05.934194    1888 kuberuntime_manager.go:427] No ready sandbox for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)" can be found. Need to start a new one
I1105 12:09:05.934194    1888 kuberuntime_manager.go:617] computePodActions got {KillPod:true CreateSandbox:true SandboxID:c860f81272a9c2da14802c973a7c2eb3f5326f3811e6ccfcea63ba123d35aa32 Attempt:7 NextInitContainerToStart:nil ContainersToStart:[0] ContainersToKill:map[]} for pod "win-webserver-8648d6f7b8-8wx7p_prosgao(0503f429-ffc5-11e9-b09e-000d3a8a4ef0)"
I1105 12:09:05.935202    1888 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"prosgao", Name:"win-webserver-8648d6f7b8-8wx7p", UID:"0503f429-ffc5-11e9-b09e-000d3a8a4ef0", APIVersion:"v1", ResourceVersion:"423503", FieldPath:""}): type: 'Normal' reason: 'SandboxChanged' Pod sandbox changed, it will be killed and re-created.
I1105 12:09:05.949214    1888 docker_service.go:474] Setting cgroup parent to: ""
I1105 12:09:06.110562    1888 secret.go:186] Setting up volume default-token-w5cd5 for pod 0503f429-ffc5-11e9-b09e-000d3a8a4ef0 at c:\var\lib\kubelet\pods\0503f429-ffc5-11e9-b09e-000d3a8a4ef0\volumes\kubernetes.io~secret\default-token-w5cd5
I1105 12:09:06.110562    1888 secret.go:210] Received secret prosgao/default-token-w5cd5 containing (4) pieces of data, 11233 total bytes
I1105 12:09:06.110562    1888 empty_dir.go:260] pod 0503f429-ffc5-11e9-b09e-000d3a8a4ef0: mounting tmpfs for volume wrapped_default-token-w5cd5
I1105 12:09:06.110562    1888 mount_windows.go:61] mounting source ("tmpfs"), target ("c:\\var\\lib\\kubelet\\pods\\0503f429-ffc5-11e9-b09e-000d3a8a4ef0\\volumes\\kubernetes.io~secret\\default-token-w5cd5"), with options ([])

Comment 1 sumehta 2019-11-12 15:26:06 UTC
@gaoshang Can you try pulling the mcr.microsoft.com/windows/servercore:ltsc2019 image on the instance first before the deployment?
Instructions present here : https://docs.google.com/document/d/1zAidTs8wbWHzamh4G3pwdaPomyT-doSbo9xTAcIK9no/edit#heading=h.9sbyxso0hjcb
This could be because of the timing out of kubelet run, given the size of docker image for windows

Comment 2 gaoshang 2019-11-13 16:03:36 UTC
(In reply to sumehta from comment #1)
> @gaoshang Can you try pulling the
> mcr.microsoft.com/windows/servercore:ltsc2019 image on the instance first
> before the deployment?
> Instructions present here :
> https://docs.google.com/document/d/1zAidTs8wbWHzamh4G3pwdaPomyT-
> doSbo9xTAcIK9no/edit#heading=h.9sbyxso0hjcb
> This could be because of the timing out of kubelet run, given the size of
> docker image for windows

After pulling mcr.microsoft.com/windows/servercore:ltsc2019 image on the instance first, pod still can not be created in a new project, found replicaset error:
"Error creating: pods "win-webserver-8648d6f7b8-" is forbidden: unable to validate against any security context constraint"

# oc get pod
No resources found.

# oc get all
NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/win-webserver   LoadBalancer   172.30.46.81   <pending>     80:30685/TCP   41m

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/win-webserver   0/1     0            0           41m

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/win-webserver-8648d6f7b8   1         0         0       7m12s

# oc describe replicaset.apps/win-webserver-8648d6f7b8
Name:           win-webserver-8648d6f7b8
Namespace:      prosgao
Selector:       app=win-webserver,pod-template-hash=8648d6f7b8
Labels:         app=win-webserver
                pod-template-hash=8648d6f7b8
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/win-webserver
Replicas:       0 current / 1 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=win-webserver
           pod-template-hash=8648d6f7b8
  Containers:
   windowswebserver:
    Image:      mcr.microsoft.com/windows/servercore:ltsc2019
    Port:       <none>
    Host Port:  <none>
    Command:
      powershell.exe
      -command
      <#code used from https://gist.github.com/wagnerandrade/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ;  ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus)  } ; 
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  5m35s (x21 over 27m)  replicaset-controller  Error creating: pods "win-webserver-8648d6f7b8-" is forbidden: unable to validate against any security context constraint: []

Comment 3 gaoshang 2019-11-26 11:03:05 UTC
Update: This bug also exist in OCP 4.3.0-0.nightly-2019-11-24-183610, I think it's a SCC related issue and can be workaround by following steps:

Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2019-11-24-183610   True        False         5m35s   Cluster version is 4.3.0-0.nightly-2019-11-24-183610
windows-machine-config-operator commit:
# git show
commit 1eb1f983774101b5077828fd2efb4dfb711d5886

1. Install OCP 4.3 and scale up a windows node.

2. Create a new project, edit restricted scc, change following 2 section
from:
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs

to:
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny

# oc new-project prosgao
# oc edit scc restricted
# oc replace -f /tmp/oc-edit-hlxvr.yaml

3. Now windows pod can be created
# oc create -f https://gist.githubusercontent.com/suhanime/683ee7b5a2f55c11e3a26a4223170582/raw/86376218c26eadc0e709607b9a3354f275c52132/WinWebServer.yaml
# oc get pod
NAME                             READY   STATUS    RESTARTS   AGE
win-webserver-79b64df8b9-5cgk6   1/1     Running   0          52s

Comment 4 gaoshang 2019-12-24 07:04:44 UTC
Update:
Creating window pod will fail with restricted security context constraints, another workaround is to use privileged scc

Steps:
1, create new project and add privileged scc to user
# oc new-project winc
# oc adm policy add-scc-to-user privileged system:serviceaccount:winc:default
# oc get scc privileged -o yaml | grep users -A 5
users:
- system:admin
- system:serviceaccount:openshift-infra:build-controller
- system:serviceaccount:winc:default

2, in deployment WinWebServer.yaml, add privileged securityContext
# cat WinWebServer.yaml | grep containers: -A 6
      containers:
      - name: windowswebserver
        image: mcr.microsoft.com/windows/servercore:ltsc2019
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true        
        command:
3, create deployment, windows pod works
# oc create -f WinWebServer.yaml
# oc get pods
NAME                            READY   STATUS    RESTARTS   AGE
win-webserver-7fd94cd8f-pzt2c   1/1     Running   0          5m22s

Comment 5 Sebastian Soto 2020-01-20 19:45:23 UTC
This looks to be hitting this error https://github.com/docker/docker-ce/blob/58a1084222834a52f8e20e9641aa5b5fb927bef0/components/engine/daemon/oci_windows.go#L321

I've noticed that spinning up a pod in a namespace other than default causes these security options to be added to the pod container spec by default:
```
    securityContext:
      capabilities:
        drop:
        - KILL
        - MKNOD
        - SETGID
        - SETUID
      runAsUser: 1000560000
```
and these to the pod spec:
```
  securityContext:
    fsGroup: 1000560000
    seLinuxOptions:
      level: s0:c24,c4

```

Comment 6 Sebastian Soto 2020-01-20 20:37:56 UTC
The hostconfig (C:\ProgramData\docker\containers\<container_id>\hostconfig)for a pod hitting this error contains:
```
  "SecurityOpt": [
    "label=level:s0:c24,c4"
  ],
```
Which is an invalid value, the only valid key is "credentialspec"

Comment 7 Sebastian Soto 2020-01-21 18:23:01 UTC
The SELinux options are coming from the SCC attached to the project/namespace https://docs.openshift.com/container-platform/4.2/authentication/managing-security-context-constraints.html

Comment 8 Aravindh Puthiyaparambil 2020-01-21 19:28:17 UTC
*** Bug 1785787 has been marked as a duplicate of this bug. ***

Comment 9 Sebastian Soto 2020-01-30 14:53:44 UTC
This bug can be worked around by disabling SCC in specific namespaces. This should not be used in production, and in general any namespace that this has been done to should not be used to run linux pods.
To skip SCC for a namespace the label "openshift.io/run-level = 1"  should be applied to the namespace. This will apply to both linux and windows pods, and thus linux pods should not be deployed into this namespace.
This information will be added to the development preview doc.

Long term, we may add a webhook that will mutate Windows pods to remove non-windows options from the pod. This will make this a non-issue and remove the need to do the above workaround.

Comment 10 Sebastian Soto 2020-01-30 15:41:37 UTC
Followup work will be tracked in https://issues.redhat.com/browse/WINC-213

Comment 11 Aravindh Puthiyaparambil 2020-01-30 16:22:21 UTC
@gaoshang please close this bug given we have a workaround. For GA in the operator time frame, the feature we are adding will overcome this problem.

Comment 12 gaoshang 2020-02-03 03:02:17 UTC
Sure, closed this bug and will follow up WINC-213, thanks.


Note You need to log in before you can comment on or make changes to this bug.