Hide Forgot
Description of problem: Pod gets "CrashLoopBackOff" status with failed sync of starting container Version-Release number of selected component (if applicable): oc v3.4.0.19+346a31d kubernetes v1.4.0+776c994 How reproducible: Always Steps to Reproduce: 1. Create a disk of ntfs format 2. Create a pod with azure disk volume of ntfs type apiVersion: v1 kind: Pod metadata: name: azntfs spec: containers: - image: aosqe/hello-openshift name: azure volumeMounts: - name: azure mountPath: /mnt/azure securityContext: runAsUser: 1000060001 fsGroup: 123456 seLinuxOptions: level: "s0:c8,c2" volumes: - name: azure azureDisk: diskName: ntfs.vhd diskURI: https://someaccount.blob.core.windows.net/vhds/ntfs.vhd fsType: ntfs-3g 3. Check pod status Actual results: Pods failed because of failed sync of SElinux reliabling: $ oc describe pods azntfs FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1h 1h 1 {default-scheduler } Normal ScheduledSuccessfully assigned azntfs to wehe-node1-1.eastus.cloudapp.azure.com 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} spec.containers{azure} Normal Created Created container with docker id 06a92a487439; Security:[seccomp=unconfined] 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} spec.containers{azure} Warning Failed Failed to start container with docker id 06a92a487439 with error: Error response from daemon: {"message":"SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/f15ea0b5-a19f-11e6-a4c3-000d3a12bded/volumes/kubernetes.io~azure-disk/azure is not allowed: \"operation not supported\""} 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} spec.containers{azure} Normal Created Created container with docker id f1f4a6572662; Security:[seccomp=unconfined] 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} spec.containers{azure} Warning Failed Failed to start container with docker id f1f4a6572662 with error: Error response from daemon: {"message":"SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/f15ea0b5-a19f-11e6-a4c3-000d3a12bded/volumes/kubernetes.io~azure-disk/azure is not allowed: \"operation not supported\""} 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} spec.containers{azure} Normal Created Created container with docker id 8c97202a7df7; Security:[seccomp=unconfined] 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} spec.containers{azure} Warning Failed Failed to start container with docker id 8c97202a7df7 with error: Error response from daemon: {"message":"SELinux relabeling of /var/lib/origin/openshift.local.volumes/pods/f15ea0b5-a19f-11e6-a4c3-000d3a12bded/volumes/kubernetes.io~azure-disk/azure is not allowed: \"operation not supported\""} 1h 1h 1 {kubelet wehe-node1-1.eastus.cloudapp.azure.com} Warning FailedSyncError syncing pod, skipping: failed Expected results: Pod can be running Additional info: Log from master: ov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518053 1053 replica_set.go:362] Pod azntfs updated, objectMeta {Name:azntfs GenerateName: Namespace:test SelfLink:/api/v1/namespaces/test/pods/azntfs UID:f15ea0b5-a19f-11e6-a4c3-000d3a12bded ResourceVersion:8152 Generation:0 CreationTimestamp:2016-11-03 16:31:40 +0800 CST DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[openshift.io/scc:privileged] OwnerReferences:[] Finalizers:[] ClusterName:} -> {Name:azntfs GenerateName: Namespace:test SelfLink:/api/v1/namespaces/test/pods/azntfs UID:f15ea0b5-a19f-11e6-a4c3-000d3a12bded ResourceVersion:8350 Generation:0 CreationTimestamp:2016-11-03 16:31:40 +0800 CST DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[openshift.io/scc:privileged] OwnerReferences:[] Finalizers:[] ClusterName:}. Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518134 1053 replica_set.go:238] No ReplicaSets found for pod azntfs, ReplicaSet controller will avoid syncing Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518159 1053 jobcontroller.go:166] No jobs found for pod azntfs, job controller will avoid syncing Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518172 1053 daemoncontroller.go:364] Pod azntfs updated. Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518205 1053 daemoncontroller.go:293] No daemon sets found for pod azntfs, daemon set controller will avoid syncing Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518215 1053 disruption.go:307] updatePod called on pod "azntfs" Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518223 1053 disruption.go:361] No PodDisruptionBudgets found for pod azntfs, PodDisruptionBudget controller will avoid syncing. Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.518227 1053 disruption.go:310] No matching pdb for pod "azntfs" Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.528034 1053 pet_set.go:238] No PetSets found for pod azntfs, PetSet controller will avoid syncing Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.528071 1053 replication_controller.go:379] Pod azntfs updated, objectMeta {Name:azntfs GenerateName: Namespace:test SelfLink:/api/v1/namespaces/test/pods/azntfs UID:f15ea0b5-a19f-11e6-a4c3-000d3a12bded ResourceVersion:8152 Generation:0 CreationTimestamp:2016-11-03 16:31:40 +0800 CST DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[openshift.io/scc:privileged] OwnerReferences:[] Finalizers:[] ClusterName:} -> {Name:azntfs GenerateName: Namespace:test SelfLink:/api/v1/namespaces/test/pods/azntfs UID:f15ea0b5-a19f-11e6-a4c3-000d3a12bded ResourceVersion:8350 Generation:0 CreationTimestamp:2016-11-03 16:31:40 +0800 CST DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[openshift.io/scc:privileged] OwnerReferences:[] Finalizers:[] ClusterName:}. Nov 03 17:24:52 wehe-master1.eastus.cloudapp.azure.com atomic-openshift-master[1053]: I1103 17:24:52.528136 1053 replication_controller.go:256] No controllers found for pod azntfs, replication manager will avoid syncing Log from node: Nov 03 16:57:58 wehe-node1-1.eastus.cloudapp.azure.com atomic-openshift-node[3677]: I1103 16:57:58.159684 3677 kubelet.go:2347] SyncLoop (SYNC): 1 pods; azntfs_test(f15ea0b5-a19f-11e6-a4c3-000d3a12bded) Nov 03 16:57:58 wehe-node1-1.eastus.cloudapp.azure.com atomic-openshift-node[3677]: I1103 16:57:58.159868 3677 kubelet.go:2761] Generating status for "azntfs_test(f15ea0b5-a19f-11e6-a4c3-000d3a12bded)" Nov 03 16:57:58 wehe-node1-1.eastus.cloudapp.azure.com atomic-openshift-node[3677]: I1103 16:57:58.160045 3677 status_manager.go:312] Ignoring same status for pod "azntfs_test(f15ea0b5-a19f-11e6-a4c3-000d3a12bded)", status: {Phase:Running Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-11-03 16:31:40 +0800 CST Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-11-03 16:31:40 +0800 CST Reason:ContainersNotReady Message:containers with unready status: [azure]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2016-11-03 16:31:40 +0800 CST Reason: Message:}] Message: Reason: HostIP:172.27.17.8 PodIP:10.1.0.11 StartTime:2016-11-03 16:31:40 +0800 CST InitContainerStatuses:[] ContainerStatuses:[{Name:azure State:{Waiting:0xc420fac9c0 Running:<nil> Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:0xc421297880} Ready:false RestartCount:9 Image:aosqe/hello-openshift ImageID:docker://sha256:caa46d03cf599cd2e98f40accd8256efa362e2212e70a903beb5b6380d2c461c ContainerID:docker://9ab72b537897b9f89871300a45705fb0b2c1850118d70d9a5adaa096f1bf4883}]} Nov 03 16:57:58 wehe-node1-1.eastus.cloudapp.azure.com atomic-openshift-node[3677]: I1103 16:57:58.160129 3677 volume_manager.go:324] Waiting for volumes to attach and mount for pod "azntfs_test(f15ea0b5-a19f-11e6-a4c3-000d3a12bded)" Nov 03 16:57:58 wehe-node1-1.eastus.cloudapp.azure.com atomic-openshift-node[3677]: I1103 16:57:58.180411 3677 atomic_writer.go:141] pod test/azntfs volume default-token-f3ey3: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/f15ea0b5-a19f-11e6-a4c3-000d3a12bded/volumes/kubernetes.io~secret/default-token-f3ey3