Bug 1468243

Summary: pods creation fails with kubernets v1.6.1+5115d708d7 GitCommit:fff65cf on ocp v3.6
Product: OpenShift Container Platform Reporter: Elvir Kuric <ekuric>
Component: NodeAssignee: Derek Carr <decarr>
Status: CLOSED WORKSFORME QA Contact: DeShuai Ma <dma>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: aos-bugs, ekuric, jokerman, mifiedle, mmccomas, shberry, sjenning
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: aos-scalability-36
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-06 20:49:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs none

Description Elvir Kuric 2017-07-06 12:49:16 UTC
Created attachment 1294930 [details]
logs

Description of problem:

After upgrading to latest OCP packages,creation of new pods fails 

Version-Release number of selected component (if applicable):

OCP packages :

atomic-openshift-node-3.6.135-1.git.0.56fd7dc.el7.x86_64
atomic-openshift-sdn-ovs-3.6.135-1.git.0.56fd7dc.el7.x86_64
atomic-openshift-clients-3.6.135-1.git.0.56fd7dc.el7.x86_64
atomic-openshift-master-3.6.135-1.git.0.56fd7dc.el7.x86_64
atomic-openshift-3.6.135-1.git.0.56fd7dc.el7.x86_64
tuned-profiles-atomic-openshift-node-3.6.135-1.git.0.56fd7dc.el7.x86_64

# kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1+5115d708d7", GitCommit:"fff65cf", GitTreeState:"clean", BuildDate:"2017-07-05T18:23:39Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1+5115d708d7", GitCommit:"fff65cf", GitTreeState:"clean", BuildDate:"2017-07-05T18:23:39Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}


How reproducible:

I believe packages are affected 


Steps to Reproduce:
1. update to above packages
2. try to create new pods 


Actual results:
pods creation will fail. In logs it will be visible as attached 

Expected results:

Check logs for an attempt to create pods 
Additional info:

Comment 1 Seth Jennings 2017-07-06 17:05:28 UTC
This a 3.5 -> 3.6 upgrade or a 3.6 -> newer 3.6 upgrade?

Comment 2 Derek Carr 2017-07-06 17:10:18 UTC
did you reconcile roles after upgrade?

Comment 3 Elvir Kuric 2017-07-06 17:22:12 UTC
(In reply to Derek Carr from comment #2)
> did you reconcile roles after upgrade?
what is process to do this?

(In reply to Seth Jennings from comment #1)
> This a 3.5 -> 3.6 upgrade or a 3.6 -> newer 3.6 upgrade?
this was upgrade from 3.5-> latest 3.6

Comment 4 Seth Jennings 2017-07-06 17:27:27 UTC
Process for reconciling roles after upgrade is here:
https://docs.openshift.org/latest/install_config/upgrading/manual_upgrades.html#updating-policy-definitions

Comment 5 Derek Carr 2017-07-06 17:39:06 UTC
to clarify the bug, the issue looks to be following:

1. user creates a daemonset
2. daemonset controller attempts to create the pod

Actual result:

daemonset is denied based on policy the ability to create a pod.

Jul  6 08:14:09 gprfs013 atomic-openshift-master: E0706 08:14:09.536877   63878 daemoncontroller.go:630] unable to create pods: User "system:serviceaccount:kube-system:daemon-set-controller" cannot create pods in project "cnscluster"

Comment 6 Mike Fiedler 2017-07-06 20:49:10 UTC
I logged in to the cluster and reconciled roles/rolebindings/sccs per comment 4.   Pods are creating successfully now.   Closing this bz.  

@ekuric Please re-open if you disagree.