Description of problem: atomic run projectatomic/helloapache fails because the host image /usr/bin/kubernetes is a symbolic link to /usr/bin/openshift. When the host filesystem is mounted on "/host" and manipulated by the atomicapp container, /host/usr/bin/kubectl is a broken symbolic link to /usr/bin/openshift. Symlinking openshift to kubectl using a relative path fixes this: # cd /usr/bin # ln -sf openshift kubectl Version-Release number of selected component (if applicable): 2.0 How reproducible: Always Steps to Reproduce: 1. In the RHEL7.2 libvirt vagrant box, run "atomic run projectatomic/helloapache" (as root) Actual results: Failure (error below) Expected results: A running pod Additional info: # atomic run projectatomic/helloapache ... 2015-11-11 14:08:00,531 - kubernetes - DEBUG - Given config: {u'image': u'centos/httpd', 'provider': 'kubernetes', 'namespace': 'default', u'hostport': 80} 2015-11-11 14:08:00,531 - kubernetes - INFO - Using namespace default 2015-11-11 14:08:00,531 - kubernetes - INFO - trying kubectl at /host/usr/bin/kubectl 2015-11-11 14:08:00,532 - kubernetes - INFO - trying kubectl at /host/usr/local/bin/kubectl 2015-11-11 14:08:00,533 - atomicapp.utils - INFO - atomicapp.status.error.message=No kubectl found in /usr/bin/kubectl:/usr/local/bin/kubectl 2015-11-11 14:08:00,533 - atomicapp.run - ERROR - No kubectl found in /usr/bin/kubectl:/usr/local/bin/kubectl Traceback (most recent call last): File "/usr/bin/atomicapp", line 9, in <module> load_entry_point('atomicapp==0.1.12', 'console_scripts', 'atomicapp')() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/cli/main.py", line 238, in main cli.run() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/cli/main.py", line 214, in run args.func(args) File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/cli/main.py", line 53, in cli_run if ae.run() is not None: File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/run.py", line 239, in run self._dispatchGraph() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/run.py", line 132, in _dispatchGraph self._processComponent(component, graph_item) File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/run.py", line 221, in _processComponent provider.init() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/providers/kubernetes.py", line 53, in init self.kubectl = self._find_kubectl(Utils.getRoot()) File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/providers/kubernetes.py", line 96, in _find_kubectl raise ProviderFailedException("No kubectl found in %s" % ":".join(test_paths)) atomicapp.plugin.ProviderFailedException: No kubectl found in /usr/bin/kubectl:/usr/local/bin/kubectl
If you have openshift then why not use openshift instead of kubernetes? Also I wonder if the symbolic link should be relative and not absolute. Is there a "rule of thumb" with regards to absolute vs relative symbolic links?
https://github.com/openshift/origin/pull/5919
(In reply to Dusty Mabe from comment #1) > If you have openshift then why not use openshift instead of kubernetes? In rhel7 if you check kubernetes client package then below is result so for rhel by default openshift binary available and all kubernetes cli client symlink to it. [root@localhost site-packages]# rpm -ql kubernetes-client /usr/bin/kubectl /usr/bin/openshift /usr/share/bash-completion/completions/kubectl /usr/share/doc/kubernetes-client-1.1.0 /usr/share/doc/kubernetes-client-1.1.0/HACKING.md /usr/share/doc/kubernetes-client-1.1.0/README.md /usr/share/doc/kubernetes-client-1.1.0/UPGRADE.md /usr/share/licenses/kubernetes-client-1.1.0 /usr/share/licenses/kubernetes-client-1.1.0/LICENSE /usr/share/man/man1/kubectl-annotate.1.gz /usr/share/man/man1/kubectl-api-versions.1.gz /usr/share/man/man1/kubectl-attach.1.gz /usr/share/man/man1/kubectl-cluster-info.1.gz /usr/share/man/man1/kubectl-config-set-cluster.1.gz /usr/share/man/man1/kubectl-config-set-context.1.gz /usr/share/man/man1/kubectl-config-set-credentials.1.gz /usr/share/man/man1/kubectl-config-set.1.gz /usr/share/man/man1/kubectl-config-unset.1.gz /usr/share/man/man1/kubectl-config-use-context.1.gz /usr/share/man/man1/kubectl-config-view.1.gz /usr/share/man/man1/kubectl-config.1.gz /usr/share/man/man1/kubectl-create.1.gz /usr/share/man/man1/kubectl-delete.1.gz /usr/share/man/man1/kubectl-describe.1.gz /usr/share/man/man1/kubectl-exec.1.gz /usr/share/man/man1/kubectl-expose.1.gz /usr/share/man/man1/kubectl-get.1.gz /usr/share/man/man1/kubectl-label.1.gz /usr/share/man/man1/kubectl-logs.1.gz /usr/share/man/man1/kubectl-namespace.1.gz /usr/share/man/man1/kubectl-patch.1.gz /usr/share/man/man1/kubectl-port-forward.1.gz /usr/share/man/man1/kubectl-proxy.1.gz /usr/share/man/man1/kubectl-replace.1.gz /usr/share/man/man1/kubectl-rolling-update.1.gz /usr/share/man/man1/kubectl-run.1.gz /usr/share/man/man1/kubectl-scale.1.gz /usr/share/man/man1/kubectl-stop.1.gz /usr/share/man/man1/kubectl-version.1.gz /usr/share/man/man1/kubectl.1.gz [root@localhost site-packages]# ls -l /usr/bin/ | grep kubectl lrwxrwxrwx. 1 root root 18 Nov 13 09:53 kubectl -> /usr/bin/openshift > > Also I wonder if the symbolic link should be relative and not absolute. Is > there a "rule of thumb" with regards to absolute vs relative symbolic links? This is as part of package itself. Now interesting bit: I run helloapache with "atomic" and with "atomicapp" and later one is worked but previous failed. ================ Atomicapp run ========================================= [vagrant@localhost atomicapp]$ atomicapp run projectatomic/helloapache 2015-11-17 08:29:51,136 - atomicapp.plugin - DEBUG - Loading providers from /usr/lib/python2.7/site-packages/atomicapp/providers 2015-11-17 08:29:51,146 - atomicapp.cli.main - ERROR - Exception caught: LockFailed('failed to create /run/lock/localhost.localdomain-5d0fd740.18787-533408688649599144',) 2015-11-17 08:29:51,146 - atomicapp.cli.main - ERROR - Run the command again with -v option to get more information. [vagrant@localhost atomicapp]$ sudo !! sudo atomicapp run projectatomic/helloapache 2015-11-17 08:30:05,847 - atomicapp.plugin - DEBUG - Loading providers from /usr/lib/python2.7/site-packages/atomicapp/providers 2015-11-17 08:30:05,855 - atomicapp.nulecule.base - INFO - Unpacking image: projectatomic/helloapache to /var/lib/atomicapp/projectatomic-helloapache-394a92efed01 2015-11-17 08:30:05,962 - atomicapp.nulecule.container - INFO - Skipping pulling Docker image: projectatomic/helloapache 2015-11-17 08:30:05,962 - atomicapp.nulecule.container - INFO - Extracting nulecule data from image: projectatomic/helloapache to /var/lib/atomicapp/projectatomic-helloapache-394a92efed01 46bdc383b5e990520fcb6b9b7165567b4f5826655301767ee6092797cfafe1b2 2015-11-17 08:30:10,049 - atomicapp.plugin - WARNING - Configuration option 'providerconfig' not found 2015-11-17 08:30:10,049 - atomicapp.plugin - WARNING - Configuration option 'providerconfig' not found 2015-11-17 08:30:10,049 - kubernetes - INFO - Using namespace default 2015-11-17 08:30:10,049 - kubernetes - INFO - trying kubectl at /usr/bin/kubectl 2015-11-17 08:30:10,050 - kubernetes - INFO - found kubectl at /usr/bin/kubectl 2015-11-17 08:30:10,050 - kubernetes - INFO - Deploying to Kubernetes Your application resides in /var/lib/atomicapp/projectatomic-helloapache-394a92efed01 Please use this directory for managing your application ====================Atomic Run================================== [vagrant@localhost ~]$ atomic run projectatomic/helloapache docker run -it --rm --privileged -v /home/vagrant:/atomicapp -v /run:/run -v /:/host --net=host --name helloapache -e NAME=helloapache -e IMAGE=projectatomic/helloapache projectatomic/helloapache -v run /atomicapp docker run -it --rm --privileged -v /home/vagrant:/atomicapp -v /run:/run -v /:/host --net=host --name helloapache -e NAME=helloapache -e IMAGE=projectatomic/helloapache projectatomic/helloapache -v run /atomicapp 2015-11-17 12:57:48,098 - atomicapp.install - INFO - App name is projectatomic/helloapache, will be populated to /atomicapp 2015-11-17 12:57:48,098 - atomicapp.utils - INFO - atomicapp.status.info.message=Loading app projectatomic/helloapache . 2015-11-17 12:57:48,099 - atomicapp.nulecule_base - DEBUG - Path to answers file given, loading /atomicapp/answers.conf 2015-11-17 12:57:48,101 - atomicapp.utils - INFO - atomicapp.status.answer.message={"helloapache-app": {"image": "centos/httpd", "hostport": 80}, "general": {"prefix": "/host", "namespace": "default", "provider": "kubernetes"}} 2015-11-17 12:57:48,101 - atomicapp.nulecule_base - DEBUG - {u'prefix': u'/host', u'namespace': u'default', u'provider': u'kubernetes'} 2015-11-17 12:57:48,159 - atomicapp.nulecule_base - DEBUG - Output of docker images cmd: 2015-11-17 12:57:48,159 - atomicapp.utils - INFO - atomicapp.status.info.message=Pulling image projectatomic/helloapache ... Trying to pull repository registry.access.redhat.com/projectatomic/helloapache ... not found Trying to pull repository docker.io/projectatomic/helloapache ... latest: Pulling from projectatomic/helloapache 47d44cb6f252: Already exists 168a69b62202: Already exists 812e9d9d677f: Already exists 4234bfdd88f8: Already exists ce20c473cd8a: Already exists aa955be47b8d: Already exists 66f399ae5470: Already exists cb750aae0088: Already exists 7fe96179c9ea: Already exists dc7b715c4ad5: Already exists 6196db07cd2c: Already exists ac5361a13aee: Already exists c1d6e688f65d: Already exists 8e705f3bc4bf: Already exists a6b6f2b8d30c: Already exists d63f82dbd649: Already exists 9b8c19f78cc2: Already exists 89a5824690c1: Already exists 76fd3a376b54: Already exists Digest: sha256:2b90e3433b6e8768bd12b6b8b788f860838508c012a16f63707b76114da0f329 Status: Image is up to date for docker.io/projectatomic/helloapache:latest 2015-11-17 12:57:58,271 - atomicapp.nulecule_base - DEBUG - {u'prefix': u'/host', u'namespace': u'default', u'provider': u'kubernetes'} 2015-11-17 12:57:58,272 - atomicapp.install - DEBUG - Creating a container with name helloapache-8f9b4045ff99 2015-11-17 12:57:58,272 - atomicapp.install - DEBUG - /usr/bin/docker run --name helloapache-8f9b4045ff99 --entrypoint /bin/true projectatomic/helloapache 2015-11-17 12:57:59,938 - atomicapp.utils - INFO - Using temporary directory /tmp/nulecule-egILSj 2015-11-17 12:57:59,938 - atomicapp.install - DEBUG - ['/usr/bin/docker', 'cp', 'helloapache-8f9b4045ff99:/application-entity', '/tmp/nulecule-egILSj'] 2015-11-17 12:58:00,525 - atomicapp.install - DEBUG - Application entity data copied to /tmp/nulecule-egILSj 2015-11-17 12:58:00,525 - atomicapp.utils - INFO - atomicapp.status.info.message=Copied app successfully. helloapache-8f9b4045ff99 2015-11-17 12:58:01,295 - atomicapp.utils - INFO - atomicapp.status.info.message=Loading app_id helloapache-app . 2015-11-17 12:58:01,303 - atomicapp.nulecule_base - DEBUG - Setting app id to helloapache-app 2015-11-17 12:58:01,303 - atomicapp.install - DEBUG - Nulecule path for pulled image: /tmp/nulecule-egILSj/application-entity/Nulecule 2015-11-17 12:58:01,304 - atomicapp.install - DEBUG - App ID: helloapache-app 2015-11-17 12:58:01,304 - atomicapp.nulecule_base - DEBUG - Version check successful: specversion == 0.0.2 2015-11-17 12:58:01,304 - atomicapp.utils - INFO - atomicapp.status.info.message=Checking all artifacts 2015-11-17 12:58:01,304 - atomicapp.nulecule_base - DEBUG - Provider: docker 2015-11-17 12:58:01,304 - atomicapp.utils - INFO - atomicapp.status.info.message=Artifact file://artifacts/docker/hello-apache-pod_run: OK. 2015-11-17 12:58:01,304 - atomicapp.nulecule_base - DEBUG - Provider: kubernetes 2015-11-17 12:58:01,304 - atomicapp.utils - INFO - atomicapp.status.info.message=Artifact file://artifacts/kubernetes/hello-apache-pod.json: OK. 2015-11-17 12:58:01,305 - atomicapp.utils - INFO - atomicapp.status.info.message=All artifacts OK. 2015-11-17 12:58:01,305 - atomicapp.nulecule_base - INFO - Artifacts for helloapache-app present for these providers: docker, kubernetes 2015-11-17 12:58:01,305 - atomicapp.utils - INFO - atomicapp.status.info.message=Loading Nulecule file. 2015-11-17 12:58:01,305 - atomicapp.install - INFO - Installing dependencies for helloapache-app 2015-11-17 12:58:01,305 - atomicapp.utils - DEBUG - {u'artifacts': {u'docker': [u'file://artifacts/docker/hello-apache-pod_run'], u'kubernetes': [u'file://artifacts/kubernetes/hello-apache-pod.json']}, u'params': [{u'default': u'centos/httpd', u'name': u'image', u'description': u'The webserver image'}, {u'default': 80, u'name': u'hostport', u'description': u'The host TCP port as the external endpoint'}], u'name': u'helloapache-app'} 2015-11-17 12:58:01,305 - atomicapp.install - DEBUG - Component helloapache-app is part of the app 2015-11-17 12:58:01,305 - atomicapp.install - DEBUG - Values: {u'helloapache-app': {u'image': u'centos/httpd', u'hostport': 80}} 2015-11-17 12:58:01,306 - atomicapp.utils - INFO - atomicapp.status.info.message=All dependencies installed successfully. 2015-11-17 12:58:01,306 - atomicapp.install - DEBUG - {u'helloapache-app': {u'image': u'centos/httpd', u'hostport': 80}} 2015-11-17 12:58:01,306 - atomicapp.nulecule_base - DEBUG - Data given {u'helloapache-app': {u'image': u'centos/httpd', u'hostport': 80}} 2015-11-17 12:58:01,306 - atomicapp.install - DEBUG - {u'helloapache-app': {u'image': u'centos/httpd', u'hostport': 80}, u'general': {u'prefix': u'/host', u'namespace': u'default', u'provider': u'kubernetes'}} 2015-11-17 12:58:01,307 - atomicapp.utils - INFO - atomicapp.status.answer.message={"helloapache-app": {"image": "centos/httpd", "hostport": 80}, "general": {"prefix": "/host", "namespace": "default", "provider": "kubernetes"}} 2015-11-17 12:58:01,307 - atomicapp.utils - INFO - atomicapp.status.info.message=Install Successful. 2015-11-17 12:58:01,307 - atomicapp.utils - INFO - atomicapp.status.info.message=Install Successful. 2015-11-17 12:58:01,307 - atomicapp.utils - DEBUG - Using working directory /atomicapp/.workdir 2015-11-17 12:58:01,307 - atomicapp.plugin - DEBUG - Loading providers from /usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/providers 2015-11-17 12:58:01,319 - atomicapp.nulecule_base - DEBUG - Setting app id to helloapache-app 2015-11-17 12:58:01,319 - atomicapp.nulecule_base - DEBUG - Version check successful: specversion == 0.0.2 2015-11-17 12:58:01,319 - atomicapp.nulecule_base - DEBUG - Path to answers file given, loading /atomicapp/answers.conf 2015-11-17 12:58:01,320 - atomicapp.nulecule_base - DEBUG - Provider: docker 2015-11-17 12:58:01,320 - atomicapp.utils - INFO - atomicapp.status.info.message=Artifact file://artifacts/docker/hello-apache-pod_run: OK. 2015-11-17 12:58:01,320 - atomicapp.nulecule_base - DEBUG - Provider: kubernetes 2015-11-17 12:58:01,321 - atomicapp.utils - INFO - atomicapp.status.info.message=Artifact file://artifacts/kubernetes/hello-apache-pod.json: OK. 2015-11-17 12:58:01,321 - atomicapp.utils - INFO - atomicapp.status.info.message=All artifacts OK. 2015-11-17 12:58:01,321 - atomicapp.nulecule_base - INFO - Artifacts for helloapache-app present for these providers: docker, kubernetes 2015-11-17 12:58:01,321 - atomicapp.utils - DEBUG - {u'artifacts': {u'docker': [u'file://artifacts/docker/hello-apache-pod_run'], u'kubernetes': [u'file://artifacts/kubernetes/hello-apache-pod.json']}, u'params': [{u'default': u'centos/httpd', u'name': u'image', u'description': u'The webserver image'}, {u'default': 80, u'name': u'hostport', u'description': u'The host TCP port as the external endpoint'}], u'name': u'helloapache-app'} 2015-11-17 12:58:01,321 - atomicapp.run - DEBUG - Processing component 'helloapache-app' and graph item '{u'artifacts': {u'docker': [u'file://artifacts/docker/hello-apache-pod_run'], u'kubernetes': [u'file://artifacts/kubernetes/hello-apache-pod.json']}, u'params': [{u'default': u'centos/httpd', u'name': u'image', u'description': u'The webserver image'}, {u'default': 80, u'name': u'hostport', u'description': u'The host TCP port as the external endpoint'}], u'name': u'helloapache-app'}' 2015-11-17 12:58:01,321 - atomicapp.plugin - DEBUG - Found provider <class 'kubernetes.KubernetesProvider'> 2015-11-17 12:58:01,321 - atomicapp.nulecule_base - DEBUG - Param prefix already in general with value /host 2015-11-17 12:58:01,321 - atomicapp.nulecule_base - DEBUG - Param provider already in general with value kubernetes 2015-11-17 12:58:01,322 - atomicapp.nulecule_base - DEBUG - Param namespace already in general with value default 2015-11-17 12:58:01,322 - atomicapp.plugin - WARNING - Configuration option 'providerconfig' not found 2015-11-17 12:58:01,322 - atomicapp.utils - INFO - atomicapp.status.info.message=Deploying component helloapache-app ... 2015-11-17 12:58:01,322 - atomicapp.run - INFO - Using provider kubernetes for component helloapache-app 2015-11-17 12:58:01,322 - atomicapp.run - DEBUG - Templating artifact /atomicapp/artifacts/kubernetes/hello-apache-pod.json 2015-11-17 12:58:01,323 - atomicapp.nulecule_base - DEBUG - Param prefix already in general with value /host 2015-11-17 12:58:01,323 - atomicapp.nulecule_base - DEBUG - Param provider already in general with value kubernetes 2015-11-17 12:58:01,323 - atomicapp.nulecule_base - DEBUG - Param namespace already in general with value default 2015-11-17 12:58:01,323 - atomicapp.run - DEBUG - Config: {u'prefix': u'/host', u'provider': u'kubernetes', u'namespace': u'default', u'image': u'centos/httpd', u'hostport': 80} 2015-11-17 12:58:01,323 - atomicapp.run - DEBUG - {u'prefix': u'/host', u'provider': u'kubernetes', u'namespace': u'default', u'image': u'centos/httpd', u'hostport': 80} 2015-11-17 12:58:01,323 - atomicapp.plugin - DEBUG - Writing artifact to /atomicapp/.workdir/helloapache-app/artifacts/kubernetes/hello-apache-pod.json 2015-11-17 12:58:01,324 - kubernetes - DEBUG - Given config: {u'prefix': u'/host', u'provider': u'kubernetes', u'namespace': u'default', u'image': u'centos/httpd', u'hostport': 80} 2015-11-17 12:58:01,324 - kubernetes - INFO - Using namespace default 2015-11-17 12:58:01,324 - kubernetes - INFO - trying kubectl at /host/usr/bin/kubectl 2015-11-17 12:58:01,324 - kubernetes - INFO - trying kubectl at /host/usr/local/bin/kubectl 2015-11-17 12:58:01,325 - atomicapp.utils - INFO - atomicapp.status.error.message=No kubectl found in /usr/bin/kubectl:/usr/local/bin/kubectl 2015-11-17 12:58:01,325 - atomicapp.run - ERROR - No kubectl found in /usr/bin/kubectl:/usr/local/bin/kubectl Traceback (most recent call last): File "/usr/bin/atomicapp", line 9, in <module> load_entry_point('atomicapp==0.1.12', 'console_scripts', 'atomicapp')() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/cli/main.py", line 238, in main cli.run() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/cli/main.py", line 214, in run args.func(args) File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/cli/main.py", line 53, in cli_run if ae.run() is not None: File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/run.py", line 239, in run self._dispatchGraph() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/run.py", line 132, in _dispatchGraph self._processComponent(component, graph_item) File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/run.py", line 221, in _processComponent provider.init() File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/providers/kubernetes.py", line 53, in init self.kubectl = self._find_kubectl(Utils.getRoot()) File "/usr/lib/python2.7/site-packages/atomicapp-0.1.12-py2.7.egg/atomicapp/providers/kubernetes.py", line 96, in _find_kubectl raise ProviderFailedException("No kubectl found in %s" % ":".join(test_paths)) atomicapp.plugin.ProviderFailedException: No kubectl found in /usr/bin/kubectl:/usr/local/bin/kubectl
It works when you run it with atomicapp directly because you aren't running in a container (i.e. you are running in the host), which means /usr/bin/kubectl is found and no symlink needs to be made. If the symlink were relative then everything should work. Colin opened an upstream issue on this: https://github.com/openshift/origin/pull/5919 Is it good enough for us to wait on that fix or do we need to workaround today?
Till the issue is not fixed in upstream i.e. https://github.com/openshift/origin/pull/5919 we can fix the Vagrantbox of CDK by doing this in the %post section of the kickstart file. rm /usr/bin/kubectl cd /usr/bin ln -s openshift kubectl
This is not an issue anymore in CDK beta 3 and future releases. The issue came as we built the CDK 2.0 beta2 from RHEL 2.2 bits which were unreleased. Now the build scripts of CDK 2.0 are fixed to build from released content of RHEL 7.