Description of problem: Check the logs for apiman/apiman-gateway pods, will meet below errors: Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused Version-Release number of selected component (if applicable): openshift v1.1.5-20-ga343783 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 fabric8/apiman:2.2.94 fabric8/apiman-gateway:2.2.94 How reproducible: always Steps to Reproduce: 1. Start OpenShift server 2. Deploy the EFK stack 3. Create apiman and apiman-gateway pods oc new-app --docker-image=fabric8/apiman:2.2.94 oc new-app --docker-image=fabric8/apiman-gateway:2.2.94 4. Check pods state oc get pod 5. Check logs for apiman and apiman-gateway pods oc logs apiman-1-jfemw oc logs apiman-gateway-1-y1uo8 Actual results: Step 4: NAME READY STATUS RESTARTS AGE apiman-1-jfemw 1/1 Running 0 27m apiman-gateway-1-y1uo8 1/1 Running 0 27m logging-curator-1-efy1s 1/1 Running 0 38m logging-deployer-tedu7 0/1 Completed 0 39m logging-es-v33gqeru-1-ckwqq 1/1 Running 0 38m logging-fluentd-l7s7b 1/1 Running 0 29m logging-kibana-1-umnud 2/2 Running 0 38m Step 5; please refer to log files in attachment Expected results: Should not such errors in apiman/apiman-gateway pods Additional info:
Created attachment 1141658 [details] apiman-gateway logs
Created attachment 1141659 [details] apiman pod logs
@chunchen From what effort does this originate? Are we testing apiman functionality related to a trello card? AFAIK there is no apiman functionality ready. Additionally, the current proposal would make the need to deploy logging and apiman unnecessary
@Jeff Cantrill Yes, the bug is related to trello cards(intservice_publice_213 and intservice_publice_216), sorry for my mistake. I am re-testing with fabric8/elasticsearch-k8s:2.2.1 image to deploy elasticsearch.
I met below errors in elasticsearch-k8s pod: $ oc new-app --docker-image=fabric8/elasticsearch-k8s:2.2.1 $ oc logs elasticsearch-k8s-1-dzjs4 Exception in thread "main" java.lang.IllegalArgumentException: Could not resolve placeholder 'SERVICE' at org.elasticsearch.common.property.PropertyPlaceholder.parseStringValue(PropertyPlaceholder.java:128) at org.elasticsearch.common.property.PropertyPlaceholder.replacePlaceholders(PropertyPlaceholder.java:81) at org.elasticsearch.common.settings.Settings$Builder.replacePropertyPlaceholders(Settings.java:1173) at org.elasticsearch.node.internal.InternalSettingsPreparer.initializeSettings(InternalSettingsPreparer.java:131) at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100) at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107) at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100) at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) Refer to the log for complete error details. Dose it mean that I only can run related images via "docker run" directly to test cards for now?
@chunchen can you identify which trello card this is testing.
You also need to start an elastic search instance: oc new-app --docker-image=fabric8/elasticsearch-v1:2.2.94
@Jeff Cantrill https://trello.com/c/JA4c13p9/213-2-apiman-gateway-configurable-indexes-apiman https://trello.com/c/H254T272/216-2-apiman-organization-to-namespace-apiman
(In reply to Kurt T Stam from comment #7) > You also need to start an elastic search instance: > > oc new-app --docker-image=fabric8/elasticsearch-v1:2.2.94 This image is not existing, I tried below testing: [chunchen@F17-CCY scripts]$ oc new-app --docker-image=fabric8/elasticsearch-v1:2.2.94 error: no match for "fabric8/elasticsearch-v1:2.2.94" The 'oc new-app' command will match arguments to the following types: 1. Images tagged into image streams in the current project or the 'openshift' project - if you don't specify a tag, we'll add ':latest' 2. Images in the Docker Hub, on remote registries, or on the local Docker engine 3. Templates in the current project or the 'openshift' project 4. Git repository URLs or local paths that point to Git repositories --allow-missing-images can be used to point to an image that does not exist yet. See 'oc new-app -h' for examples. [root@ip-172-18-10-41 ~]# docker search fabric8/elasticsearch-v1 INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED [root@ip-172-18-10-41 ~]# docker search fabric8/elasticsearch INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED docker.io docker.io/fabric8/elasticsearch-k8s a kubernetes aware Elasticsearch image 1 docker.io docker.io/fabric8/elasticsearch 0 docker.io docker.io/fabric8/elasticsearch-logstash-template 0
@Kurt T Stam Please ignore Comment #9, I have created elasticsearch-v1 pod by downloading file [1]. [chunchen@F17-CCY Downloads]$ oc get pod NAME READY STATUS RESTARTS AGE apiman-gateway-jecxz 1/1 Running 0 12m apiman-onydw 1/1 Running 0 12m elasticsearch-v1-th21w 2/2 Running 0 12m fabric8-ija7n 1/1 Running 0 53m [1] http://repo1.maven.org/maven2/io/fabric8/ipaas/distro/distro/2.2.94/distro-2.2.94-templates.zip
Could you help to change this bug to ON_QA, I have deployed APIMan successfully with the new steps: http://fpaste.org/348144/95033291/ ? so, I can mark it as verified. Thanks!
Oh,sorry, still find similar errors in apiman pod, please refer to the attachment named "apiman-new-logs", others info as below: 1: [chunchen@F17-CCY daily]$ oc get pod oc get route NAME READY STATUS RESTARTS AGE apiman-gateway-3w4e6 1/1 Running 0 2h apiman-rbico 1/1 Running 0 2h elasticsearch-v1-dnbo0 2/2 Running 1 2h fabric8-wcujt 1/1 Running 0 2h [chunchen@F17-CCY daily]$ oc get route NAME HOST/PORT PATH SERVICE TERMINATION LABELS apiman apiman-chun1.router.default.svc.cluster.local apiman apiman-gateway apiman-gateway-chun1.router.default.svc.cluster.local apiman-gateway elasticsearch-v1 elasticsearch-v1-chun1.router.default.svc.cluster.local elasticsearch-v1 elasticsearch-v1-masters elasticsearch-v1-masters-chun1.router.default.svc.cluster.local elasticsearch-v1-masters fabric8 fabric8.ec2-52-90-106-124.compute-1.amazonaws.com fabric8 2: [chunchen@F17-CCY scripts]$ oc rsh elasticsearch-v1-dnbo0 # curl localhost:9200 { "status" : 200, "name" : "Blind Faith", "cluster_name" : "elasticsearch", "version" : { "number" : "1.6.0", "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", "build_timestamp" : "2015-06-09T13:36:34Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" } # # curl 127.0.0.1:9200 { "status" : 200, "name" : "Blind Faith", "cluster_name" : "elasticsearch", "version" : { "number" : "1.6.0", "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", "build_timestamp" : "2015-06-09T13:36:34Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }
Created attachment 1142515 [details] apiman-new-logs
Can you make sure the elesticsearch cluster is up before starting apiman? From the logs I can tell it found the right elasticsearch cluster to connect to. Just kill the apiman pods and see if the error clears.
When I tried to access the elesticsearch cluster via browser, I met 503 page, please refer to the sceenshot named "es-cluster-503-page" in attachment: The latest apiman templates downloaded from http://repo1.maven.org/maven2/io/fabric8/ipaas/distro/distro/2.2.103/distro-2.2.103-templates.zip [chunchen@F17-CCY main]$ oc process -f apiman-2.2.103.json | oc create -f - service "apiman" created service "apiman-gateway" created service "elasticsearch-v1" created service "elasticsearch-v1-masters" created replicationcontroller "apiman" created replicationcontroller "apiman-gateway" created replicationcontroller "elasticsearch-v1" created [chunchen@F17-CCY daily]$ oc scale rc/elasticsearch-v1 --replicas=2 [chunchen@F17-CCY daily]$ oc get pod NAME READY STATUS RESTARTS AGE apiman-bwx12 1/1 Running 0 26m apiman-gateway-c07nr 1/1 Running 0 26m elasticsearch-v1-u8cmk 2/2 Running 0 23m elasticsearch-v1-y3kkk 2/2 Running 0 26m fabric8-wptm6 1/1 Running 0 33m [chunchen@F17-CCY daily]$ oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE apiman 172.30.76.194 <none> 7070/TCP 26m apiman-gateway 172.30.63.213 <none> 7777/TCP 26m elasticsearch-v1 172.30.249.174 9200/TCP 26m elasticsearch-v1-masters None 9300/TCP 26m fabric8 172.30.30.104 80/TCP 33m [chunchen@F17-CCY daily]$ oc get route NAME HOST/PORT PATH SERVICE TERMINATION LABELS apiman apiman-chun1.router.default.svc.cluster.local apiman apiman-gateway apiman-gateway-chun1.router.default.svc.cluster.local apiman-gateway elasticsearch-v1 elasticsearch-v1-chun1.router.default.svc.cluster.local elasticsearch-v1 elasticsearch-v1-masters elasticsearch-v1-masters-chun1.router.default.svc.cluster.local elasticsearch-v1-masters fabric8 fabric8.ec2-54-84-229-60.compute-1.amazonaws.com fabric8 provider=fabric8
Created attachment 1143693 [details] es-cluster-503-page
Sorry for typo, the response page is 502 in Comment #15. Do we need create a separate pod for elesticsearch cluster? There is not such object definition in the latest apiman template for now.
1. Yes you need a separate elastic search pod. We first considered reusing the one that comes with logging but now we are going with a dedicated ES for apiman&apiman-gateway instead. 2. Your should be able to see the elasticsearch-v1 instance by using http://elasticsearch-v1.<your-kubernetes-domain> (don't use the -masters). 3. I will make sure the fabric8/elasticsearch-v1:2.2.96 will be on dockerhub.
ad 3: https://hub.docker.com/r/fabric8/elasticsearch-v1/ is there now.
I tried with below steps, no meet original errors in apiman/apiman-gatway pods, but met "Permission denied" error in elasticsearch-v1 pod: 1). Create apps $ oc new-app --docker-image=fabric8/apiman:2.2.108 $ oc new-app --docker-image=fabric8/apiman-gateway:2.2.108 $ oc new-app --docker-image=fabric8/elasticsearch-v1:2.2.108 2). Check logs of elasticsearch-v1 pod $ oc get pods oc NAME READY STATUS RESTARTS AGE apiman-1-fhckf 1/1 Running 0 13m apiman-gateway-1-dbgx5 1/1 Running 0 13m elasticsearch-v1-1-deploy 0/1 Error 0 11m elasticsearch-v1-1-nu41w 0/1 CrashLoopBackOff 7 11m $ oc logs elasticsearch-v1-1-nu41w /bin/sh: /put-template.sh: Permission denied $ oc logs elasticsearch-v1-1-deploy I0414 06:03:07.463202 1 deployer.go:200] Deploying chun1/elasticsearch-v1-1 for the first time (replicas: 1) I0414 06:03:07.467327 1 recreate.go:126] Scaling chun1/elasticsearch-v1-1 to 1 before performing acceptance check I0414 06:03:09.497146 1 recreate.go:131] Performing acceptance check of chun1/elasticsearch-v1-1 I0414 06:03:09.497246 1 lifecycle.go:445] Waiting 600 seconds for pods owned by deployment "chun1/elasticsearch-v1-1" to become ready (checking every 1 seconds; 0 pods previously accepted) F0414 06:13:09.497512 1 deployer.go:70] update acceptor rejected chun1/elasticsearch-v1-1: pods for deployment "chun1/elasticsearch-v1-1" took longer than 600 seconds to become ready -------- Additionally, do we need to use image "fabric8/elasticsearch-v1:2.2.108" instead of "fabric8/elasticsearch-k8s:1.6.0" in template file named "apiman-2.2.108.json"? The file is come from http://repo1.maven.org/maven2/io/fabric8/ipaas/distro/distro/2.2.108/distro-2.2.108-templates.zip .
Created attachment 1147049 [details] apiman-2.2.108.json
According to comment #20, and we use apiman deployer image to deploy in future, the apiman deployer image card [1]. The "Permission denied" did not met when deploying apiman stack with deployer image. So, change the severity/priority to low level. [1] https://trello.com/c/nDfWr99C/217-5-apiman-deployer-image-apiman
Closing as APIMan is no longer a deliverable for API management on OpenShift.