Bug 1569928 - [online-int] Failed to create EAP CD and A-MQ persistent based application with template eap-cd-amq-persistent-s2i
Summary: [online-int] Failed to create EAP CD and A-MQ persistent based application wi...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Templates
Version: unspecified
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Abhishek Gupta
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-20 10:17 UTC by wewang
Modified: 2018-08-13 01:35 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description wewang 2018-04-20 10:17:01 UTC
Description of problem:

Cannot deploy eap-app with "error: update acceptor rejected eap-app-1" when creating EAP CD and A-MQ persistent based application

Version-Release number of selected component (if applicable):
OpenShift Master:    v3.9.14 (online version 3.6.0.90) 
Kubernetes Master:    v1.9.1+a0ce1bc657 
OpenShift Web Console:    v3.9.14 

How reproducible:
Always

Steps to Reproduce:
1. $ oc new-app --template=eap-cd-amq-persistent-s2i
2. $oc get pods
$ oc get pods
NAME                  READY     STATUS      RESTARTS   AGE
eap-app-1-build       0/1       Completed   0          13m
eap-app-1-deploy      0/1       Error       0          12m
eap-app-amq-1-f5z2z   1/1       Running     0          13m

3.Check the log
$ oc logs eap-app-1-deploy
--> Scaling eap-app-1 to 1
error: update acceptor rejected eap-app-1: pods for rc 'wewang5/eap-app-1' took longer than 600 seconds to become available
 4.$ oc describe pod eap-app-1-deploy
Name:         eap-app-1-deploy
Namespace:    wewang5
Node:         ip-172-31-52-148.ec2.internal/172.31.52.148
Start Time:   Fri, 20 Apr 2018 17:27:27 +0800
Labels:       openshift.io/deployer-pod-for.name=eap-app-1
Annotations:  kubernetes.io/limit-ranger=LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment
              openshift.io/deployment-config.name=eap-app
              openshift.io/deployment.name=eap-app-1
              openshift.io/scc=restricted
Status:       Failed
IP:           10.131.2.121
Containers:
  deployment:
    Container ID:   docker://66105d5dbac18b23fbee1af72987ea569769b557f8e2062ff56e6f4c61d6ad7f
    Image:          registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.9.14
    Image ID:       docker-pullable://registry.reg-aws.openshift.com:443/openshift3/ose-deployer@sha256:6856892fc4447f6391bd2523a25fe3b94d78a4330d8124eadbe945ad5f120390
    Port:           <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 20 Apr 2018 17:27:31 +0800
      Finished:     Fri, 20 Apr 2018 17:37:32 +0800
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:     80m
      memory:  409Mi
    Environment:
      KUBERNETES_MASTER:  https://ip-172-31-48-173.ec2.internal
      OPENSHIFT_MASTER:   https://ip-172-31-48-173.ec2.internal
      BEARER_TOKEN_FILE:  /var/run/secrets/kubernetes.io/serviceaccount/token
      OPENSHIFT_CA_DATA:  -----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE1MDE3MTAzOTgwHhcNMTcwODAyMjE0NjM3WhcNMjIwODAx
MjE0NjM4WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MDE3MTAzOTgw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+u1c9cCGAMrRWL8hwp/4K
SPcr5NazR6SUssYAqMsbVtYoYz1I+gWG1ajaGH4F8lYYVGI4LEmcTok2Aje7bZFq
iZwqLoTAckyfxBIpeLLukC/WDbGimRGT52lklkOl6MMF9v3IPJnWhLYQ3ideQ5+m
b7TPRJ7hN6dw4WzkXrfKhY7MGhahrkCt6j8F8UvpLF8AaMGQBdmTUnrIF7ypRlJd
GgShS+NndgYcg/nwl+j84kiKIhULO8o3bBemNmu/qEYijsFuD5XFO1hHJVRMoRt3
oJOxyjlSCMwnF6HnG/7nAl3qyuHTjNwHCGv5wqyHz9GQSmwY7jolYgcJZbvJjx4N
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQBKeSd6+XWAL376KTimLRoN7UjOpatB8KVYtdFY+17xNsJ2
1eJLw8cqO8NRISLq4fYN7LQVCjHcaEK8ZeLIlzyVep6pmMgMcoTasP1ytmLfSddO
hx+l967ioMjcRcMyDaEnl0VgxMrs2ExT/2A6V3S4/m+N2SkVilgcLMqBfINs4uEu
B/wdJag+EiCbNyxI1GnaFK6f+AV+l0nQVh8APlBM7KHmorb7kX6ta5DAX7KXEvSv
I2JLLyRthbcFYE9RUra3opV+DX0zbL6PpnT0ccy0NwGwC7rktbtrTObv6RWLYfVW
afkzHlwrMIyG7ewQzvDmsuyW0TiUaqC/zFuy+4Il
-----END CERTIFICATE-----

      OPENSHIFT_DEPLOYMENT_NAME:       eap-app-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:  wewang5
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-zlzdx (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  deployer-token-zlzdx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  deployer-token-zlzdx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  type=compute
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type    Reason                 Age   From                                    Message
  ----    ------                 ----  ----                                    -------
  Normal  Scheduled              24m   default-scheduler                       Successfully assigned eap-app-1-deploy to ip-172-31-52-148.ec2.internal
  Normal  SuccessfulMountVolume  24m   kubelet, ip-172-31-52-148.ec2.internal  MountVolume.SetUp succeeded for volume "deployer-token-zlzdx"
  Normal  Pulling                24m   kubelet, ip-172-31-52-148.ec2.internal  pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.9.14"
  Normal  Pulled                 24m   kubelet, ip-172-31-52-148.ec2.internal  Successfully pulled image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.9.14"
  Normal  Created                24m   kubelet, ip-172-31-52-148.ec2.internal  Created container
  Normal  Started                24m   kubelet, ip-172-31-52-148.ec2.internal  Started container
Actual results:
eap-app pod is not running

Expected results:
eap-app pod is running

Comment 1 Ben Parees 2018-08-09 22:15:30 UTC
Is this still broken?

It's been months (And several upgrades) so i would hope not (the original problem appears to have been an infrastructure issue w/ the nodes).

Comment 2 wewang 2018-08-10 02:00:58 UTC
Tested in online-int version:v3.9.31
met error:API server is missing declared resources: /v1, Kind=Route, /v1, Kind=ImageStream, /v1, Kind=BuildConfig, /v1, Kind=DeploymentConfig


steps:
$ oc create -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/secrets/eap7-app-secret.json
secret/eap7-app-secret created

$ oc new-app --template=eap-cd-amq-persistent-s2i

     * With parameters:
        * Application Name=eap-app
        * Custom https Route Hostname=
        * Git Repository URL=https://github.com/jboss-developer/jboss-eap-quickstarts.git
        * Git Reference=openshift
        * Context Directory=helloworld-mdb
        * AMQ Volume Size=1Gi
        * JMS Connection Factory JNDI Name=java:jboss/DefaultJMSConnectionFactory
        * Split the data directory?=false
        * AMQ Protocols=openwire
        * Queues=HelloWorldMDBQueue
        * Topics=HelloWorldMDBTopic
        * AMQ Serializable Packages=
        * Server Keystore Secret Name=eap7-app-secret
        * Server Keystore Filename=keystore.jks
        * Server Keystore Type=
        * Server Certificate Name=
        * Server Keystore Password=
        * AMQ Username=useriKU # generated
        * AMQ Password=7RrcMIl3 # generated
        * AMQ Role=admin
        * AMQ Mesh Discovery Type=dns
        * AMQ Storage Limit=100 gb
        * Github Webhook Secret=Svh4kT6f # generated
        * Generic Webhook Secret=3QK0X4Ut # generated
        * ImageStream Namespace=openshift
        * JGroups Secret Name=eap7-app-secret
        * JGroups Keystore Filename=jgroups.jceks
        * JGroups Certificate Name=
        * JGroups Keystore Password=
        * JGroups Cluster Password=TWHXEBuI # generated
        * Deploy Exploded Archives=false
        * Maven mirror URL=
        * Maven Additional Arguments=-Dcom.redhat.xpaas.repo.jbossorg
        * ARTIFACT_DIR=
        * MEMORY_LIMIT=1Gi

error: API server is missing declared resources: /v1, Kind=Route, /v1, Kind=ImageStream, /v1, Kind=BuildConfig, /v1, Kind=DeploymentConfig

Comment 3 Ben Parees 2018-08-10 03:23:50 UTC
Seems like there's a missing step in your recreate instructions, because when i do it the template is not found:

$ oc version
oc v3.11.0-alpha.0+ec51b90-815-dirty
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://api.free-int.openshift.com:443
openshift v3.11.0-0.10.0
kubernetes v1.11.0+d4cacc0

$  oc create -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/secrets/eap7-app-secret.json
secret/eap7-app-secret created

$ oc new-app --template=eap-cd-amq-persistent-s2i
error: unable to locate any templates loaded in accessible projects with name "eap-cd-amq-persistent-s2i"


Did you manually install the template first?

If I do use the template from the upstream source, it seems to work:

$  oc new-app -f https://raw.githubusercontent.com/jboss-container-images/jboss-eap-7-openshift-image/eap-cd/templates/eap-cd-amq-persistent-s2i.json
--> Deploying template "bparees/eap-cd-amq-persistent-s2i" for "https://raw.githubusercontent.com/jboss-container-images/jboss-eap-7-openshift-image/eap-cd/templates/eap-cd-amq-persistent-s2i.json" to project bparees

     JBoss EAP CD + AMQ 7 (with https)
     ---------
     An example JBoss Enterprise Application Platform continuous delivery application using Red Hat JBoss AMQ with persistence and secure communication using https. For more information about using this template, see https://github.com/jboss-container-images/jboss-eap-7-openshift-image/blob/eap-cd/README.adoc

     A new JBoss EAP CD and AMQ persistent based application with SSL support has been created in your project. The username/password for accessing the AMQ service is useriq6/SmVohXLr. Please be sure to create the following secrets: "eap7-app-secret" containing the keystore.jks file used for serving secure content; "eap7-app-secret" containing the jgroups.jceks file used for securing JGroups communications.

     * With parameters:
        * Application Name=eap-app
        * Custom https Route Hostname=
        * Git Repository URL=https://github.com/jboss-developer/jboss-eap-quickstarts.git
        * Git Reference=openshift
        * Context Directory=helloworld-mdb
        * AMQ Volume Size=1Gi
        * JMS Connection Factory JNDI Name=java:jboss/DefaultJMSConnectionFactory
        * Split the data directory?=false
        * AMQ Protocols=openwire
        * Queues=HelloWorldMDBQueue
        * Topics=HelloWorldMDBTopic
        * AMQ Serializable Packages=
        * Server Keystore Secret Name=eap7-app-secret
        * Server Keystore Filename=keystore.jks
        * Server Keystore Type=
        * Server Certificate Name=
        * Server Keystore Password=
        * AMQ Username=useriq6 # generated
        * AMQ Password=SmVohXLr # generated
        * AMQ Role=admin
        * AMQ Mesh Discovery Type=dns
        * AMQ Storage Limit=100 gb
        * Github Webhook Secret=a4Xf7tul # generated
        * Generic Webhook Secret=cPi5Bpa3 # generated
        * ImageStream Namespace=openshift
        * JGroups Secret Name=eap7-app-secret
        * JGroups Keystore Filename=jgroups.jceks
        * JGroups Certificate Name=
        * JGroups Keystore Password=
        * JGroups Cluster Password=dEGEnESW # generated
        * Deploy Exploded Archives=false
        * Maven mirror URL=
        * Maven Additional Arguments=-Dcom.redhat.xpaas.repo.jbossorg
        * ARTIFACT_DIR=
        * MEMORY_LIMIT=1Gi

W0809 23:18:27.278319   19334 newapp.go:1207] Unable to check for circular build input: Unable to check for circular build input/outputs: imagestreams.image.openshift.io "eap-cd-openshift" not found
--> Creating resources ...
    service "eap-app" created
    service "secure-eap-app" created
    service "eap-app-ping" created
    service "eap-app-amq-tcp" created
    service "eap-app-amq-mesh" created
    route.route.openshift.io "eap-app" created
    route.route.openshift.io "secure-eap-app" created
    imagestream.image.openshift.io "eap-app" created
    buildconfig.build.openshift.io "eap-app" created
    deploymentconfig.apps.openshift.io "eap-app" created
    deploymentconfig.apps.openshift.io "eap-app-amq" created
    persistentvolumeclaim "eap-app-amq-claim" created
--> Success
    Access your application via route 'eap-app-bparees.d800.free-int.openshiftapps.com' 
    Access your application via route 'secure-eap-app-bparees.d800.free-int.openshiftapps.com' 
    Build scheduled, use 'oc logs -f bc/eap-app' to track its progress.
    Run 'oc status' to view your app.

at least as far as creating the resources from the template (further than your recreate got).

No pods actually start up, but that would appear to be a cluster problem, no builds are starting either.

Also that cluster is at v3.9 which seems strange given that prod is at 3.10.

Comment 4 Ben Parees 2018-08-10 03:26:07 UTC
also it looks like you used a v3.11 client with a v3.9 server, which is more than the n-1 compatibility guarantee.  That may be part of why you're seeing the resource issue.

Comment 5 wewang 2018-08-10 06:33:10 UTC
@Ben Parees, it's online-int bug not free-int bug:> good catch, I am using v3.11 client.now using v3.9 client(oc v3.9.40), it works pod is running.

[wewang@wen-local cucushift]$ oc get pods
NAME              READY     STATUS      RESTARTS   AGE
eap-app-1-build   0/1       Completed   0          2m
eap-app-1-snl94   1/1       Running     0          33s

so newer client cannot used with older server, right? if so, I think the bug can be verified

Comment 6 Ben Parees 2018-08-10 12:56:39 UTC
> so newer client cannot used with older server, right? 

They must be within one version of each other.  So you can use a 3.11 client with a 3.10 server, but not a 3.9 server.

Comment 7 wewang 2018-08-13 01:35:43 UTC
ok, thanks your response, will verify the bug.


Note You need to log in before you can comment on or make changes to this bug.