+++ This bug was initially created as a clone of Bug #1253726 +++ But is designed to be an abstraction "so that the same concepts used by EAP can be used by developers for other application types / frameworks USECASE: That is not defined: As a "Developer" I would like to write an application that is "scalable", and has the ability to use "libraries/tools" to form a "cluster" so that data between my pods is replicated or shared (in case one pod is lost). How do I "discover" or get "notifications" about members of my cluster (should match the number of "pods" defined in cluster settings). Document URL: NONE Section Number and Name: No section. I'm suggesting a new one. Describe the issue: The issue here is we do not describe how a "developer" from with in a pod can / should collect information from the Kubernetties or OpenShift API's (events) to be notified or see what "peers" (pods) it has. Suggestions for improvement: Create a documenation section that "documents" how EAP / JGroups uses the KUBE_PING or DNS_PING protocols to query the OpenShift and Kubernetties API's to see members of the cluster. EAP gets the list of pods from kubernetes, in the specified namespace, and filtered by application label. For example: https://HOST:443/api/v1/namespaces/NAMESPACE/pods?labelSelector=application%3APPLICATION-NAME JGroups then parses the JSON it gets back from the API call to form the cluster. source code is here: https://github.com/jboss-openshift/openshift-ping/tree/1.0.0.Beta8/kube/src/main/java/org/openshift/ping/kube (Look at KubePing.java and Client.java , mostly) Additional information: https://docs.openshift.com/enterprise/3.1/rest_api/kubernetes_v1.html#list-or-watch-objects-of-kind-endpoints Can likely be used to watched a stream of add, update, and remove notifications. However we need to explain to customer how a "service account" or SCC needs to be created / enabled to access these endpoints. If I am not mistaken EAP enables Authorization (to the API), which must be granted to the service account the pod is running under to be allowed to access Kubernetes' REST api. This is done on the command line. These are needed to get the KUBE_PING to work or to allow a pod to access the kube service. So this too must be explained / elaborated on so developers can be successful. Example 1. Policy commands http://ce-docs.usersys.redhat.com/openshift/eap/6.4/clustering.html Using the default service account in the myproject namespace: # oc policy add-role-to-user view system:serviceaccount:$(oc project -q):default -n $(oc project -q) Using the eap-service-account in the myproject namespace: # oc policy add-role-to-user view system:serviceaccount:$(oc project -q):eap-service-account -n $(oc project -q)
Tracking this in PR https://github.com/openshift/openshift-docs/pull/7565
FWIW, we've moved away from kube-ping because it requires adding view perms to the service account running the pods. DNS is a more natural fit, as you can use a service to group the endpoints. A headless service works nicely for this, which allows you to do a simple dns query to list all the siblings' pod IPs. That said, for EAP, we are using the "tolerate unready pods" annotation, as cluster formation occurs during launch, before the server is "ready." I'd say that's a caveat based on how whatever technology you're using performs its setup. (EAP works without this setting, but the pods initially form a cluster of one, which is then merged into the "real" cluster once it's ready, which can load the cluster with state transfer requests, which probably isn't ideal, especially if you're scaling up to handle load.)
Thanks Eric, Ben, Rob As Eric said, we want to add information that's generic to OpenShift.
defining a headless service that groups the pods and then using DNS to look up the endpoints for the DNS entry is probably the recommendation these days, as Rob notes. Does not require additional permissions.
Update the PR https://github.com/openshift/openshift-docs/pull/7565 to include following sections: - headless service definition - endpoint discovery using headless service example
Updated the PR to include additional information about `publishNotReadyAddresses` as shared in GitHub PR.
Please check the updates in PR https://github.com/openshift/openshift-docs/pull/7565 GitHub rendered version is available at https://github.com/gaurav-nelson/openshift-docs/blob/7920f855141324d4ca29b4315edaf2bb92fac83d/architecture/core_concepts/pods_and_services.adoc#headless-services
the doc PR looks good, mark this as verified. thanks
Commit pushed to master at https://github.com/openshift/openshift-docs https://github.com/openshift/openshift-docs/commit/59997fe6443bd7e2cef0f88a9cea3ad4b0abe399 Merge pull request #7565 from gaurav-nelson/bug1293988-fixes Headlless service and pod discovery
Changes are live at https://docs.openshift.com/container-platform/3.9/architecture/core_concepts/pods_and_services.html#headless-services