Clayton, could you comment on the strategy here? Do we want to have the switch to proactively disable connections from older client versions, or let's just take the specific issues and make older clients work?

There is a workaround that works today and allows you to reject commands if client and server don't match to the *exact* same version.

Use the global "--match-server-version" flag, and alias the "oc" command to make sure it is *always* executed with that flag. For example:

alias oc="/path/to/oc --match-server-version"

I think this resolves the problem: https://github.com/openshift/origin/pull/7741 .  It is now possible to reject old clients or skewed clients from making API requests by http verb.  That would allow someone to prevent mutating requests, but still allow read-only requests.

Setting to ON_QA for testing. Notice this may need the addition of new test cases to cover the new configuration options.

David, this might also need some docs. I opened a docs issue, mind sharing an overview of the required configs in there? https://github.com/openshift/openshift-docs/issues/1712

Not merged into OSE yet.
Temporarily, verified with latest Origin:
1. Modify master config as follows:
policyConfig:
  ...
  legacyClientPolicyConfig:
    legacyClientPolicy: deny-old-clients
    ...

Then restart openshift. (Master version: openshift v1.1.3-536-gda71953
kubernetes v1.2.0-alpha.7-703-gbc4550d)

2. In client side, use old oc.
1> $ oc version
oc v3.0.1.100
kubernetes v1.1.0-alpha.0-1605-g44c91b1
2> $ oc login and create project
3> $ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json --loglevel=6
It will give message showing version not match:
Error from server: error when creating "...hello-pod.json": userVersion 1.1.0 is older than the server version 1.2.0; mutation is denied

Will verify OSE server with latest puddle when code merged.
And add new cases when clear Doc is available.

Questions:
1. Should above step 2.3 show openshift version? It shows kube version.
2. `oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/deployment/deployment1.json` succeeds creating dc, which differs from pod. OpenShift and Kube resources by design behave differently?
3. In the context of above old oc and latest openshift, following commands just fail but do not show version unmatch info. Should they also tell customer about version message? Because lastest oc will succeed in following commands.
$ oc set triggers dc hooks
Error: unknown command "set" for "oc"
$ oc logs dc/hooks
Error from server: ... cannot get pods/hooks in project ...
$ oc tag --source=docker centos/ruby-22-centos7 myis:latest
$ oc import-image myis
error: only image streams with spec.dockerImageRepository set may have images imported


Thank you.

Alexander: you mentioned "do not work" and "half work" situations, could you give one example for each? Not sure whether question 3 in comment 10 hits.

I failed to consider OSE/origin combinations.  I'm going to switch the implementation to one that's slightly harder to configure, but provides the flexibility we need to handle those combinations.  Sorry for the trouble, but thanks for your examples.

The new pull to do this is https://github.com/openshift/origin/pull/7888.  It has merged.  The referenced pull has example configs for no-skew and specific denies based on regexes.

I've asked in https://github.com/openshift/openshift-docs/issues/1712 where I should document this.

Test steps:
1. Change master config under "policyConfig:" as follows:
  userAgentMatchingConfig:
    defaultRejectionMessage: "Your client is too old.  Go to https://example.org to update it."
    deniedClients:
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'
    - regex: '\w+/v1\.2.+kubernetes/\w{7}'

Then restart openshift

2. On client side, oc login and create a project
3. Prepare different clients:
$ v1.1.3/oc version
oc v1.1.3-631-g2d2d659
kubernetes v1.2.0-origin-41-g91d3e75
$ v3.2.0.4/oc version
oc v3.2.0.4
kubernetes v1.2.0-origin-41-g91d3e75
$ v3.1.1.6/oc version
oc v3.1.1.6-16-g5327e56
kubernetes v1.1.0-origin-1107-g4c8e6f4

4. Test different clients:
1) $ v1.1.3/oc get dc
2) $ v1.1.3/oc get pod
3) $ v3.2.0.4/oc get dc
4) $ v3.2.0.4/oc get pod
5) $ v3.1.1.6/oc get dc
6) $ v3.1.1.6/oc get pod

Test results:
4.
1) ~ 5) commands succeed.
6) command fails with mesg:
Error from server: Your client is too old.  Go to https://example.org to update it.
The test results are expected.

But if we use deniedClients and requiredClients together, openshift fails to restart successfully.
E.g. allow openshift binary v1.x.* or v3.x.* where x>1 but deny kube binary v1.1.*):
  userAgentMatchingConfig:
    defaultRejectionMessage: "Your client is too old.  Go to https://example.org to update it."
    deniedClients:
    - regex: '\w+/v1\.1.+kubernetes/\w{7}'
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'

Or allow v1.x/v3.x but deny v1.1.2 openshift binary:
    deniedClients:
    - regex: '\w+/v1\.1\.2 \(.+/.+\) openshift/\w{7}'
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'

Then restart openshift，it fails, and we get logs:
E0317 06:53:03.579754   24306 ensure.go:164] Unable to create default security context constraint privileged.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.580409   24306 ensure.go:164] Unable to create default security context constraint nonroot.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.581036   24306 ensure.go:164] Unable to create default security context constraint hostmount-anyuid.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.581664   24306 ensure.go:164] Unable to create default security context constraint hostaccess.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.582277   24306 ensure.go:164] Unable to create default security context constraint restricted.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.582885   24306 ensure.go:164] Unable to create default security context constraint anyuid.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.583496   24306 ensure.go:164] Unable to create default security context constraint hostnetwork.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.584077   24306 ensure.go:105] Error adding service account roles to "default" namespace: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.584710   24306 ensure.go:52] Error creating namespace openshift-infra: Your client is too old.  Go to https://example.org to update it.

For more info, see the attached logs.

Created attachment 1137346 [details]
openshift_fail_to_restart.log

The version number actually refers to the version of the library used to connect to the server.  When you're connecting to the API server to request a kube resource, it is using a kube library, so it presents a kube version.  That means that to require a level of openshift, you have actually have to have require rules for *both* the openshift level and its matching kube level.

Since the server starts and immediately tries to talk back to itself, if you don't allow the kube level, it will fail.

Verified against master:
openshift v3.2.0.4
kubernetes v1.2.0-origin-41-g91d3e75
etcd 2.2.5

With config:
    deniedClients:
    - regex: '\w+/v1\.1\.2 \(.+/.+\) openshift/\w{7}'
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'
    - regex: '\w+/v1\.2.+kubernetes/\w{7}'

Tested with step 4 in comment 15, plus:
$ v1.1.2/oc version
oc v1.1.2-256-g2903c81
kubernetes v1.2.0-origin
$ v3.2.0.3/oc version
oc v3.2.0.3
kubernetes v1.2.0-origin-41-g91d3e75

$ v1.1.2/oc get dc
Error from server: Your client is too old.  Go to https://example.org to update it.
$ v3.2.0.3/oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json # Succeed

The config succeed to allow v1.x/v3.x (x>1) openshift and v1.2.* kube library but deny v1.1.2 openshift libnary.

Hi,  David Eads, found that in v3.2.0.7, if use the config in comment 19, it would fail to deploy and build, the deployment/build is always "new" (but in comment 19, v3.2.0.4 did have no this problem). So checked Origin, it has the same problem, see below:
1. Check latest Origin version:
$ openshift version
openshift v1.1.4-294-g728e6a6
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5
2. Create config:
$ openshift start --public-master=https://<master>:8443 --write-config='openshift.local.config'
3. Configure userAgentMatchingConfig as follows:
  userAgentMatchingConfig:
    defaultRejectionMessage: ""
    deniedClients: null
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'
    - regex: '\w+/v1\.2.+kubernetes/\w{7}'

Then start openshift:
$ openshift start --master-config='openshift.local.config/master/master-config.yaml' --node-config='openshift.local.config/node-ip-172-18-12-85.ec2.internal/node-config.yaml' &> openshift.log &
4. On client side (same version as openshift):
$ oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json
5. Check deployment, rc, build, pod (Other resources like svc, have no problem, so snipped)
$ oc get rc
NAME         DESIRED   CURRENT   AGE
database-1   0         0         49m
$ oc deploy database
database deployment #1 new about an hour ago
$ oc get build
NAME                  TYPE      FROM      STATUS    STARTED   DURATION
ruby-sample-build-1   Source    Git       New
$ oc get pod    # Get empty result
$ oc get dc database
NAME       REVISION   REPLICAS   TRIGGERED BY
database   1          1          config
6. If create Pod json file, the pod is running:
$ oc create -f origin/examples/hello-openshift/hello-pod.json
$ oc get pod
NAME              READY     STATUS    RESTARTS   AGE
hello-openshift   1/1       Running   0          5s

In sum, deployment and build failed.

If replace the first regex with following choice **respectively**, and repeat the above steps, we got the same result:
    - regex: 'v[13]\.[1-9].+openshift'
    - regex: 'v1\.1.+openshift'
    - regex: 'v1\..+openshift'
    - regex: 'v1.+openshift'

Strangely, if, instead, replace with:
    - regex: 'v.+openshift'
Then the problem is gone (Just removed '1').

To confirm this problem is regression, check the origin git logs and use hack/build-go.sh to compile this version:
bd17b6f - Merge pull request #7888 from deads2k/make-user-agent-a-regex
$ openshift version
openshift v1.1.4-27-gbd17b6f
kubernetes v1.2.0-origin-41-g91d3e75
etcd 2.2.5

Then repeat the above steps, the problem does not exist!

Yeah, it broke when the SA user agents were changed.  Fix here: https://github.com/openshift/origin/pull/8257

Hi, found new issue. When using the configuration of comment 15, the function works for oc command client, but fails for other clients such as web and curl whose User-Agent mismatch the configuration:
1. Web console login returns simple "An error has occurred" (no more info), and the url displays "console/error?error=user_fetch_failed".
Use FireFox's Web Console tool, found network request like:
User-Agent: "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"

2. curl API also fail:
curl -H "Authorization: Bearer cDo6QQO2Z0ujJ2If6J36Rdjh4FniqwCKJc9w8STp6yA" "https://<master>:8443/oapi/v1/users/~" -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Your client is too old.  Go to https://example.org to update it.",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Use -v to curl, it shows: "User-Agent: curl/7.29.0"

The configuration function's original intention might be designed for oc command. If user has to cover other web and curl tool, etc., clients, then this configuration may be not of good use.

If the user really wants to enforce a whitelist, then they'll have to name everything they want in that whitelist.  I suspect that a blacklist will be easier to manage, but I could also envision people trying to enforce standardized access to ensure that mutating requests match.

Since the rules can be mapped to specific verbs, they could ban mutating requests while allowing non-mutating requests.  Since the whitelist and blacklist can be used together, I could see someone blocking POST and PUT by whitelist and the rest by blacklist.  I think that would allow the console to mostly work, while requiring an exact match to mutate data.

Thanks. Did not dive into the PR code. Now found httpVerbs field, and updated former TC and added that checkpoint in new TC. The function works well. Suggestion: will make user not confused if state a note for the usage of comment 22 when writing Doc for this function in future.
BTW, comment 21 is merged and works.

Verified in openshift v3.2.0.16 following steps in comment 20.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064