Bug 1306590 - Optionally reject connection from older client versions
Optionally reject connection from older client versions
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Command Line Interface (Show other bugs)
3.1.0
All Linux
urgent Severity urgent
: ---
: ---
Assigned To: Fabiano Franz
Wei Sun
:
Depends On:
Blocks: 1267746 1339502
  Show dependency treegraph
 
Reported: 2016-02-11 06:50 EST by Alexander Koksharov
Modified: 2016-05-30 06:30 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 12:28:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Koksharov 2016-02-11 06:50:44 EST
Description of problem:
When we update OpenShift, many customers still use older versions of oc client. Using older versions against the new API causes them serious troubles because, some things work, some other things do not work and some other things half work.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
- The API should reject connections from older oc clients with a descriptive user-friendly error message
- If the API accepts connections from older oc clients, it should behave in a really compatible way and properly reject non-supported actions when needed (again with descriptive user-friendly error messages)

Additional info:
Athough this is a RFE, this is quite important for us, as this is a big usability problem for our users and has caused many issues to them, overloading our team with support issues.
Comment 2 Fabiano Franz 2016-02-16 15:02:04 EST
Clayton, could you comment on the strategy here? Do we want to have the switch to proactively disable connections from older client versions, or let's just take the specific issues and make older clients work?
Comment 6 Fabiano Franz 2016-02-19 09:30:02 EST
There is a workaround that works today and allows you to reject commands if client and server don't match to the *exact* same version.

Use the global "--match-server-version" flag, and alias the "oc" command to make sure it is *always* executed with that flag. For example:

alias oc="/path/to/oc --match-server-version"
Comment 8 David Eads 2016-03-08 15:27:47 EST
I think this resolves the problem: https://github.com/openshift/origin/pull/7741 .  It is now possible to reject old clients or skewed clients from making API requests by http verb.  That would allow someone to prevent mutating requests, but still allow read-only requests.
Comment 9 Fabiano Franz 2016-03-08 15:41:01 EST
Setting to ON_QA for testing. Notice this may need the addition of new test cases to cover the new configuration options.

David, this might also need some docs. I opened a docs issue, mind sharing an overview of the required configs in there? https://github.com/openshift/openshift-docs/issues/1712
Comment 10 Xingxing Xia 2016-03-09 05:50:48 EST
Not merged into OSE yet.
Temporarily, verified with latest Origin:
1. Modify master config as follows:
policyConfig:
  ...
  legacyClientPolicyConfig:
    legacyClientPolicy: deny-old-clients
    ...

Then restart openshift. (Master version: openshift v1.1.3-536-gda71953
kubernetes v1.2.0-alpha.7-703-gbc4550d)

2. In client side, use old oc.
1> $ oc version
oc v3.0.1.100
kubernetes v1.1.0-alpha.0-1605-g44c91b1
2> $ oc login and create project
3> $ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json --loglevel=6
It will give message showing version not match:
Error from server: error when creating "...hello-pod.json": userVersion 1.1.0 is older than the server version 1.2.0; mutation is denied

Will verify OSE server with latest puddle when code merged.
And add new cases when clear Doc is available.

Questions:
1. Should above step 2.3 show openshift version? It shows kube version.
2. `oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/deployment/deployment1.json` succeeds creating dc, which differs from pod. OpenShift and Kube resources by design behave differently?
3. In the context of above old oc and latest openshift, following commands just fail but do not show version unmatch info. Should they also tell customer about version message? Because lastest oc will succeed in following commands.
$ oc set triggers dc hooks
Error: unknown command "set" for "oc"
$ oc logs dc/hooks
Error from server: ... cannot get pods/hooks in project ...
$ oc tag --source=docker centos/ruby-22-centos7 myis:latest
$ oc import-image myis
error: only image streams with spec.dockerImageRepository set may have images imported


Thank you.
Comment 12 Xingxing Xia 2016-03-09 06:18:41 EST
Alexander: you mentioned "do not work" and "half work" situations, could you give one example for each? Not sure whether question 3 in comment 10 hits.
Comment 13 David Eads 2016-03-09 08:50:22 EST
I failed to consider OSE/origin combinations.  I'm going to switch the implementation to one that's slightly harder to configure, but provides the flexibility we need to handle those combinations.  Sorry for the trouble, but thanks for your examples.
Comment 14 David Eads 2016-03-16 16:37:32 EDT
The new pull to do this is https://github.com/openshift/origin/pull/7888.  It has merged.  The referenced pull has example configs for no-skew and specific denies based on regexes.

I've asked in https://github.com/openshift/openshift-docs/issues/1712 where I should document this.
Comment 15 Xingxing Xia 2016-03-17 06:49:35 EDT
Test steps:
1. Change master config under "policyConfig:" as follows:
  userAgentMatchingConfig:
    defaultRejectionMessage: "Your client is too old.  Go to https://example.org to update it."
    deniedClients:
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'
    - regex: '\w+/v1\.2.+kubernetes/\w{7}'

Then restart openshift

2. On client side, oc login and create a project
3. Prepare different clients:
$ v1.1.3/oc version
oc v1.1.3-631-g2d2d659
kubernetes v1.2.0-origin-41-g91d3e75
$ v3.2.0.4/oc version
oc v3.2.0.4
kubernetes v1.2.0-origin-41-g91d3e75
$ v3.1.1.6/oc version
oc v3.1.1.6-16-g5327e56
kubernetes v1.1.0-origin-1107-g4c8e6f4

4. Test different clients:
1) $ v1.1.3/oc get dc
2) $ v1.1.3/oc get pod
3) $ v3.2.0.4/oc get dc
4) $ v3.2.0.4/oc get pod
5) $ v3.1.1.6/oc get dc
6) $ v3.1.1.6/oc get pod

Test results:
4.
1) ~ 5) commands succeed.
6) command fails with mesg:
Error from server: Your client is too old.  Go to https://example.org to update it.
The test results are expected.
Comment 16 Xingxing Xia 2016-03-17 07:08:55 EDT
But if we use deniedClients and requiredClients together, openshift fails to restart successfully.
E.g. allow openshift binary v1.x.* or v3.x.* where x>1 but deny kube binary v1.1.*):
  userAgentMatchingConfig:
    defaultRejectionMessage: "Your client is too old.  Go to https://example.org to update it."
    deniedClients:
    - regex: '\w+/v1\.1.+kubernetes/\w{7}'
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'

Or allow v1.x/v3.x but deny v1.1.2 openshift binary:
    deniedClients:
    - regex: '\w+/v1\.1\.2 \(.+/.+\) openshift/\w{7}'
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'

Then restart openshift,it fails, and we get logs:
E0317 06:53:03.579754   24306 ensure.go:164] Unable to create default security context constraint privileged.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.580409   24306 ensure.go:164] Unable to create default security context constraint nonroot.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.581036   24306 ensure.go:164] Unable to create default security context constraint hostmount-anyuid.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.581664   24306 ensure.go:164] Unable to create default security context constraint hostaccess.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.582277   24306 ensure.go:164] Unable to create default security context constraint restricted.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.582885   24306 ensure.go:164] Unable to create default security context constraint anyuid.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.583496   24306 ensure.go:164] Unable to create default security context constraint hostnetwork.  Got error: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.584077   24306 ensure.go:105] Error adding service account roles to "default" namespace: Your client is too old.  Go to https://example.org to update it.
E0317 06:53:03.584710   24306 ensure.go:52] Error creating namespace openshift-infra: Your client is too old.  Go to https://example.org to update it.

For more info, see the attached logs.
Comment 17 Xingxing Xia 2016-03-17 07:12 EDT
Created attachment 1137346 [details]
openshift_fail_to_restart.log
Comment 18 David Eads 2016-03-17 07:42:53 EDT
The version number actually refers to the version of the library used to connect to the server.  When you're connecting to the API server to request a kube resource, it is using a kube library, so it presents a kube version.  That means that to require a level of openshift, you have actually have to have require rules for *both* the openshift level and its matching kube level.

Since the server starts and immediately tries to talk back to itself, if you don't allow the kube level, it will fail.
Comment 19 Xingxing Xia 2016-03-18 05:29:03 EDT
Verified against master:
openshift v3.2.0.4
kubernetes v1.2.0-origin-41-g91d3e75
etcd 2.2.5

With config:
    deniedClients:
    - regex: '\w+/v1\.1\.2 \(.+/.+\) openshift/\w{7}'
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'
    - regex: '\w+/v1\.2.+kubernetes/\w{7}'

Tested with step 4 in comment 15, plus:
$ v1.1.2/oc version
oc v1.1.2-256-g2903c81
kubernetes v1.2.0-origin
$ v3.2.0.3/oc version
oc v3.2.0.3
kubernetes v1.2.0-origin-41-g91d3e75

$ v1.1.2/oc get dc
Error from server: Your client is too old.  Go to https://example.org to update it.
$ v3.2.0.3/oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json # Succeed

The config succeed to allow v1.x/v3.x (x>1) openshift and v1.2.* kube library but deny v1.1.2 openshift libnary.
Comment 20 Xingxing Xia 2016-03-28 03:13:24 EDT
Hi,  David Eads, found that in v3.2.0.7, if use the config in comment 19, it would fail to deploy and build, the deployment/build is always "new" (but in comment 19, v3.2.0.4 did have no this problem). So checked Origin, it has the same problem, see below:
1. Check latest Origin version:
$ openshift version
openshift v1.1.4-294-g728e6a6
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5
2. Create config:
$ openshift start --public-master=https://<master>:8443 --write-config='openshift.local.config'
3. Configure userAgentMatchingConfig as follows:
  userAgentMatchingConfig:
    defaultRejectionMessage: ""
    deniedClients: null
    requiredClients:
    - regex: '\w+/v(1\.[1-9]|3\.[1-9]).+openshift/\w{7}'
    - regex: '\w+/v1\.2.+kubernetes/\w{7}'

Then start openshift:
$ openshift start --master-config='openshift.local.config/master/master-config.yaml' --node-config='openshift.local.config/node-ip-172-18-12-85.ec2.internal/node-config.yaml' &> openshift.log &
4. On client side (same version as openshift):
$ oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json
5. Check deployment, rc, build, pod (Other resources like svc, have no problem, so snipped)
$ oc get rc
NAME         DESIRED   CURRENT   AGE
database-1   0         0         49m
$ oc deploy database
database deployment #1 new about an hour ago
$ oc get build
NAME                  TYPE      FROM      STATUS    STARTED   DURATION
ruby-sample-build-1   Source    Git       New
$ oc get pod    # Get empty result
$ oc get dc database
NAME       REVISION   REPLICAS   TRIGGERED BY
database   1          1          config
6. If create Pod json file, the pod is running:
$ oc create -f origin/examples/hello-openshift/hello-pod.json
$ oc get pod
NAME              READY     STATUS    RESTARTS   AGE
hello-openshift   1/1       Running   0          5s

In sum, deployment and build failed.

If replace the first regex with following choice **respectively**, and repeat the above steps, we got the same result:
    - regex: 'v[13]\.[1-9].+openshift'
    - regex: 'v1\.1.+openshift'
    - regex: 'v1\..+openshift'
    - regex: 'v1.+openshift'

Strangely, if, instead, replace with:
    - regex: 'v.+openshift'
Then the problem is gone (Just removed '1').

To confirm this problem is regression, check the origin git logs and use hack/build-go.sh to compile this version:
bd17b6f - Merge pull request #7888 from deads2k/make-user-agent-a-regex
$ openshift version
openshift v1.1.4-27-gbd17b6f
kubernetes v1.2.0-origin-41-g91d3e75
etcd 2.2.5

Then repeat the above steps, the problem does not exist!
Comment 21 David Eads 2016-03-28 08:11:13 EDT
Yeah, it broke when the SA user agents were changed.  Fix here: https://github.com/openshift/origin/pull/8257
Comment 22 Xingxing Xia 2016-04-11 06:21:27 EDT
Hi, found new issue. When using the configuration of comment 15, the function works for oc command client, but fails for other clients such as web and curl whose User-Agent mismatch the configuration:
1. Web console login returns simple "An error has occurred" (no more info), and the url displays "console/error?error=user_fetch_failed".
Use FireFox's Web Console tool, found network request like:
User-Agent: "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"

2. curl API also fail:
curl -H "Authorization: Bearer cDo6QQO2Z0ujJ2If6J36Rdjh4FniqwCKJc9w8STp6yA" "https://<master>:8443/oapi/v1/users/~" -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Your client is too old.  Go to https://example.org to update it.",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Use -v to curl, it shows: "User-Agent: curl/7.29.0"

The configuration function's original intention might be designed for oc command. If user has to cover other web and curl tool, etc., clients, then this configuration may be not of good use.
Comment 23 David Eads 2016-04-11 08:04:18 EDT
If the user really wants to enforce a whitelist, then they'll have to name everything they want in that whitelist.  I suspect that a blacklist will be easier to manage, but I could also envision people trying to enforce standardized access to ensure that mutating requests match.

Since the rules can be mapped to specific verbs, they could ban mutating requests while allowing non-mutating requests.  Since the whitelist and blacklist can be used together, I could see someone blocking POST and PUT by whitelist and the rest by blacklist.  I think that would allow the console to mostly work, while requiring an exact match to mutate data.
Comment 24 Xingxing Xia 2016-04-13 23:32:19 EDT
Thanks. Did not dive into the PR code. Now found httpVerbs field, and updated former TC and added that checkpoint in new TC. The function works well. Suggestion: will make user not confused if state a note for the usage of comment 22 when writing Doc for this function in future.
BTW, comment 21 is merged and works.
Comment 25 Xingxing Xia 2016-04-18 22:08:59 EDT
Verified in openshift v3.2.0.16 following steps in comment 20.
Comment 27 errata-xmlrpc 2016-05-12 12:28:39 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064

Note You need to log in before you can comment on or make changes to this bug.