Bug 1776259 - KubeAPI server doesn't accept patch requests > 1MB
Summary: KubeAPI server doesn't accept patch requests > 1MB
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.4.0
Assignee: Stefan Schimanski
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On: 1776226
Blocks: 1776275
TreeView+ depends on / blocked
 
Reported: 2019-11-25 11:18 UTC by Lukasz Szaszkiewicz
Modified: 2020-05-04 11:17 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:16:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:17:24 UTC

Description Lukasz Szaszkiewicz 2019-11-25 11:18:02 UTC
The following PR[1] fixes bounds checking of large JSON patches in KubeAPI server. Before that sending/applying large (1 MB) patches were not possible[2]. The PR has been already backported to the master branch [3]

[1]https://github.com/kubernetes/kubernetes/pull/84963/files
[2]https://github.com/kubernetes/kubernetes/issues/84908
[3]https://github.com/openshift/origin/pull/24183

Comment 2 Lukasz Szaszkiewicz 2019-12-09 09:09:28 UTC
Yes, the fix was added to 4.3 release.

Originally this issue was targeting https://github.com/openshift/origin/pull/24197 (release 3.10) but I am not sure if we want to merge it there.
/cc @mfojtik

Comment 5 Ke Wang 2020-01-03 08:33:19 UTC
[Verification ENV:]
$ oc version
Client Version: v4.4.0
Server Version: 4.4.0-0.nightly-2019-12-20-210709
Kubernetes Version: v1.17.0


-Prepare the testing data for verification.

$ for i in {1..1024}; do echo -n a >> bytes-1k.txt; done                                                                                                                                        
$ for i in {1..1034}; do cat bytes-1k.txt >> bytes-1m-more.txt; done    
    
$ cat > mypod.json << EOF    
{    
    "kind": "Pod",    
    "apiVersion": "v1",    
    "metadata": {    
        "name": "mypod"    
    },    
    "spec": {    
        "containers": [    
            {    
                "name": "mypod",    
                "image": "openshift/hello-openshift",    
                "env": [    
                    {    
                        "name": "ENV0",    
                        "value": "`cat bytes-1m-more.txt`"    
                    }    
                ]    
            }    
        ]    
    }    
}    
EOF

$ ls -lh bytes-1m-more.txt
-rw-rw-r--. 1 ... ... 1.1M  1月  2 16:55 bytes-1m-more.txt

$ oc create -f mypod.json
pod/mypod created

- verify json patch > 1M:

$ oc create deployment hello-openshift --image=openshift/hello-openshift
$ oc set env deployment hello-openshift ENV0=abc
$ echo -n '[{"op": "replace", "path": "/spec/template/spec/containers/0/env/0/value", "value": "' > patch.json
$ cat bytes-1m-more.txt >> patch.json
$ echo -n '"}]' >> patch.json

$ ls -lh bytes-1m-more.txt
-rw-rw-r--. 1 ... ... 1.1M  1月  2 16:55 bytes-1m-more.txt

$ OPENSHIFT_API_URL=`oc config view --minify -o jsonpath='{.clusters[*].cluster.server}'`
$ OPENSHIFT_API_TOKEN=`oc whoami -t`
$ DP_LINK=$(oc get deployment hello-openshift -o json | jq .metadata.selfLink | sed 's/\///' | sed 's/"//g')

$ curl -v -k -X PATCH "$OPENSHIFT_API_URL/$DP_LINK" -H "Content-Type: application/json-patch+json" -H "Authorization: Bearer $OPENSHIFT_API_TOKEN" -d @patch.json

[Expected Results:]

*   Trying 13.228.193.227:6443...
* TCP_NODELAY set
* Connected to ... openshift.com (13.228.193.227) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api....openshift.com
*  start date: Jan  3 01:29:29 2020 GMT
*  expire date: Feb  2 01:29:30 2020 GMT
*  issuer: OU=openshift; CN=kube-apiserver-lb-signer
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55caec389a00)
> PATCH /apis/apps/v1/namespaces/prj1/deployments/hello-openshift HTTP/2
> Host: api...openshift.com:6443
> User-Agent: curl/7.65.3
> Accept: */*
> Content-Type: application/json-patch+json
> Authorization: Bearer  ...
> Content-Length: 1058904
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2000)!
* We are completely uploaded and fine
< HTTP/2 200 
< audit-id: fa29877c-4317-4055-85ac-39c76627668c
< content-type: application/json
< date: Fri, 03 Jan 2020 08:17:06 GMT
< 
{
  "kind": "Deployment",
  "apiVersion": "apps/v1",
  "metadata": {
    "name": "hello-openshift",
    "namespace": "prj1",
    "selfLink": "/apis/apps/v1/namespaces/prj1/deployments/hello-openshift",
    "uid": "e71c3123-6aec-4d7a-8675-d3e3fe7e06ec",
    "resourceVersion": "119639",
    "generation": 5,
    "creationTimestamp": "2020-01-03T07:16:00Z",
    "labels": {
      "app": "hello-openshift"
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "4"
    }
  ...
      "spec": {
        "containers": [
          {
            "name": "hello-openshift",
            "image": "openshift/hello-openshift",
            "env": [
              {
                "name": "ENV0",
                "value": "aa ... "
              }
            ],
            "resources": {
              
            },
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {
          
        },
        "schedulerName": "default-scheduler"
      }
   ...

Comment 7 errata-xmlrpc 2020-05-04 11:16:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.