Bug 1748061

Parameters are meant to process reasonably sized items in memory. In your test the script is generating 50000 random characters that are base64 encoded, more than likely hitting the default limit of 64kB of data to read in.

Please note that using a ConfigMap or Secret has its limits - etcd only supports objects less than 1MB in size. Any Secret or ConfigMap therefore must have less than 1MB of data - anything larger and the user needs to create a Persistent Volume.

Hi Adam,

thank you for your feedback. I'd like to re-open the discussion on the behaviour with a bit more context.

The real topic of this Bugzilla is to investigate a mismatch between the `--param-file` and `--param` options for the `oc process` command.

Let's assume that [1] is a long TLS certificate chain (base64 encoded) saved in a filed called "PARAMS_b64" which shall be included into the following `ConfigMap`:

~~~ 
$ cat configmap-template.yaml
apiVersion: v1
kind: Template
metadata:
  name: security-template
  annotations:
    description: "Description"
objects:
  - metadata:
      name: configmap-test
    apiVersion: v1
    data:
    caTLSRootCert: ${TestCert} 
    kind: ConfigMap
parameters:
  - name: TestCert
    require: true
~~~

Now, the following command does not work:

~~~
$ oc process -f configmap-template.yaml --param-file=PARAMS_b64
error: Cannot read variables from file "PARAMS_b64": bufio.Scanner: token too long
~~~

However, this does work:

~~~
$ oc process -f configmap-template.yaml --param=$(cat PARAMS_b64) | oc apply -f -
configmap/configmap-test configured
~~~

This behaviour is true for OpenShift Container Platform version 3.11 and 4.2 (potentially all others as well).

Please can you advise on whether or not this is working as intended in your opinion? We would like to explore if this behaviour should be adjusted (at the very least in OpenShift 4.x).

Thank you very much and kind regards,
Christian

---

[1] - https://pastebin.com/raw/pz08tSMJ

Thank you, Christian, for your feedback and reproducer. I think this does warrant further investigation since `--param` and `--param-file` inputs should be able to be processed equally.

Setting target to 4.4.0 given the priority and severity.

Did some quick research in the code.  Ultimately, this is currently a function of the fact that the newapp helpers in `oc` are using "github.com/joho/godotenv"

First, see https://github.com/openshift/oc/blob/master/pkg/helpers/newapp/app/env.go#L164 and the `gotdotenv.Read(filename)` call 

gotdotenv is using the default bufio.NewScanner which means a max size of 64K ... it does not provide a means to override that.

To fix this we would need to move off of gotdotenv in newapp.  Not overly onerous, but not a couple of lines of code neither (the current gotdotenv path is 50 to 100 lines).  
Plus regression testing for all the paths using these newapp helpers.

Given Christians pretty easy workaround wrt --param=$(cat PARAMS_b64), and the subsequent low severity of this, and where we are with templates and oc new-app more and more being in maintenance mode,
I would content we close this as WONTFIX.

Thoughts gentlemen?

Thanks for your work on this Gabe. 

The presented "work around" to demonstrate this bug / inconsistency might not be applicable in any case.

If possible, could you please allow me some time to discuss this internally before closing this again as WONTFIX?

Thanks and kind regards,
Christian

Sure we can sit tight for a bit Christian ... and oops on the misdirected needsinfo :-)

Conversely waiting on takes from Adam, et. al. on our end wrt should we spend time on this given there is an open customer case with this, regardless of priority

wrt the cost of fixing this ... I've got a prototype PR up of what that might look like:  https://github.com/openshift/oc/pull/177

Hi Gabe,

thanks for your work on the PR!

As mentioned before, I'm working to give you a bit more information on why a fix makes sense from the customer perspective. Please stay tuned for more to come from that front.

From my point of view, making the behaviour of `--param-file` and `param` consistent would improve our product regardless of any specific use case (simply to improve the developer experience, which is crucial).

Kind regards,
Christian

#! /bin/bash

for l in $(seq 20000 10000 300000); do
    echo -n "BLA=" > PARAMS
    tr -cd '[:alnum:]' < /dev/urandom | head -c $l | base64 -w 0 >> PARAMS

    cat << EOF | 4.4/oc process -f - --param-file=PARAMS --local >/dev/null && echo "OK $l" || echo "Fail: $l" 
apiVersion: v1
kind: Template
parameters:
  - name: BLA
objects:
- apiVersion: v1
  kind: Secret
  metadata:
    name: BLAtest
  data:
    password: \${BLA}
  type: Opaque
EOF
done

The result:
OK 20000
OK 30000
OK 40000
OK 50000
OK 60000
OK 70000
OK 80000
OK 90000
OK 100000
OK 110000
OK 120000
OK 130000
OK 140000
OK 150000

Could create resource with large Input Parameters without error.
$ oc version 
Client Version: 4.4.0-0.nightly-2019-12-14-103510
Server Version: 4.4.0-0.nightly-2019-12-14-103510
Kubernetes Version: v1.16.2

Also tried more biger Input Parameters 3000000, create succeed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581