Bug 1625518 - Pipeline DSL doesn't prevent `oc` command execution from shell injection
Summary: Pipeline DSL doesn't prevent `oc` command execution from shell injection
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 3.11.0
Hardware: All
OS: All
high
high
Target Milestone: ---
: 4.1.0
Assignee: Gabe Montero
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-05 06:42 UTC by Yuxiang Zhu
Modified: 2019-06-04 10:40 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the jenkins durable task controller plugin bourne shell suport would mismanage special shell characters specificed as part of oc parameters included in openshift jenkins client plugin DSL Consequence: underlying oc invocations would end up being incorrect because the special characters would be incorrectly modified Fix: the openshift jenkins client plugin moved off of the durable task plugin bourne shell support and onto lower level jenkins computer/proc invocation support which would leave the data alone Result: openshift jenkins client plugin DSL can no function with the use of special shell characters
Clone Of:
Environment:
Last Closed: 2019-06-04 10:40:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:40:43 UTC

Description Yuxiang Zhu 2018-09-05 06:42:23 UTC
Description of problem:
When using OpenShift Pipeline DSL to interact with the OpenShift API server, special shell characters are not passed to `oc` but explained by Bash (on a Linux slave) or Command Prompt (on Windows slave).

We hit this issue when using Pipeline DSL to manage the CI/CD workloads for our projects. For example, if an option passed to a selector contains whitespaces or special shell character (like `#`, `;`, `$`), those characters will be handled by shell so that the `oc` process will not receive the option.

Furthermore, this also lead to a vulnerability: There are customers run pipeline jobs behind a self-service portal. Crackers can construct malicious strings to hack into the pipeline build and run arbitrary commands on the slave unless they are secured filtered.

Version-Release number of selected component (if applicable):
3.11

How reproducible:
I create a BuildConfig to demo this shell injection.


Steps to Reproduce:
1. Create a new app of ruby-hello-world:
oc new-app https://github.com/openshift/ruby-hello-world
2. Create this BuildConfig:

kind: "BuildConfig"
apiVersion: "v1"
metadata:
  name: "bug-shell-injection"
spec:
  strategy:
    jenkinsPipelineStrategy:
      jenkinsfile: |-
        def label = "mypod-${UUID.randomUUID().toString()}"
        podTemplate(cloud: 'openshift', label: label,
          containers: [containerTemplate(name: 'jnlp', image: 'docker.io/openshift/jenkins-slave-base-centos7:latest')],
        )
        {
          node(label) {
            stage('Options with Special Characters') {
                openshift.withCluster() {
                  try {
                    // because arguments contain special bash characters, the following step will fail.
                    openshift.set('env', 'dc/ruby-hello-world', 'VAR1=PR#123', 'VAR2=hello world')
                    echo 'Environment variables updated.' // will never run
                  } catch (Exception e) {
                    echo "Error running openshift.set"
                  }
                }
            }

            stage('Shell Injection') {
                openshift.withCluster() {
                  // ping will be run. Using `exit 1` to make the step Failed
                  try {
                    openshift.selector('po', '|| ping google.com -c 5 > out.txt ; echo {} | tee --').object()
                  } catch (Exception e) {
                    echo "Error running openshift.selector().object()"
                  }
                  pingResult = readFile 'out.txt'
                  echo pingResult
                }
            }
          }
        }

3. Start a new build:
oc start-build bug-shell-injection

Actual results:
stage('Options with Special Characters'): Cannot be finished because special characters are not correctly passed to `oc`.
stage('Shell Injection') is a malicious stage: It inject a command `ping google.com` to the pipeline build step. ping results can be found from the build log.

Expected results:
stage('Options with Special Characters') should print 'Environment variables updated.'.
stage('Shell Injection') should not run the malicious command `ping google.com`.

Additional info:
A possible fix is to run `oc` command using the `Launcher` Jenkins API rather than DurableTask. I've create a PR to show this solution: https://github.com/openshift/jenkins-client-plugin/pull/176

Comment 1 Gabe Montero 2018-09-14 19:25:23 UTC
The PR from  Yuxiang Zhu  has merged.  Thanks !!

For now, given the depth of the change, only targeting the v4.0 jenkins images.  As the change gains traction upstream, based on results, we can revisit backporting to older releases.

Comment 2 Gabe Montero 2018-09-14 21:14:14 UTC
PR https://github.com/openshift/jenkins/pull/697 is updating the centos image

Comment 3 Gabe Montero 2018-09-15 00:40:25 UTC
Job https://buildvm.openshift.eng.bos.redhat.com:8443/job/devex/job/devex%252Fjenkins-plugins/66/ has been submitted to update the 4.0 plugin rpm with v1.0.17 of the client plugin.

Moving to post ... I don't think 4.0 images are showing up in brew just yet.  Between checking periodically and monitoring the various mailing lists, I'll move to QA once a RHEL image is ready to test.

The centos image should be available in a few hours.

Comment 4 Gabe Montero 2018-11-26 22:30:21 UTC
Forgot to move this out of post ... the client plugin in the 4 .0 images is well past 1.0.17 now 

moving to ON QA

Comment 5 wewang 2018-11-28 09:55:26 UTC
Hi Gabe, Tested in client plugin v1.0.21, met another issue:default sa cannot get dc in the same project, not sure if authentication issue, or I need add steps for it. 

steps:
1. Create a new app of ruby-hello-world:
   $oc new-app https://github.com/openshift/ruby-hello-world

2. Create this BuildConfig:bug-shell-injection
 Updated  openshift.set('env', 'dc/ruby-hello-world', 'VAR1=PR#123', 'VAR2=hello world') to openshift.set('env', 'dc/ruby-hello-world', 'VAR1=PR#123'), seems oc set env did not support first format.

4. Create jenkins instance
 $oc new-app jenkins-persistent

3. Start a new pipeline build:
 $oc start-build bug-shell-injection

4.Check Logs: https://url.corp.redhat.com/d6d6034

Comment 6 Gabe Montero 2018-12-03 21:19:09 UTC
Hey Wen Wang,

Sorry for the delay in getting to this.  The thanksgiving holiday and 4.0 are my excuses.

Unfortunately, my delay resulted in https://url.corp.redhat.com/d6d6034 no longer being available.


That said, if the sa cannot get to the dc, that is a config / setup issue.

A couple of points:

1) You said "default sa" ... the jenkins templates like jenkins-persistent create a "jenkins" service account that has the permissions needed

Did you change something so the jenkins pod would not use the jenkins SA?

2) were you using a 4.0 openshift ? ... I only ask because we removed the auto provisioning in 4.0, and if you were using 4.0, creating the build config before calling new-app on jenkins-persistent is OK.  If you tried on 3.11, I would tell you to new-app jenkins-persistent *BEFORE* creating the pipeline build config

3) could you provide the complete build config yaml for the bug-shell-injection build config ? ... at worst, I can try to reproduce myself if need be

thanks

Comment 7 wewang 2018-12-04 03:20:32 UTC
I didn't change it, yes I use 4.0 openshift(v4.0.0-0.81.0) here's log:https://url.corp.redhat.com/5600f40, seems role and rolebinding related issue, tested in 3.11, did not meet the problem.

Comment 8 Gabe Montero 2018-12-04 14:53:39 UTC
I still get ` Could not getting paste data: Paste does not exist, has expired or has been deleted.` when I try to look at your log URL.

But I'll try to reproduce on 4.0 myself and we can go from there.

Comment 10 Gabe Montero 2018-12-05 20:17:28 UTC
OK I think I figured it out.

"system:serviceaccount:jenkins2:default" is incorrect

it should be using "system:serviceaccount:jenkins2:jenkins" when provisioning via our jenkins templates.

For our maven and nodejs sample pod templates, we set up the serviceAccount to be the "jenkins" SA during startup.

See https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/kube-slave-common.sh#L81-L158

But I know realized after you re-reading your description that you are defining your own pod template using the base slave image:

podTemplate(cloud: 'openshift', label: label,
          containers: [containerTemplate(name: 'jnlp', image: 'docker.io/openshift/jenkins-slave-base-centos7:latest')],
        )

On your 3.11 cluster, either you are using a different pod template, or you have authorized the default SA in the namespace on that cluster to have access in that namespace, etc.

To really understand, you should pull the pod template definition from  the working 3.11 jenkins instance and compare with the pod template definition from the 4.0 jenkins instance.

But that is secondary to moving forward with verifying this fix.

You need to either switch to one of our sample pod templates, or you need to augment the pod template you are defining to fill in all the fields we fill in with https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/kube-slave-common.sh#L81-L158

Do one of those options and try again please.

Comment 11 wewang 2018-12-06 06:40:44 UTC
Thanks so much Gabe, you always understood qe's confusion. I add a step: 
oc policy add-role-to-user edit system:serviceaccount:wen:default for my  Comment5, and it works now, the bug verfied. also thanks Yuxiang Zhu  

here's log:
Running on jenkins-slave-spsm8-vzq99 in /home/jenkins/workspace/wen/wen-bug-shell-injection
[Pipeline] {

[Pipeline] stage
[Pipeline] { (Options with Special Characters)

[Pipeline] echo

[Pipeline] _OcContextInit

[Pipeline] readFile

[Pipeline] readFile
[Pipeline] _OcAction

[Pipeline] echo
Environment variables updated.   //expected result,hooray!

[Pipeline] }
[Pipeline] // stage

[Pipeline] stage
[Pipeline] { (Shell Injection)
[Pipeline] _OcContextInit
[Pipeline] readFile
[Pipeline] readFile
[Pipeline] _OcAction

Error: unknown shorthand flag: 'c' in -c

Comment 12 Gabe Montero 2018-12-06 14:45:12 UTC
glad it is sorted out Wen!!

Comment 15 errata-xmlrpc 2019-06-04 10:40:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.