Bug 1032436

Summary:	Unable to list and add env variable any more after user try to add env variable with an invalid UTF-8 value by RESTAPI for app
Product:	OpenShift Online	Reporter:	Nan Wei <nwei>
Component:	Master	Assignee:	Lili Nader <lnader>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	dmcphers, mfisher, xtian
Target Milestone:	---	Keywords:	UpcomingRelease
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1034647 (view as bug list)		Environment:
Last Closed:	2014-01-24 03:33:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1034647

Description Nan Wei 2013-11-20 08:21:33 UTC

Description of problem:
User try to add env variable with an invalid UTF-8 value like "bar%B3" by RESTAPI, restapi retrun success, then try to list and add env var, it throws some error messages as follows: "Unable to complete the requested operation due to: invalid byte sequence in UTF-8". 
Version-Release number of selected component (if applicable):
devenv_4051
How reproducible:
100%
Steps to Reproduce:
1. The second time to add new env variable and its value is special characters by rest api
curl -k -H 'Accept: application/xml' --user nwei https://ec2-54-227-44-71.compute-1.amazonaws.com/broker/rest/domain/nweid/application/ptest/environment-variables -X POST -d name=foo1 -d value='bar%B3'
Enter host password for user 'nwei':
<message>
      <severity>info</severity>
      <text>Added environment variable 'foo1' to application ptest</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
    </message>
2. Check env variable output
[weinan@dhcp-65-25 test]$ rhc env list -a ptest
Unable to complete the requested operation due to: invalid byte sequence in
UTF-8.
Reference ID: 5cd86030999c4664a7b288302f65ad18
3.  Add one valid env variable by rhc
[weinan@dhcp-65-25 test]$ rhc env set name=value -a ptest 
Setting environment variable(s) ... 
Unable to complete the requested operation due to: invalid byte sequence in
UTF-8.
Reference ID: 8cc1d531ed9f6459fc3df38264243437
Actual results:
It throws some error message: "Unable to complete the requested operation due to: invalid byte sequence in UTF-8.
Reference ID: 2183148205576bb4af231690e8909bed" all the time.
Expected results:
User should not be allowed to add an invalid UTF value for a user env var, and it should not break the following list and add env var operation.
Additional info:

Comment 1 Lili Nader 2013-11-22 00:55:39 UTC

Here's the findings from the debugging

1. curl encodes the data (-d or --data) as ascii.  The ascii encoded version of "bar%B3" is "bar\xB3". FYI, it works fine if  --data-urlencode is used instead.

2. rails expects everything to be encoded in UTF-8 and "bar\xB3" contains an invalid UTF-8 byte sequence.

3. The value is passed on to node without ever calling any methods on it so rails/ruby does not barf until user tries to read back the value from node at which point "invalid byte sequence in UTF-8" is raised.

4. This issue it not unique to env vars.  All other APIs have the same issue although the exception is raised earlier it the process.

Finally the fix.  Check all inputs for valid UTF-8 encoding.  See commit 

https://github.com/lnader/origin-server/commit/845ce9eef98d7313602a50e1a40ecbd14a86906b

in pull request

https://github.com/openshift/origin-server/pull/4213

Comment 2 Lili Nader 2013-11-26 22:46:43 UTC

new pull request with additional checking for ruby 1.8.7 

https://github.com/openshift/origin-server/pull/4239

Comment 3 openshift-github-bot 2013-11-27 00:45:09 UTC

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/374793256163bdc93dcda88fa95f6afd08581ee7
Bug 1032436

Comment 4 Nan Wei 2013-11-28 05:33:11 UTC

Version-Release number of selected component (if applicable):
devenv_4080

[weinan@dhcp-65-25 Downloads]$ curl -k -H 'Accept: application/xml' --user "nwei:redhat" https://ec2-54-211-32-152.compute-1.amazonaws.com/broker/rest/domain/nweid/application/ptest/environment-variables -X POST -d name=foo1 -d value='bar%BBB3'
<message>
      <severity>error</severity>
      <text>Only valid UTF-8 encoded inputs are accepted</text>
      <exit-code nil="true"></exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
</message>
[weinan@dhcp-65-25 Downloads]$ rhc env list ptest
[weinan@dhcp-65-25 Downloads]$ curl -k -H 'Accept: application/xml' --user nwei https://ec2-54-211-32-152.compute-1.amazonaws.com/broker/rest/domain/nweid/application/ptest/environment-variables -X POST -d name=foo1 -d value='bar%3bbb'
<message>
      <severity>info</severity>
      <text>Added environment variable 'foo1' to application ptest</text>
      <exit-code>0</exit-code>
      <field nil="true"></field>
      <index nil="true"></index>
</message>