Description of problem: User try to add env variable with an invalid UTF-8 value like "bar%B3" by RESTAPI, restapi retrun success, then try to list and add env var, it throws some error messages as follows: "Unable to complete the requested operation due to: invalid byte sequence in UTF-8". Version-Release number of selected component (if applicable): devenv_4051 How reproducible: 100% Steps to Reproduce: 1. The second time to add new env variable and its value is special characters by rest api curl -k -H 'Accept: application/xml' --user nwei https://ec2-54-227-44-71.compute-1.amazonaws.com/broker/rest/domain/nweid/application/ptest/environment-variables -X POST -d name=foo1 -d value='bar%B3' Enter host password for user 'nwei': <message> <severity>info</severity> <text>Added environment variable 'foo1' to application ptest</text> <exit-code>0</exit-code> <field nil="true"></field> <index nil="true"></index> </message> 2. Check env variable output [weinan@dhcp-65-25 test]$ rhc env list -a ptest Unable to complete the requested operation due to: invalid byte sequence in UTF-8. Reference ID: 5cd86030999c4664a7b288302f65ad18 3. Add one valid env variable by rhc [weinan@dhcp-65-25 test]$ rhc env set name=value -a ptest Setting environment variable(s) ... Unable to complete the requested operation due to: invalid byte sequence in UTF-8. Reference ID: 8cc1d531ed9f6459fc3df38264243437 Actual results: It throws some error message: "Unable to complete the requested operation due to: invalid byte sequence in UTF-8. Reference ID: 2183148205576bb4af231690e8909bed" all the time. Expected results: User should not be allowed to add an invalid UTF value for a user env var, and it should not break the following list and add env var operation. Additional info:
Here's the findings from the debugging 1. curl encodes the data (-d or --data) as ascii. The ascii encoded version of "bar%B3" is "bar\xB3". FYI, it works fine if --data-urlencode is used instead. 2. rails expects everything to be encoded in UTF-8 and "bar\xB3" contains an invalid UTF-8 byte sequence. 3. The value is passed on to node without ever calling any methods on it so rails/ruby does not barf until user tries to read back the value from node at which point "invalid byte sequence in UTF-8" is raised. 4. This issue it not unique to env vars. All other APIs have the same issue although the exception is raised earlier it the process. Finally the fix. Check all inputs for valid UTF-8 encoding. See commit https://github.com/lnader/origin-server/commit/845ce9eef98d7313602a50e1a40ecbd14a86906b in pull request https://github.com/openshift/origin-server/pull/4213
new pull request with additional checking for ruby 1.8.7 https://github.com/openshift/origin-server/pull/4239
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/374793256163bdc93dcda88fa95f6afd08581ee7 Bug 1032436
Version-Release number of selected component (if applicable): devenv_4080 [weinan@dhcp-65-25 Downloads]$ curl -k -H 'Accept: application/xml' --user "nwei:redhat" https://ec2-54-211-32-152.compute-1.amazonaws.com/broker/rest/domain/nweid/application/ptest/environment-variables -X POST -d name=foo1 -d value='bar%BBB3' <message> <severity>error</severity> <text>Only valid UTF-8 encoded inputs are accepted</text> <exit-code nil="true"></exit-code> <field nil="true"></field> <index nil="true"></index> </message> [weinan@dhcp-65-25 Downloads]$ rhc env list ptest [weinan@dhcp-65-25 Downloads]$ curl -k -H 'Accept: application/xml' --user nwei https://ec2-54-211-32-152.compute-1.amazonaws.com/broker/rest/domain/nweid/application/ptest/environment-variables -X POST -d name=foo1 -d value='bar%3bbb' <message> <severity>info</severity> <text>Added environment variable 'foo1' to application ptest</text> <exit-code>0</exit-code> <field nil="true"></field> <index nil="true"></index> </message>