Bug 1088956

The issue is reproducible also on upstream testsuite on Branch_2_3, setting up LC_ALL and LANG enviroment variables to some cp* encoding and running the testsuite with command:

mvn clean verify -pl :resteasy-jaxb-provider,:resteasy-jaxrs,:resteasy-test-arquillian -DfailIfNoTests=false -Darquillian -Dtest=TestNamespace -Djboss.home=path_to_eap_installation

I found RFC 2616 [1] from which I understand that encoding specified in request should be followed.

[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.4.1

Hi Katerina, could you please report it on JIRA for this bug? Or the RESTEasy development team won't fix it.

Hi Weinan, according to our instructions, bugzilla is the only tool for tracking EAP issues. If you need any additional tickets for the development team, kindly open it yourself.

Hi Ron, could you please help to check this issue? Seems it's a development relative bug.

Hey Wei,

I almost missed this issue. It went to a Thunderbird folder I rarely look at.

I'll take a look. Fortunately, I have a working Windows laptop. I guess we'll have to test on Android as well, some day. ;)

-Ron

This also works:

      @POST
      @Path("text/accepts")
      @Consumes("text/plain")
      public String textAccepts(String movie)
      {
         return movie;
      }

      ...

      ClientRequest request = new ClientRequest(generateURL("/text/accepts"));
      Map<String, String> params = new HashMap<String, String>();
      params.put("charset", "UTF-16");
      MediaType mt = new MediaType("text", "plain", params);
      request.body(mt, "La Règle du Jeu");
      request.accept(mt);
      ClientResponse<?> response = request.post();

Note that the JAX-RS 2.0 spec says "When writing responses, implementations SHOULD respect application-supplied character set metadata and SHOULD use UTF-8 if a character set is not specified by the application or if the application specifies a character set that is unsupported."

So, I think that

      ClientRequest request = new ClientRequest(generateURL("/text/accepts"));
      Map<String, String> params = new HashMap<String, String>();
      params.put("charset", "UTF-16");
      MediaType mt = new MediaType("text", "plain", params);
      request.body(mt, "La Règle du Jeu");
//      request.accept(mt);
      ClientResponse<?> response = request.post();

should result in the result being encoded in UTF-8 rather than UTF-16.

Actually, I think that Katerina's comment "I think, that if request has specified encoding in the body it should be recognized by the server and response should be returned in the same encoding." makes sense, and might be worth adding to the spec. But, for now, I don't think the spec supports the idea.

Comments?

Hi Ron,

The JAX-RS 2.0 spec also says that keywords like SHOULD etc. are to be interpreted as described in RFC 2119. 
RFC 2119 says:
"SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course."

So I think the jax-rs spec actually supports the idea. What do you think?

Good point, Katerina. Let's see what Bill Burke says.

Set the Target Release to TBD EAP 6  so that this bug appears in Release Notes dashboards and can be included as required.

Ron, Katerina, could someone please supply some information about this issue for inclusion in the 6.3.0 Release Notes (Known Issues).

Hi Scott, a brief summary:

When encoding is specified in the body of the client request, Resteasy returns response in encoding of the server, not in encoding of original request. To receive response in specified encoding request.accept(mediaType) header must be specified or @Produces annotation for the resource can be used.

Thanks Katerina.

I've added a release note text and marked this for inclusion in the 6.3.0 Release Notes. 

I've also changed the Target Release to 6.4.0 to ensure this gets picked up in our Release Note filters. Feel free to return it to TBD 6 after the 6.3.0 GA.

(Also clearing NEEDINFO)

Hi Katerina,

I'm having a reversal of thinking on this issue. As you pointed out, the client isn't saying anything to the server about charsets. The JAXB implementation is guessing the charset, and, in Windows, it's guessing wrong. As I mentioned in Comment 6, everything works fine if the client tells the server which charset it's sending and which charset it expects in return.

I'm thinking that it's reasonable to expect the client to take that responsibility, and that it would be sufficient for me to just fix the tests.

What do you think?

-Ron

Hi Ron,

I looked at RESTEASY-1066 and Stuart is suggesting there:

"Looking at the Resteasy DefaultTextPlain provider if no charset is provided then it just uses String.getBytes() which is problematic because this is platform dependent, and may provide something completely bogus in some locales. I think it should either use String.getBytes("ISO-8859-1") to follow the defaults recommended by the HTTP spec, or use UTF-8 (and set the charset in the Content-Type header) and follow the JAX-RS spec recommendation."

Ok, determining encoding using charset of the entity seems going beyond the spec. 
(I thought we could extract Media type from request body request.getBodyContentType(); And give it to the server.)

But we should return consistent results on all platforms. Therefore without charset it should return ISO-8859-1 or UTF-8 as Stuart suggest. Sounds reasonable?

Great point, Katerina. I forgot about what Stuart said on RESTEASY-1056 when I wrote my last comment. In fact, I've noticed that the Resteasy text oriented providers, including the JAXB providers, are inconsistent with respect to charsets. I'll sort that out, make sure UTF-8 is the default, and fix TestNamespace.

Meant RESTEASY-1066.

I have modified the DefaultTextPlain and StringTextStar in resteasy-jaxrs, as well as providers/jaxb, so that responses will default to using UTF-8.

The changes to Branch_2_3 are covered by pull request https://github.com/resteasy/Resteasy/pull/551.

Pull request #551 merged, RESTEASY-1066 closed.

Verified in EAP 6.4.0.DR11.