opened Jasper case 00060857

Yaniv per Jasper response we could:
- create a different validator (same as now less the problematic character)
- remove validator all together
- change label ?

If you think we can force jasper to deliver a host fix ... we can wait.
Anyway if we can't than this should be a KB only solution, as we do not want to relax security for all customers.

Thoughts ?

I see that everywhere that there are Parentheses () in simplified Chinese in the ad hoc domains the domain gets broken.

For example, 某个时刻的主机已使用的最大交换空间（MB）.

The way the validator interprets the Parentheses is braking the domain.

Do we want to go through all the domains and change the translation so it will not include them ?

By the way, the original English text does not include parentheses.

For example, the text for the above in English is 
HOST.MAX_SWAP_USED_MB.DESCR=The host's max swap used in MB at the history datetime

I think this will be the best solution.

(Ccing Tony)

Would it be possible to modify that in code instead of translations?
In both Chinese and Japanese, we usually add the unit in brackets at the end of the string. It is clear for Chinese/Japanese users.

In fact, I just realized that brackets used in zh-CN translations are double-bit.

Japanese translation for the same string did not break the domain and it uses single-bit brackets:

履歴日時におけるホストの Swap 最大使用量 (MB 単位)

(In reply to Yuko Katabami from comment #5)
> In fact, I just realized that brackets used in zh-CN translations are
> double-bit.
> 
> Japanese translation for the same string did not break the domain and it
> uses single-bit brackets:
> 
> 履歴日時におけるホストの Swap 最大使用量 (MB 単位)

I can change the zh-CN translation if it is necessary.

Thank you Tony.

@Shirly,

Would it be possible to fix this in the code or editing translations is the only solution?

Yaniv, What do you think? It will mean we need to change the validator which is part of the security.

(In reply to Shirly Radco from comment #8)
> Yaniv, What do you think? It will mean we need to change the validator which
> is part of the security.

Jasper need to provide a validator they consider secure and can support.
They need to also commit to fix using the validator in following releases.

Jasper's response:

"Regarding the parenthesis problem, are you sure your entire chain of requests is URIEncoded for UTF-8?

For example, if using Tomcat, your connector inside server.xml should specifying URIEncoding="UTF-8" like:

    <Connector port="8070" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               URIEncoding="UTF-8"
               redirectPort="8443" />

Without the above setting I did the following which created a security validation error.  During login I chose zh_CN Chinese (china) I created an ad hoc, saved the ad hoc, and I gave it a name consisting of chinese characters as well as the ascii  parenthesis characters.  See attached screenshot, bottom-most ad hoc view.  

Clicking on the view at the bottom of the screenshot results in the 5321 error.

But adding the URIEncoding of UTF-8 resolves this problem. 

If you are using https or a load-balancer, or http proxy, you'll need to ensure the URIEncoding is set everywhere.  

Can you please confirm your URIEncoding, and then compare your repro steps with mine?"

Didi, do you understand where we need to do this change?
Can you help test this solution?

A quick search found [1]. Based on this, pushed patch 43220.

Not sure how to test. Do we have a reproducer? A system already showing this?

[1] https://developer.jboss.org/blogs/mukeshghildiyal/2012/01/09/uriencoding-in-jboss7

(In reply to Yedidyah Bar David from comment #11)
> A quick search found [1]. Based on this, pushed patch 43220.
> 
> Not sure how to test. Do we have a reproducer? A system already showing this?
> 
> [1]
> https://developer.jboss.org/blogs/mukeshghildiyal/2012/01/09/uriencoding-in-
> jboss7

I'll test it.

Jasper's response:
"Hi All,
I reproduced the problem with the parenthesis in JBoss AS 7.1.1, and resolved it via utf-8 settings.

I tried two things simultaneously, so I'm not sure which was the resolution:

a) Added the following to startup scripts:
JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=utf-8"

b) set utf-8 system properties within /jboss-as-7.1.1.Final/standalone/configuration/standalone.xml

<system-properties>
     <property name="org.apache.catalina.connector.URI_ENCODING" value="UTF-8"/>
     <property name="org.apache.catalina.connector.USE_BODY_ENCODING_FOR_QUERY_STRING" value="true"/>
</system-properties>

I understand you are using EAP, perhaps you can find the equivalent settings in EAP?

My test matched the steps I used for the uploaded screenshot from June 30th.

Thanks,
Greg"


Can we update the patch please?

(In reply to Yaniv Dary from comment #13)
> Jasper's response:
> "Hi All,
> I reproduced the problem with the parenthesis in JBoss AS 7.1.1, and
> resolved it via utf-8 settings.
> 
> I tried two things simultaneously, so I'm not sure which was the resolution:
> 
> a) Added the following to startup scripts:
> JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=utf-8"

Above is the only part missing in https://gerrit.ovirt.org/43220
right?

> 
> b) set utf-8 system properties within
> /jboss-as-7.1.1.Final/standalone/configuration/standalone.xml
> 
> <system-properties>
>      <property name="org.apache.catalina.connector.URI_ENCODING"
> value="UTF-8"/>
>      <property
> name="org.apache.catalina.connector.USE_BODY_ENCODING_FOR_QUERY_STRING"
> value="true"/>
> </system-properties>
> 
> I understand you are using EAP, perhaps you can find the equivalent settings
> in EAP?
> 
> My test matched the steps I used for the uploaded screenshot from June 30th.
> 
> Thanks,
> Greg"
> 
> 
> Can we update the patch please?

patch updated, can you help verifying it?

(In reply to Sandro Bonazzola from comment #15)
> patch updated, can you help verifying it?

Will do, thanks!

Tested didn't work. Updated the case.

(In reply to Yaniv Dary from comment #17)
> Tested didn't work. Updated the case.

Jasper have let us know that they can use Chinese without issues. The patch doesn't seem to work and set the encoding correctly. Can you check why?

Juan - can you please have a look? Thanks.

Yaniv, perhaps for some reason they do need also JAVA_OPTS? Updated the patch. Can you try again?

I wasn't able to reproduce the issue, but looking at the description it looks like the parenthesis in that input text are not regular parenthesis (code points 40 and 21) but "full width parenthesis" (code points 65288 and 65289, hex ff08 and ff09). If the regular expression that appears in the log is used with the standard Java implementation it won't accept those characters. I tested that with the following Java program:

---8<---
import java.util.regex.*;

public class Test {
    public static void main(String[] args) {
        // There are many kinds of parenthesis in Unicode, including the
        // regular ones and the full width ones (code points 65288 and 65289):
        String regularParenthesis = "()"; // Unicode 40 and 41
        String fullWidthParenthesis = "（）"; // Unicode 65288 and 65289

        // The pattern matches regular parenthensis only:
        Pattern pattern = Pattern.compile("^[\u0080\u0099\u2019\\p{L}\\p{M}\\p{N}\\p{Pd}\\p{Pi}\\p{Pf}\\p{Pc}\\p{Po}\\p{Sc}\\p{So}\\p{Sm}\\p{Cf}\\p{Z}\\s\\=\\(\\)\\[\\]\\~\\|\\{\\}\\`\\^\\+&&[^<>]]*$");
        Matcher regularMatcher = pattern.matcher(regularParenthesis);
        Matcher fullWidthMatcher = pattern.matcher(fullWidthParenthesis);

        // So this succeeds:
        System.out.println("regular: " + regularMatcher.matches());
        System.out.println("fullWidth: " + fullWidthMatcher.matches());
    }
}
--->8---

Compiled it with "javac Test.java" and run it with "java Test" and the result, as expected, is this:

  regular: true
  fullWidth: false

To accept those two characters I added them to the regular expression:

  Pattern.compile("^[\uff08\uff09...");

Then the characters are accepted:

  regular: true
  fullWidth: true

(In reply to Juan Hernández from comment #21)
> I wasn't able to reproduce the issue, but looking at the description it
> looks like the parenthesis in that input text are not regular parenthesis
> (code points 40 and 21) but "full width parenthesis" (code points 65288 and
> 65289, hex ff08 and ff09). If the regular expression that appears in the log
> is used with the standard Java implementation it won't accept those
> characters. I tested that with the following Java program:
[snip]

Juan, thanks for the analysis, but as far as my limited understanding goes,
what you suggest is requiring a patch from jasper (comment 9), and when we
asked for this, they said that's not needed, and suggested (comment 13) a
change in configuration. What they suggested seems to me to have been
obsoleted [1], and based on that I tried the patch [2] that Yaniv said that
still does not work. Any other suggestion?

Yaniv - it might be simpler to give up and change the translations to use
single-bit parens, if that's not too problematic. Or ask jasper again,
saying their suggestion does not seem to work for us.

[1] https://developer.jboss.org/blogs/mukeshghildiyal/2012/01/09/uriencoding-in-jboss7
[2] https://gerrit.ovirt.org/43220

All what I can say is that the regular expression that Jasper is currently using doesn't accept full width parenthesis. If we want to accept them then that regular expression needs to be changed, either by Jasper or by us.

Note that you may be able to change it modifying the file "WEB-INF/classes/esapi/validation.properties", seems to be the "Validator.AlphaNumPunctuation" property.

What sed command can I to replace "full width parenthesis" to regular parenthesis and resolve this issue?

(In reply to Yaniv Dary from comment #24)
> What sed command can I to replace "full width parenthesis" to regular
> parenthesis and resolve this issue?

sed does not support unicode.

You can do this with python, something like:

re.sub(u'\uff08', '(', line))

Created attachment 1095436 [details]
wide-to-narrow-parens.py

A script to translate wide parenthesis to regular ones.

Usage:

path/to/wide-to-narrow-parens.py inp1 inp2 ...

It overwrites the files inp1 inp2 etc.

Shirly says that this didn't work, because the files we deal with aren't UTF-8 but java properties files. Perhaps this will work:

sed -i -e 's/\\uFF08/(/g' -e 's/\\uFF09/)/g'

Shirly, can you attach one of those properties files to the bug?

Created attachment 1095884 [details]
Example of simplified Chinese localization file

If what you need is to transform that kind of file into something that doesn't contain full width parenthesis then the solution proposed by Didi in comment 27 will work.

I'd suggest to add the "i" option to the sed commands, to ignore the case of the hex digits:

  sed -i -e 's/\\uFF08/(/gi' -e 's/\\uFF09/)/gi'

Created attachment 1096123 [details]
Fix script

If, for whatever the reason, you happen to have files that don't use the "\uxxxx" escape sequences but Unicode characters, then the solution in comment 27 won't work, and you will need a tool that supports Unicode. You could use the attached script, which in runs a Java program to do the fix. But I won't recommend it if you can solve the problem with the solution proposed by Didi.

Shouldn't I use 

sed -i -e 's/\\uFF08/u0028/gi' -e 's/\\uFF09/u0029/gi' ?

(In reply to Shirly Radco from comment #32)
> Shouldn't I use 
> 
> sed -i -e 's/\\uFF08/u0028/gi' -e 's/\\uFF09/u0029/gi' ?

I guess you can, not sure you must. If you want to, you need to escape the other 'u's too:

sed -i -e 's/\\uFF08/\\u0028/gi' -e 's/\\uFF09/\\u0029/gi'

verified in rhevm-reports-3.6.1.1-1.el6ev.noarch

Shirly, can you take over? Not sure if this bug requires doc text and/or what to write there. Thanks.

We have a customer who might be hitting this but I want to verify this before tagging his issue to the BZ. Any assistance would be greatly appreciated. Below are the errors they are seeing when creating ad-hoc reports:
~~~
# vim /var/log/ovirt-engine-reports/jasperserver.log
2016-01-12 15:28:15,069 ERROR AdhocAjaxController,ajp-/127.0.0.1:8712-13:888 - ad hoc controller exception    ---------->trying to create "ad hoc" view but failed
java.lang.RuntimeException: exception initting query
        at com.jaspersoft.ji.adhoc.strategy.DataStrategyUnroller.unrollDDS(DataStrategyUnroller.java:60)
        at com.jaspersoft.ji.adhoc.strategy.DataStrategyUnroller.wrap(DataStrategyUnroller.java:31)
        at com.jaspersoft.ji.adhoc.strategy.DataStrategyServiceImpl.getDataStrategy(DataStrategyServiceImpl.java:78)
        at com.jaspersoft.ji.adhoc.strategy.DataStrategyServiceImpl.getDataStrategy(DataStrategyServiceImpl.java:27)
        at com.jaspersoft.ji.adhoc.service.AdhocEngineServiceImpl.getDataStrategy(AdhocEngineServiceImpl.java:396)
...
Caused by: java.lang.IllegalArgumentException: resource id: vm_latest_with_del.created_by_user_name does not exist -------->something wrong with "vm_latest_..."?
        at com.jaspersoft.ji.adhoc.strategy.DomainDataStrategy.resolveDataSet(DomainDataStrategy.java:260)
        at com.jaspersoft.ji.adhoc.strategy.DomainDataStrategy.getBaseQueryDataSet(DomainDataStrategy.java:157)
        at com.jaspersoft.ji.adhoc.strategy.CommonDomainDataStrategy.createQuery(CommonDomainDataStrategy.java:757)
        at com.jaspersoft.ji.adhoc.strategy.DataStrategyUnroller.unrollDDS(DataStrategyUnroller.java:58)
        ... 130 more

~~~

- There are a lot of other error messages in /var/log/ovirt-engine-reports/server.log:
~~~
2016-01-12 10:11:49,791 INFO  [stdout] (ajp-/127.0.0.1:8712-1) 2016-01-12 10:11:49,782 ERROR Validator,ajp-/127.0.0.1:8712-1:408 - [SECURITY FAILURE context=label-ViewRepositoryEdit_context, key=label, type(AlphaNumPunctuation)=^[\u0080\u0099\u2019\p{L}\p{M}\p{N}\p{Pd}\p{Pi}\p{Pf}\p{Pc}\p{Po}\p{Sc}\p{So}\p{Sm}\p{Cf}\p{Z}\s\=\(\)\[\]\~\|\{\}\`\^\+&&[^<>]]*$, input=1æ<U+009C><U+0088>使ç<U+0094>¨æ<U+0083><U+0085>å<U+0086>µ, maxLength=5000, isBlacklist=false]
2016-01-12 15:28:15,078 INFO  [stdout] (ajp-/127.0.0.1:8712-13) 2016-01-12 15:28:15,069 ERROR AdhocAjaxController,ajp-/127.0.0.1:8712-13:888 - ad hoc controller exception
2016-01-12 15:33:56,864 INFO  [stdout] (ajp-/127.0.0.1:8712-10) 2016-01-12 15:33:56,855 ERROR AdhocAjaxController,ajp-/127.0.0.1:8712-10:888 - ad hoc controller exception
2016-01-12 15:42:27,583 INFO  [stdout] (ajp-/127.0.0.1:8712-9) 2016-01-12 15:42:27,575 ERROR ReportGeneratorController,ajp-/127.0.0.1:8712-9:96 - report generator controller exception: 执行上一个请求时出错。
2016-01-12 15:42:27,630 INFO  [stdout] (ajp-/127.0.0.1:8712-9) 2016-01-12 15:42:27,623 ERROR ReportGeneratorController,ajp-/127.0.0.1:8712-9:641 - Original exception overridden by exception handling failure

From these log entries do you believe they are seeing the same issue as documented in this BZ?

Regards,

Frank

(In reply to Frank DeLorey from comment #37)
> We have a customer who might be hitting this but I want to verify this
> before tagging his issue to the BZ. Any assistance would be greatly
> appreciated. Below are the errors they are seeing when creating ad-hoc
> reports:
> ~~~
> # vim /var/log/ovirt-engine-reports/jasperserver.log
> 2016-01-12 15:28:15,069 ERROR AdhocAjaxController,ajp-/127.0.0.1:8712-13:888
> - ad hoc controller exception    ---------->trying to create "ad hoc" view
> but failed
> java.lang.RuntimeException: exception initting query
>         at
> com.jaspersoft.ji.adhoc.strategy.DataStrategyUnroller.
> unrollDDS(DataStrategyUnroller.java:60)
>         at
> com.jaspersoft.ji.adhoc.strategy.DataStrategyUnroller.
> wrap(DataStrategyUnroller.java:31)
>         at
> com.jaspersoft.ji.adhoc.strategy.DataStrategyServiceImpl.
> getDataStrategy(DataStrategyServiceImpl.java:78)
>         at
> com.jaspersoft.ji.adhoc.strategy.DataStrategyServiceImpl.
> getDataStrategy(DataStrategyServiceImpl.java:27)
>         at
> com.jaspersoft.ji.adhoc.service.AdhocEngineServiceImpl.
> getDataStrategy(AdhocEngineServiceImpl.java:396)
> ...
> Caused by: java.lang.IllegalArgumentException: resource id:
> vm_latest_with_del.created_by_user_name does not exist -------->something
> wrong with "vm_latest_..."?
>         at

This is not related. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1202109
and 
https://bugzilla.redhat.com/show_bug.cgi?id=1207235


> com.jaspersoft.ji.adhoc.strategy.DomainDataStrategy.
> resolveDataSet(DomainDataStrategy.java:260)
>         at
> com.jaspersoft.ji.adhoc.strategy.DomainDataStrategy.
> getBaseQueryDataSet(DomainDataStrategy.java:157)
>         at
> com.jaspersoft.ji.adhoc.strategy.CommonDomainDataStrategy.
> createQuery(CommonDomainDataStrategy.java:757)
>         at
> com.jaspersoft.ji.adhoc.strategy.DataStrategyUnroller.
> unrollDDS(DataStrategyUnroller.java:58)
>         ... 130 more
> 
> ~~~
> 
> - There are a lot of other error messages in
> /var/log/ovirt-engine-reports/server.log:
> ~~~
> 2016-01-12 10:11:49,791 INFO  [stdout] (ajp-/127.0.0.1:8712-1) 2016-01-12
> 10:11:49,782 ERROR Validator,ajp-/127.0.0.1:8712-1:408 - [SECURITY FAILURE
> context=label-ViewRepositoryEdit_context, key=label,
> type(AlphaNumPunctuation)=^[\u0080\u0099\u2019\p{L}\p{M}\p{N}\p{Pd}\p{Pi}\p{P
> f}\p{Pc}\p{Po}\p{Sc}\p{So}\p{Sm}\p{Cf}\p{Z}\s\=\(\)\[\]\~\|\{\}\`\^\+&&[^<>]]
> *$, input=1æ<U+009C><U+0088>使ç<U+0094>¨æ<U+0083><U+0085>å<U+0086>µ,
> maxLength=5000, isBlacklist=false]
> 2016-01-12 15:28:15,078 INFO  [stdout] (ajp-/127.0.0.1:8712-13) 2016-01-12
> 15:28:15,069 ERROR AdhocAjaxController,ajp-/127.0.0.1:8712-13:888 - ad hoc
> controller exception
> 2016-01-12 15:33:56,864 INFO  [stdout] (ajp-/127.0.0.1:8712-10) 2016-01-12
> 15:33:56,855 ERROR AdhocAjaxController,ajp-/127.0.0.1:8712-10:888 - ad hoc
> controller exception
> 2016-01-12 15:42:27,583 INFO  [stdout] (ajp-/127.0.0.1:8712-9) 2016-01-12
> 15:42:27,575 ERROR ReportGeneratorController,ajp-/127.0.0.1:8712-9:96 -
> report generator controller exception: 执行上一个请求时出错。
> 2016-01-12 15:42:27,630 INFO  [stdout] (ajp-/127.0.0.1:8712-9) 2016-01-12
> 15:42:27,623 ERROR ReportGeneratorController,ajp-/127.0.0.1:8712-9:641 -
> Original exception overridden by exception handling failure
> 

This error seems to me like it is indeed the same bug.

> From these log entries do you believe they are seeing the same issue as
> documented in this BZ?
> 
> Regards,
> 
> Frank

Thank you. I will add the case to this BZ for tracking purposes. The first problem is marked as fixed in 3.5.1 however they are running 3.5.5? The real problem for them though is the ad-hoc failure documented in this bug.

Created attachment 1119322 [details]
normal situation

Created attachment 1119323 [details]
abnormal situation

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0425.html

.