Bug 884646

Created attachment 658738 [details]
err1

Created attachment 658739 [details]
err2

Created attachment 658740 [details]
err3

Created attachment 658741 [details]
err4

Created attachment 658742 [details]
err5

Created attachment 658743 [details]
err6

Created attachment 658744 [details]
err7

Actually the cookie issue is caused by commons-httpclient used from performance test. commons-httpclient is by default using RFC2109 Cookie policy, which means that it adds some informations like $Version and $Path into HTTP cookie header. Parsing of those cookie attributes is not handled 100% correctly on AS7 side.

So to avoid this issue, the performance test needs to be configured so that HttpClient will use browser compatible cookie policy and won't send any cookie attributes (like Version and Path) to JPP. It could be done either by:
- Use something like:
httpClient.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY);
in class HttpRequestProcessorFactoryImpl (or make it configurable via some parameter passed to this processor factory)

- Or by setting System property "-Dapache.commons.httpclient.cookiespec=COMPATIBILITY" which will ensure that commons-httpclient will use this browser compatible policy by default.

This should help with the "Unknown Special Cookie" message and hopefully with scalability as well.

Marek, thanks for this excellent finding - I'm going to verify that now!
commons-httpclient version 3.1 is used at loaddriver, do you think changing this version could also help?
I'll update here after my verification.

Cookie issue is gone, but scalability is not what we expect anyway.
You can have a look at logs and progress here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EPP/view/6.0/view/Performance/job/epp6_portal_scalability_loggedUsers_2nodes/29

Thanks Michal! Good thing is that now we can better see other exception messages :)

Can you please try to run scalability build with disabled cluster for Picketlink IDM? This will allow us to see if bottleneck is coming from here (seems that yes according to most of exception messages).

The easiest way to do it is to comment "apiCacheProvider" and "storeCacheProvider" entries for cluster profile in file gatein/gatein.ear/portal.war/WEB-INF/conf/organization/idm-configuration.xml . It can look like this:

      <value-param>
        <name>apiCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan.xml</value>
      </value-param>
<!--
      <value-param profiles="cluster">
        <name>apiCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan-cluster.xml</value>
      </value-param>
-->
      <value-param>
        <name>storeCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan.xml</value>
      </value-param>
<!--
      <value-param profiles="cluster">
        <name>storeCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan-cluster.xml</value>
      </value-param>
-->

Good hint Marek, I've tried with disabled clustered apiCacheConfig & storeCacheConfig and exceptions during scalability are gone.
Now we have to find the way how to configure picketlink clustering.
You can have a look at job here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EPP/view/6.0/view/Performance/job/epp6_portal_scalability_loggedUsers_2nodes/30

Thanks Michal,
so for clustering configuration: is it possible to try:

- Left idm-configuration.xml as is by default (without any commented section). So it will look like this:
<value-param>
        <name>apiCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan.xml</value>
      </value-param>

      <value-param profiles="cluster">
        <name>apiCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan-cluster.xml</value>
      </value-param>

      <value-param>
        <name>storeCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan.xml</value>
      </value-param>

      <value-param profiles="cluster">
        <name>storeCacheConfig</name>
        <value>war:/conf/organization/picketlink-idm/infinispan-cluster.xml</value>
      </value-param>

- In file gatein/gatein.ear/portal.war/WEB-INF/conf/organization/picketlink-idm/infinispan-cluster.xml add attribute syncCommitPhase="false" to transaction section:
<transaction transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC" autoCommit="true" syncCommitPhase="false" />

Marek, unfortunately this didn't help.
Progress with same errors is here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EPP/view/6.0/view/Performance/job/epp6_portal_scalability_loggedUsers_2nodes/31

Hi Marek,
it seems like async replication brought expected performance improvement at picketlink clustering, but there are still some failures. Details are at builds 32 & 34.
Do you plan to do some other updates I can verify? Thanks!

Marek Posolda <mposolda> updated the status of jira GTNPORTAL-2792 to Resolved

Marek Posolda <mposolda> made a comment on jira GTNPORTAL-2792

Fixed in GateIn master in commit https://github.com/gatein/gatein-portal/commit/528e769a0b11caa1a6fc92d47d547f28ea231c3b

Scalability for logged users was fixed by above change.

Statistics were updated at https://docs.google.com/a/rhcollab.com/spreadsheet/ccc?key=0At752QrNfufDdG1CcnFfTzVNWnh6aUJQcmZjbi1SRWc#gid=11 

Results for 1,2 nodes are now comparable with results from EPP 5.2