Bug 900986 (JBPAPP6-1192)

Use attached groovy script to check ZIP distribution of javadoc.

Attachment: Added: exposed.txt
Attachment: Added: DOCcheck.groovy

Labels: Added: eap601-qe-triage

Approved for EAP 6.0.1

Labels: Removed: eap601-qe-triage 

The RPM now just wraps Ondrej's ZIP, so when it is fixed in the ZIP it will automatically (well, semi-automatically) be fixed in the RPM also.

Note that this is dependant on the review of the flags in the AS modules.

Flags were updated.

I went trough the script, and it seems it ignores the transitively exposed modules.
Last time I asked about that, those were also supposed to be included in the Javadoc.
I'm sending a mail to dev list to check whether that's still the case.

{quote}
the AS public API javadoc is generated by scripts which follow the logic discussed earlier. In short, these are the rules for inclusion:

* All modules NOT marked with   <property name="jboss.api" value="private"/> are included
* Their     <dependencies>  which are NOT marked with  <property name="jboss.api" value="private"/> are included
* "unsupported" is treated the same as "private"
* All this is merged using union.  I.e. if something is marked as private, but exposed as a non-private dependency of a non-private module, it's included.

Is the last point ok? It leads to various jars included which would make more sense if not included.
See https://issues.jboss.org/browse/JBPAPP-10099 
Perhaps a module marked private should override exposed deps?
{quote}

Here's the groovy script, FYC: http://pastebin.test.redhat.com/110669

And current ER2 Javadoc: http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1-ER2/

All right so I have this from David:
{quote}
Dependencies are non-transitive; i.e. they're not visible even if 
they're included by a public module, so there's no reason to pull them 
in.  So only the first and third bullet really make any sense.
{quote}

Seems like I will be the one rewriting the script :)

Ok, now I feel little lost.  If I understand that correctly my script should be working as expected.

Yes, your script AFAICT follows the correct idea.

I've changed the Javadoc generating script and now running it. ETA today afternoon.

Here is the new JavaDoc: http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1-ER3-master/

Jakub, why is org.apache.cxf.* etc marked as leaked? It's public in EAP. At least in the master branch.

Also, some packages are sometimes visible, sometimes not, like org.w3c.dom.html which is visible in module org.apache.xerces.
I am not sure how to deal with that since in/excludes can't go to per-class level.

Please re-run against ER3 once handed off and let me know.
Alternatively, you can build a snapshot and check against the Javadoc jar:
http://www.qa.jboss.com/~ozizka/AS7/javadoc/javadoc-EAP-6.0.1.ER3-master.jar

VPN disconnected while uploading. Uploading agian.

The org.apache.cxf.* leak info comes from module org/apache/cxf/impl/main which was previously marked as private. Unfortunately script is not dealing with case when some packages are in different modules with different visibility and I don't think it is worth the effort when u can check the report from script. 

Link: Added: This issue relates to JBPAPP-9181

result for javadoc-EAP-6.0.1.ER3-master.jar http://pastebin.test.redhat.com/111165

some jars should not be listed, but mostly output is correct.

Also take a look at [JBPAPP-9944|https://issues.jboss.org/browse/JBPAPP-9944] -as some modules.xml's contain wrong classification-

Hi, where do you get the module.xml's from? Do you convert it from the document?

No I used module.xml's from ER2 -  according to that document module classifications in those files should be correct (in the way when no modules which should be marked as private are marked as public)

Regenerating. I used wrong branch. Sorry for the noise.

Try now.
http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER3-eap6.jar
http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER3-eap6

Pls reopen if still not correct.

http://pastebin.test.redhat.com/112541


There may be some  confusion about org.apache.cxf.* as api is public and implementation isn't. However I've made some quick tweaks of control script so it now shouldn't list package as leaked if it is also part of some public module

Seems that the `<includeDependencySources>false</includeDependencySources>` is ignored.
I've changed the script to generate explicit excludes of everything not included.
Now creating javadoc, let's see.

I've also added blacklisting mechanism.

Try now:

* Jar: http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER3-eap6-02-excludes.jar
* Web: http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER3-eap6-02-excludes/

I've checked few packages from the list. They are not there now. Pls re-run the script and let me know.

I've checked few more, and some are still present: org.apache.cxf.management for instance, but that's the case when the package is in multiple jars: It's in public module modules/org/apache/cxf/main/cxf-api-2.6.3.jar.

OK, I will take a look at these and let you know. 

If the package is in both private and public module then I think it should be exposed and I will also try to improve that verification script to somehow deal with this - because the last attempt obviously didn't work (or it worked only partially as some packages are still listed even though they shouldn't be). 

So I've checked it and it's much better now. Only few possible leaks left - mostly org.apache.cxf.* packages. The problem here may be in different version of  cfx-api. In your previous comment you've mentioned  "modules/org/apache/cxf/main/cxf-api-2.6.3.jar" In 6.0.1.ER3 there is "cxf-api-2.4.9-redhat-2.jar" which does not contain " org.apache.cxf.management" package, thus the package was marked as leaked. 

here is the result of verification script:

=== Exposed docs for 4 private jar files ===
/org/apache/cxf/impl/main/cxf-common-utilities-2.4.9-redhat-2.jar
18 exposed:
   org.apache.cxf.resource
   org.apache.cxf.version
   org.apache.cxf.staxutils.transform
   org.apache.cxf.staxutils
   org.apache.cxf.helpers
   org.apache.cxf.management
   org.apache.cxf.management.annotation
   org.apache.cxf.configuration.spring
   org.apache.cxf.common.xmlschema
   org.apache.cxf.common.security
   org.apache.cxf.common
   org.apache.cxf.common.util
   org.apache.cxf.common.i18n
   org.apache.cxf.common.injection
   org.apache.cxf.common.classloader
   org.apache.cxf.common.annotation
   org.apache.cxf.common.logging
   org.apache.cxf.common.commands

/org/apache/cxf/impl/main/cxf-rt-core-2.4.9-redhat-2.jar
3 exposed:
   org.apache.cxf.interceptor.transform
   org.apache.cxf.attachment
   org.apache.cxf.configuration.spring

/org/apache/cxf/impl/main/cxf-rt-management-2.4.9-redhat-2.jar
1 exposed:
   org.apache.cxf.management

/org/slf4j/impl/main/slf4j-jboss-logmanager-1.0.0.GA-redhat-2.jar
1 exposed:
   org.slf4j.impl

Reopening - script results in previous comment.

Due to confusion about cxf-api version, there is a chance that it already is correct. 

Link: Added: This issue depends JBPAPP-9838

New try, hopefully correct:
http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER4.jar

How to run the checking script? I'd like to be able to check on my own.

Ondrej, have you changed the javadoc ZIP?  If you do we need a new jbossas-javadoc RPM to match.
If this is the case, reopen it and assign to me with the link where to get the .zip (Shelly says it should be a .zip).

Fernando, it's just a name difference. Or do you expect different content? .jar is just what the plugin spits out in the end, so it seems natural to leave it so. I just set the name to ${version}.jar. Could be eap-${version}-javadoc.jar, perhaps.

So contents are the same?  I was wondering if the contents of the JAR you gave me for the ER3 are not the latest.
So if the contents have not changed I don't need to rebuild the RPM and we are fine.

Ondra:
Get updated version of script from [eap6-javadoc folder in git|http://git.app.eng.bos.redhat.com/?p=jbossqe/eap-tests-scripts.git] and run DOCcheck.groovy 

To print leaked private packages use
{code}
groovy DOCcheck.groovy -l -doc "path to doc" -eap "path to eap"
{code} 
or use
{code}
groovy DOCcheck.groovy --help
{code} 


latest result:
/org/slf4j/impl/main/slf4j-jboss-logmanager-1.0.0.GA-redhat-2.jar
1 exposed:
   org.slf4j.impl


Beside this last leak there are several missing packages (although I have to make sure that script behaves correctly for missing packages)

@ Fernando, yes, the content of ZIP vs. JAR is the same, the diff was just what plugin I used to pack.

@Jakub, about the missing ones, there's a blacklist of sources which are not processed due to bug in Javadoc. See here: https://github.com/jbossas/jboss-as/blob/master/build/javadoc/artifactsBlacklist.txt 

Otherwise what I do is just following the metadata.

Release Notes Docs Status: Added: Not Required

* `org.slf4j.impl` - not sure why that gets included. Only 3 classes. It's explicitely excluded. Probably that will need to be deleted manually. Is it worth?
* `com.google.jtype` - it's not in /modules when I build EAP ER4. Perhaps it gets there during prod? I'd simply ignore it. That's surely not part of API.
* `picketbox-4.0.14.Final-redhat-1.jar` - that was a bug in my script. Fixed.

New Javadoc, with picketlink fixed:

http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER4-2.jar
http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER4-2/

everything is as expected (excludes), but that "org.slf4j.impl" package.   

I can not  verify this even though it is "only 3 classes" unless there is some statement saying that it is ok to leave it. From QE point of view it is not correct. 
 

I've checked javadoc plugin for new versions. The new one allows to exclude on package level.
So, here's the new javadoc without org.slf4j.impl:

http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER4-3.jar
http://www.qa.jboss.com/~ozizka/AS7/javadoc/6.0.1.ER4-3/

You did it Ondra, no leaks are present in this build.

Although  [JBPAPP-10100|https://issues.jboss.org/browse/] still remains because of those blacklisted jars. 

Can't find JBPAPP-10100, did you delete it?

[JBPAPP-10100|https://issues.jboss.org/browse/JBPAPP-10100] is still there

Reopening and setting status back to resolved. 

I was told to keep it like this until we can verify against official ER4. In fact I have closed this issue myself and that had to reopen it. So keep it this way pls.

Release Notes Docs Status: Removed: Not Required 
Docs QE Status: Removed: NEW 

Link: Added: This issue is duplicated by JBPAPP6-913

verified against ER4