Bug 1023157

Assigning to author of Java-ABRT connector. Pavel, please take a look when you get a chance.

Hi!

substitute .../jvm/java-1.7.0-openjd/ with full_path_to_jdk

As you probably noted, the java-abr-connector have two touches to jdk.
First - the .../jvm/java-1.7.0-openjd/jre/bin/java is no longer binary, but script which is launching the .../jvm/java-1.7.0-openjd/jre-abrt/bin/java.

The path was set, because the jvm binary have hardcoded paths around bin and lib.

If java-abrt-connector is installed, then .../jvm/java-1.7.0-openjd/jre-abrt/bin/java script is adding libjavaabrt.so agent into .../jvm/java-1.7.0-openjd/jre-abrt/bin/java arguments.

Please may you confirm, that the path is what is causing the freezing or libjavaabrt itself?

So you need to restore previous configuration with .../jvm/java-1.7.0-openjd/jre{,-abrt}/bin/{java,lib} where .../jvm/java-1.7.0-openjd/jre/bin/java is teh script lunching the .../jvm/java-1.7.0-openjd/jre-abrt/bin/java, and try your testrusite with java-abrt-connector installed/missing.

In both cases (wrong libjavaagent.so, or wrong paths) we will need the reproducer. Also, if your tests, or applications you are testing are dependent on .../whatever/jre/bin/java path, then they are wrong, and this bug should be reassign to authors of proper components.

I'll let Endi comment on the exact paths as used by pki/tomcat there.   I will say though that he had me try something like this to test and see if it was java-abrt.

In my test script, I added this:

cd /usr/lib/jvm/java/jre/bin
mv java java.old
ls -s ../../bin/java

Now, I will also note the following:

1. I was seeing the hang occur in RHEL7 release 20131011.n.0.
  
2. I have run the tests 10 times in the last two days for release 20131018.n.0 and not seen a hang.

3. When I apply the java link workaround to 20131011.n.0 I haven't seen the issue yet.

So, Was something fixed between 20131011.n.0 and 20131018.n.0 that may have resolved the issue?  Or just dumb luck that I haven't seen it again yet on the newer release?

I'm not sure yet if we have a good clean reproducer but, if necessary, I can help reproduce with IPA tests.

Thanks,
Scott

(In reply to Scott Poore from comment #7)
> I'll let Endi comment on the exact paths as used by pki/tomcat there.   I
> will say though that he had me try something like this to test and see if it
> was java-abrt.
> 
> In my test script, I added this:
> 
> cd /usr/lib/jvm/java/jre/bin
> mv java java.old
> ls -s ../../bin/java
> 
> Now, I will also note the following:
> 
> 1. I was seeing the hang occur in RHEL7 release 20131011.n.0.
>   
> 2. I have run the tests 10 times in the last two days for release
> 20131018.n.0 and not seen a hang.
> 
> 3. When I apply the java link workaround to 20131011.n.0 I haven't seen the
> issue yet.
> 
> So, Was something fixed between 20131011.n.0 and 20131018.n.0 that may have
> resolved the issue?  Or just dumb luck that I haven't seen it again yet on
> the newer release?
> 
> I'm not sure yet if we have a good clean reproducer but, if necessary, I can
> help reproduce with IPA tests.
> 
> Thanks,
> Scott

The abrt was made soft depndence. May  you rerun the tests with java-abrt-connector installed?

Jiri, this bug report looks like a duplicate of bug #1012827.

Jiri, 

Ah I see.  

On the builds where it's hanging we have this installed:

abrt-java-connector-1.0.5-1.el7.x86_64

On the newer builds that is not installed.

I am running tests now so this should get installed with the newer rhel7 release.  I'll post results back here when I have them.

Thanks,
Scott

Hi, just FYI when I reproduced the problem I was using the default java which on my test machine was:
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/bin/java
I saw that it's actually executing java-abrt with the abrt-java-connector.

As Scott said, I replaced the "java" file there with a soft link to:
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/bin/java
This way it wasn't using java-abrt nor the abrt-java-connector anymore.

Unfortunately I don't have the test machine anymore but hopefully Scott will be able to get some results.

Jiri, Jakub,

I saw 2 of my 4 test runs hang.  So, it does appear that abrt-java-connector is involved.

Is there anything else we need to do here to confirm if this is a dup of bug #1012827 or is this enough?

Thanks,
Scott

Scott, Jiri,

I am pretty sure that this bug is a dup of bug #1012827
It has exactly the same symptoms as we observed together with Pavel.

Here's the stack trace from Scott's test. It looks like one of the thread is stuck in a critical section within abrt-java-connector.

#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f5e9a27fa4b in os::PlatformEvent::park (this=0x7f5e94017e00)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:5440
#2  0x00007f5e9a167be4 in JvmtiRawMonitor::SimpleEnter (
    this=this@entry=0x7f5e94011280, Self=Self@entry=0x7f5e94017000)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiRawMonitor.cpp:147
#3  0x00007f5e9a1681f5 in JvmtiRawMonitor::raw_enter (
    this=this@entry=0x7f5e94011280, 
    __the_thread__=__the_thread__@entry=0x7f5e94017000)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiRawMonitor.cpp:308
#4  0x00007f5e9a14bd9c in JvmtiEnv::RawMonitorEnter (this=<optimized out>, 
    rmonitor=0x7f5e94011280)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiEnv.cpp:3059
#5  0x00007f5e98a7ecd7 in enter_critical_section (
    jvmti_env=jvmti_env@entry=0x7f5e94010bf0)
    at /usr/src/debug/abrt-java-connector-9214372f6635aa377954f26a7c4dc90477a14564/src/abrt-checker.c:565
#6  0x00007f5e98a7ed61 in callback_on_object_alloc (jvmti_env=0x7f5e94010bf0, 
    jni_env=<optimized out>, thread=<optimized out>, object=<optimized out>, 
    object_klass=0x7f5e94103078, size=<optimized out>)
    at /usr/src/debug/abrt-java-connector-9214372f6635aa377954f26a7c4dc90477a14564/src/abrt-checker.c:1980
#7  0x00007f5e9a15fbda in JvmtiExport::post_vm_object_alloc (
    thread=0x7f5e94017000, object=object@entry=0x7686f1be8)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiExport.cpp:2122
#8  0x00007f5e9a160c5f in JvmtiVMObjectAllocEventCollector::~JvmtiVMObjectAllocEventCollector (this=0x7f5e9b6b75f0, __in_chrg=<optimized out>)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiExport.cpp:2338
#9  0x00007f5e9a0e08a3 in JVM_GetStackAccessControlContext (
    env=0x7f5e940171d8, cls=<optimized out>)
    at /usr/src/debug/java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el7.x86_64/openjdk/hotspot/src/share/vm/prims/jvm.cpp:1281
#10 0x00007f5e90e2c4cc in ?? ()
#11 0x00007f5e9b6b7d20 in ?? ()
#12 0x00007f5e9b6b7cd8 in ?? ()
#13 0x000000000000006f in ?? ()
#14 0x0000000000000000 in ?? ()

(In reply to Jakub Filak from comment #13)
> Scott, Jiri,
> 
> I am pretty sure that this bug is a dup of bug #1012827
> It has exactly the same symptoms as we observed together with Pavel.

yes. looks like this. Although here is much more information, 1012827 have all flags. So closing this one. Thanx to all who provided information!

*** This bug has been marked as a duplicate of bug 1012827 ***