Hide Forgot
Description of problem: We observe a native memory leak when repeating JDI operations from Eclipse in a debuggee JVM. jemalloc reports the biggest memory leak as: [76800 bytes leaked] je_prof_backtrace (/home/sandreev/git/misc/jemalloc/src/prof.c:636 (discriminator 2)) je_malloc_default (/home/sandreev/git/misc/jemalloc/src/jemalloc.c:2289) os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:697) os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:660 (discriminator 3)) JvmtiEnvBase::allocate (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.hpp:195) JvmtiEnvBase::jvmtiMalloc (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.cpp:501) JvmtiEnv::GetMethodName (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnv.cpp:2954) jvmti_GetMethodName (/data/git/jdk11/build/linux-x86_64-normal-server-slowdebug/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:4366) methodSignature (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/util.c:728) fillInvokeRequest (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c:284) invoker_requestInvoke (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c:371) sharedInvoke (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/util.c:609) invokeStatic (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/ClassTypeImpl.c:175) debugLoop_run (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c:159) connectionInitiated (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/transport.c:296) attachThread (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/transport.c:370) JvmtiAgentThread::call_start_function (/data/git/jdk11/src/hotspot/share/prims/jvmtiImpl.cpp:85) JvmtiAgentThread::start_function_wrapper (/data/git/jdk11/src/hotspot/share/prims/jvmtiImpl.cpp:79) JavaThread::thread_main_inner (/data/git/jdk11/src/hotspot/share/runtime/thread.cpp:1752) JavaThread::run (/data/git/jdk11/src/hotspot/share/runtime/thread.cpp:1732) thread_native_entry (/data/git/jdk11/src/hotspot/os/linux/os_linux.cpp:698) start_thread (/usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:308) ?? (/usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113) OpenJDK 11 How reproducible: Run JDI operations with Eclipse in any Java program. E.g. can be done with a command handler: public class SampleHandler extends AbstractHandler { @Override public Object execute(ExecutionEvent event) throws ExecutionException { Job job = new Job("JDI operation") { @Override protected IStatus run(IProgressMonitor monitor) { int n = 20 * 60; for (int i = 0; i < n; ++i) { try { IJavaDebugTarget javaTarget = getFirstJavaDebugTarget(); if (javaTarget != null) { IJavaThread javaThread = getFirstSuspendedJavaThread(javaTarget); if (javaThread != null) { IJavaType[] javaClass = javaTarget.getJavaTypes("java.lang.String"); if (javaClass[0] instanceof IJavaClassType) { IJavaClassType jdiClassType = (IJavaClassType) javaClass[0]; IJavaObject instance = jdiClassType.newInstance("()V", null, javaThread); try { instance.disableCollection(); if (!monitor.isCanceled()) { instance.sendMessage("isEmpty", "()Z", null, javaThread, false); } } finally { instance.enableCollection(); } } } } } catch (DebugException e) { e.printStackTrace(); } } return Status.OK_STATUS; } }; job.schedule(); return null; } private IJavaDebugTarget getFirstJavaDebugTarget() { for (IDebugTarget debugTarget : DebugPlugin.getDefault().getLaunchManager().getDebugTargets()) { if (debugTarget instanceof IJavaDebugTarget) { return (IJavaDebugTarget) debugTarget; } } return null; } private IJavaThread getFirstSuspendedJavaThread(IJavaDebugTarget javaDebugTarget) throws DebugException { for (IThread thread : javaDebugTarget.getThreads()) { if (thread.isSuspended() && (thread instanceof IJavaThread)) { return (IJavaThread) thread; } } return null; } } Actual results: Memory consumption of debuggee JVM increases as JDI operations are repeated. Expected results: No memory is leaked by fillInvokeRequest() in invoker.c.
With the following change, I don't see the memory leak: diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c index 8e2981d9fa..64d9c0dbd1 100644 --- a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c +++ b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c @@ -281,7 +281,11 @@ fillInvokeRequest(JNIEnv *env, InvokeRequest *request, /* * Squirrel away the method signature */ + char *oldMethodSignature = request->methodSignature; error = methodSignature(method, NULL, &request->methodSignature, NULL); + if (oldMethodSignature != NULL) { + jvmtiDeallocate(oldMethodSignature); + } if (error != JVMTI_ERROR_NONE) { return error; } I don't know if the change is appropriate, but I assume the leak comes from overwriting the pointer request->methodSignature and not freeing the memory pointed at by the old pointer.
(In reply to Simeon Andreev from comment #3) > With the following change, I don't see the memory leak: > > diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c > b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c > index 8e2981d9fa..64d9c0dbd1 100644 > --- a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c > +++ b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c > @@ -281,7 +281,11 @@ fillInvokeRequest(JNIEnv *env, InvokeRequest *request, > /* > * Squirrel away the method signature > */ > + char *oldMethodSignature = request->methodSignature; > error = methodSignature(method, NULL, &request->methodSignature, NULL); > + if (oldMethodSignature != NULL) { > + jvmtiDeallocate(oldMethodSignature); > + } > if (error != JVMTI_ERROR_NONE) { > return error; > } > > I don't know if the change is appropriate, but I assume the leak comes from > overwriting the pointer request->methodSignature and not freeing the memory > pointed at by the old pointer. There seems to be another smaller, but similar leak: [5888 bytes leaked] je_prof_backtrace (/home/sandreev/git/misc/jemalloc/src/prof.c:636 (discriminator 2)) je_malloc_default (/home/sandreev/git/misc/jemalloc/src/jemalloc.c:2289) os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:697) os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:660 (discriminator 3)) JvmtiEnvBase::allocate (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.hpp:195) JvmtiEnvBase::jvmtiMalloc (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.cpp:501) JvmtiEnv::GetClassSignature (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnv.cpp:2365) jvmti_GetClassSignature (/data/git/jdk11/build/linux-x86_64-normal-server-slowdebug/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:3380) classSignature (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/util.c:1235) classTrack_addPreparedClass (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/classTrack.c:232) event_callback (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/eventHandler.c:643) cbClassPrepare (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/eventHandler.c:893) JvmtiExport::post_class_prepare (/data/git/jdk11/src/hotspot/share/prims/jvmtiExport.cpp:1320) InstanceKlass::link_class_impl (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:839 (discriminator 1)) InstanceKlass::link_class (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:698) InstanceKlass::initialize_impl (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:898) InstanceKlass::initialize (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:669) InterpreterRuntime::_new (/data/git/jdk11/src/hotspot/share/interpreter/interpreterRuntime.cpp:240) ?? (??:0) I assume also there the initial value of node->signature is never freed, after the call to classTrack_addPreparedClass(). But attempts to make a similar change result in a crash: Stack: [0x00007ffff7ecb000,0x00007ffff7fcc000], sp=0x00007ffff7fc8d60, free space=1015k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xb41fdc] GuardedMemory::Guard::verify() const+0x22 V [libjvm.so+0xb422cd] GuardedMemory::verify_guards() const+0x2d V [libjvm.so+0x1039b91] verify_memory(void*)+0x34 V [libjvm.so+0x103a1ca] os::free(void*)+0x8a V [libjvm.so+0xde9fa3] JvmtiEnvBase::deallocate(unsigned char*)+0x23 V [libjvm.so+0xde0cd5] JvmtiEnv::Deallocate(unsigned char*)+0x23 V [libjvm.so+0xd8c48b] jvmti_Deallocate+0x1f8 C [libjdwp.so+0x3abdf] jvmtiDeallocate+0x51 C [libjdwp.so+0x11cb1] classTrack_addPreparedClass+0x175 C [libjdwp.so+0x11f47] classTrack_initialize+0xd8 C [libjdwp.so+0x15cb2] initialize+0x435 C [libjdwp.so+0x14b06] cbEarlyVMInit+0x10a V [libjvm.so+0xdf6a30] JvmtiExport::post_vm_initialized()+0x18a V [libjvm.so+0x1242235] Threads::create_vm(JavaVMInitArgs*, bool*)+0x7eb V [libjvm.so+0xc9b92a] JNI_CreateJavaVM_inner(JavaVM_**, void**, void*)+0x109 V [libjvm.so+0xc9bc89] JNI_CreateJavaVM+0x32 C [libjli.so+0x781a] InitializeJVM+0x13b C [libjli.so+0x432e] JavaMain+0xd3
Could you please test if this patch (vs jdk11, should also apply vs newer JDKs, possibly with a little fuzz) fixes the memory leak: http://cr.openjdk.java.net/~rkennke/patch.txt Thanks, Roman
Created attachment 1858633 [details] Crash log when using the patch from comment 14. (In reply to Roman Kennke from comment #14) > Could you please test if this patch (vs jdk11, should also apply vs newer > JDKs, possibly with a little fuzz) fixes the memory leak: > > http://cr.openjdk.java.net/~rkennke/patch.txt > > Thanks, > Roman I got a JVM crash when using the reproduction steps, with a JDK 11 build that has the patch on top: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007ffff2a9a093, pid=3880, tid=3922 # # JRE version: OpenJDK Runtime Environment (11.0) (build 11-internal+0-adhoc.sandreev.jdk11) # Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc.sandreev.jdk11, mixed mode, tiered, compressed oops, g1 gc, linux-amd64) # Problematic frame: # C [libjdwp.so+0x1b093] nextArgumentTypeTag+0x13 # # Core dump will be written. Default location: /home/sandreev/temp_workspaces/contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/core.3880 # # An error report file with more information is saved as: # /home/sandreev/temp_workspaces/contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/hs_err_pid3880.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # Did I somehow mess up the build? I did: git clean -fdx git pull bash configure make images Remote is: https://github.com/openjdk/jdk11.git I'm using the build at: ./build/linux-x86_64-normal-server-release/images/jdk/
(In reply to Simeon Andreev from comment #15) > Created attachment 1858633 [details] > Crash log when using the patch from comment 14. > > (In reply to Roman Kennke from comment #14) > > Could you please test if this patch (vs jdk11, should also apply vs newer > > JDKs, possibly with a little fuzz) fixes the memory leak: > > > > http://cr.openjdk.java.net/~rkennke/patch.txt > > > > Thanks, > > Roman > > I got a JVM crash when using the reproduction steps, with a JDK 11 build > that has the patch on top: > > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x00007ffff2a9a093, pid=3880, tid=3922 > # > # JRE version: OpenJDK Runtime Environment (11.0) (build > 11-internal+0-adhoc.sandreev.jdk11) > # Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc.sandreev.jdk11, > mixed mode, tiered, compressed oops, g1 gc, linux-amd64) > # Problematic frame: > # C [libjdwp.so+0x1b093] nextArgumentTypeTag+0x13 > # > # Core dump will be written. Default location: > /home/sandreev/temp_workspaces/ > contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/core.3880 > # > # An error report file with more information is saved as: > # > /home/sandreev/temp_workspaces/ > contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/hs_err_pid3880.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > > > Did I somehow mess up the build? > > I did: > > git clean -fdx > git pull > bash configure > make images > > Remote is: > > https://github.com/openjdk/jdk11.git > > I'm using the build at: > > ./build/linux-x86_64-normal-server-release/images/jdk/ It was probably my fault. I suspect that the methodSignature can be NULL for some requests. Please try the updated patch: http://cr.openjdk.java.net/~rkennke/patch.txt Thanks, Roman
Nevermind the last patch. Please use this (i.e. deallocate the methodSignature in deleteGlobalArgumentRefs()): http://cr.openjdk.java.net/~rkennke/patch.txt
(In reply to Simeon Andreev from comment #15) > Remote is: > > https://github.com/openjdk/jdk11.git This should be (11u): https://github.com/openjdk/jdk11u
(In reply to Roman Kennke from comment #17) > Nevermind the last patch. Please use this (i.e. deallocate the > methodSignature in deleteGlobalArgumentRefs()): > > http://cr.openjdk.java.net/~rkennke/patch.txt No leak with the patch, vs leak with the Java 11 installation on my workstation: openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment 18.9 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.10+9-LTS, mixed mode, sharing) Let me check also with https://github.com/openjdk/jdk11u, if that is where the patch will be landing.
OK, jemalloc doesn't report the memory leak with the patch on top of: https://github.com/openjdk/jdk11u (master branch)
The bug has been fixed upstream in JDK19, and is going to be backported to JDK18, 17, 11 and perhaps 8 soon. https://bugs.openjdk.java.net/browse/JDK-8276990
(In reply to Roman Kennke from comment #21) > The bug has been fixed upstream in JDK19, and is going to be backported to > JDK18, 17, 11 and perhaps 8 soon. > > https://bugs.openjdk.java.net/browse/JDK-8276990 I backported the fix to JDK 18, 17 and 11. It should become available in next OpenJDK release (April, I think).
(In reply to Roman Kennke from comment #22) > (In reply to Roman Kennke from comment #21) > > The bug has been fixed upstream in JDK19, and is going to be backported to > > JDK18, 17, 11 and perhaps 8 soon. > > > > https://bugs.openjdk.java.net/browse/JDK-8276990 > > I backported the fix to JDK 18, 17 and 11. It should become available in > next OpenJDK release (April, I think). Great, thanks!
(In reply to Roman Kennke from comment #22) > (In reply to Roman Kennke from comment #21) > > The bug has been fixed upstream in JDK19, and is going to be backported to > > JDK18, 17, 11 and perhaps 8 soon. > > > > https://bugs.openjdk.java.net/browse/JDK-8276990 > > I backported the fix to JDK 18, 17 and 11. It should become available in > next OpenJDK release (April, I think). July. This went into 11.0.16 & 17.0.4.