Bug 2010221 - Bug 2009797 - Memory leak in invoker.c fillInvokeRequest() during JDI operations
Summary: Bug 2009797 - Memory leak in invoker.c fillInvokeRequest() during JDI operations
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: java-11-openjdk
Version: 7.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Roman Kennke
QA Contact: OpenJDK QA
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-04 08:51 UTC by Simeon Andreev
Modified: 2022-04-28 01:37 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
Crash log when using the patch from comment 14. (102.49 KB, text/plain)
2022-02-02 13:44 UTC, Simeon Andreev
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2009797 1 unspecified ASSIGNED Memory leak in invoker.c invoker_completeInvokeRequest() during JDI operations 2022-05-23 23:32:19 UTC
Red Hat Bugzilla 2010817 1 unspecified NEW Memory leaks in classTrack.c classTrack_addPreparedClass() during debugging 2021-11-21 17:22:04 UTC
Red Hat Bugzilla 2010850 1 unspecified NEW Memory leak in threadControl.c threadControl_onEventHandlerEntry() during debugging 2021-11-21 17:23:02 UTC
Red Hat Issue Tracker RHELPLAN-98730 0 None None None 2021-10-04 08:52:12 UTC

Internal Links: 2009797 2010817 2010850

Description Simeon Andreev 2021-10-04 08:51:25 UTC
Description of problem:


We observe a native memory leak when repeating JDI operations from Eclipse in a debuggee JVM.

jemalloc reports the biggest memory leak as:

[76800 bytes leaked]
je_prof_backtrace (/home/sandreev/git/misc/jemalloc/src/prof.c:636 (discriminator 2))
je_malloc_default (/home/sandreev/git/misc/jemalloc/src/jemalloc.c:2289)
os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:697)
os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:660 (discriminator 3))
JvmtiEnvBase::allocate (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.hpp:195)
JvmtiEnvBase::jvmtiMalloc (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.cpp:501)
JvmtiEnv::GetMethodName (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnv.cpp:2954)
jvmti_GetMethodName (/data/git/jdk11/build/linux-x86_64-normal-server-slowdebug/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:4366)
methodSignature (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/util.c:728)
fillInvokeRequest (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c:284)
invoker_requestInvoke (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c:371)
sharedInvoke (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/util.c:609)
invokeStatic (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/ClassTypeImpl.c:175)
debugLoop_run (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/debugLoop.c:159)
connectionInitiated (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/transport.c:296)
attachThread (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/transport.c:370)
JvmtiAgentThread::call_start_function (/data/git/jdk11/src/hotspot/share/prims/jvmtiImpl.cpp:85)
JvmtiAgentThread::start_function_wrapper (/data/git/jdk11/src/hotspot/share/prims/jvmtiImpl.cpp:79)
JavaThread::thread_main_inner (/data/git/jdk11/src/hotspot/share/runtime/thread.cpp:1752)
JavaThread::run (/data/git/jdk11/src/hotspot/share/runtime/thread.cpp:1732)
thread_native_entry (/data/git/jdk11/src/hotspot/os/linux/os_linux.cpp:698)
start_thread (/usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:308)
?? (/usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113)

OpenJDK 11

How reproducible:

Run JDI operations with Eclipse in any Java program. E.g. can be done with a command handler:

public class SampleHandler extends AbstractHandler {

	@Override
	public Object execute(ExecutionEvent event) throws ExecutionException {
		Job job = new Job("JDI operation") {
			@Override
			protected IStatus run(IProgressMonitor monitor) {
				int n = 20 * 60;
				for (int i = 0; i < n; ++i) {
					try {
						IJavaDebugTarget javaTarget = getFirstJavaDebugTarget();
						if (javaTarget != null) {
							IJavaThread javaThread = getFirstSuspendedJavaThread(javaTarget);
							if (javaThread != null) {
								IJavaType[] javaClass = javaTarget.getJavaTypes("java.lang.String");
								if (javaClass[0] instanceof IJavaClassType) {
									IJavaClassType jdiClassType = (IJavaClassType) javaClass[0];
									IJavaObject instance = jdiClassType.newInstance("()V", null, javaThread);
									try {
										instance.disableCollection();
										if (!monitor.isCanceled()) {
											instance.sendMessage("isEmpty", "()Z", null, javaThread, false);
										}
									} finally {
										instance.enableCollection();
									}
								}
							}
						}
					} catch (DebugException e) {
						e.printStackTrace();
					}
				}
				return Status.OK_STATUS;
			}
		};
		job.schedule();

		return null;
	}

	private IJavaDebugTarget getFirstJavaDebugTarget() {
		for (IDebugTarget debugTarget : DebugPlugin.getDefault().getLaunchManager().getDebugTargets()) {
			if (debugTarget instanceof IJavaDebugTarget) {
				return (IJavaDebugTarget) debugTarget;
			}
		}
		return null;
	}

	private IJavaThread getFirstSuspendedJavaThread(IJavaDebugTarget javaDebugTarget) throws DebugException {
		for (IThread thread : javaDebugTarget.getThreads()) {
			if (thread.isSuspended() && (thread instanceof IJavaThread)) {
				return (IJavaThread) thread;
			}
		}
		return null;
	} 
}

Actual results:

Memory consumption of debuggee JVM increases as JDI operations are repeated.

Expected results:

No memory is leaked by fillInvokeRequest() in invoker.c.

Comment 3 Simeon Andreev 2021-10-04 11:19:03 UTC
With the following change, I don't see the memory leak:

diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
index 8e2981d9fa..64d9c0dbd1 100644
--- a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
+++ b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
@@ -281,7 +281,11 @@ fillInvokeRequest(JNIEnv *env, InvokeRequest *request,
     /*
      * Squirrel away the method signature
      */
+    char *oldMethodSignature = request->methodSignature;
     error = methodSignature(method, NULL, &request->methodSignature,  NULL);
+    if (oldMethodSignature != NULL) {
+        jvmtiDeallocate(oldMethodSignature);
+    }
     if (error != JVMTI_ERROR_NONE) {
         return error;
     }

I don't know if the change is appropriate, but I assume the leak comes from overwriting the pointer request->methodSignature and not freeing the memory pointed at by the old pointer.

Comment 4 Simeon Andreev 2021-10-05 10:42:31 UTC
(In reply to Simeon Andreev from comment #3)
> With the following change, I don't see the memory leak:
> 
> diff --git a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
> b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
> index 8e2981d9fa..64d9c0dbd1 100644
> --- a/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
> +++ b/src/jdk.jdwp.agent/share/native/libjdwp/invoker.c
> @@ -281,7 +281,11 @@ fillInvokeRequest(JNIEnv *env, InvokeRequest *request,
>      /*
>       * Squirrel away the method signature
>       */
> +    char *oldMethodSignature = request->methodSignature;
>      error = methodSignature(method, NULL, &request->methodSignature,  NULL);
> +    if (oldMethodSignature != NULL) {
> +        jvmtiDeallocate(oldMethodSignature);
> +    }
>      if (error != JVMTI_ERROR_NONE) {
>          return error;
>      }
> 
> I don't know if the change is appropriate, but I assume the leak comes from
> overwriting the pointer request->methodSignature and not freeing the memory
> pointed at by the old pointer.

There seems to be another smaller, but similar leak:

[5888 bytes leaked]
je_prof_backtrace (/home/sandreev/git/misc/jemalloc/src/prof.c:636 (discriminator 2))
je_malloc_default (/home/sandreev/git/misc/jemalloc/src/jemalloc.c:2289)
os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:697)
os::malloc (/data/git/jdk11/src/hotspot/share/runtime/os.cpp:660 (discriminator 3))
JvmtiEnvBase::allocate (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.hpp:195)
JvmtiEnvBase::jvmtiMalloc (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnvBase.cpp:501)
JvmtiEnv::GetClassSignature (/data/git/jdk11/src/hotspot/share/prims/jvmtiEnv.cpp:2365)
jvmti_GetClassSignature (/data/git/jdk11/build/linux-x86_64-normal-server-slowdebug/hotspot/variant-server/gensrc/jvmtifiles/jvmtiEnter.cpp:3380)
classSignature (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/util.c:1235)
classTrack_addPreparedClass (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/classTrack.c:232)
event_callback (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/eventHandler.c:643)
cbClassPrepare (/data/git/jdk11/src/jdk.jdwp.agent/share/native/libjdwp/eventHandler.c:893)
JvmtiExport::post_class_prepare (/data/git/jdk11/src/hotspot/share/prims/jvmtiExport.cpp:1320)
InstanceKlass::link_class_impl (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:839 (discriminator 1))
InstanceKlass::link_class (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:698)
InstanceKlass::initialize_impl (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:898)
InstanceKlass::initialize (/data/git/jdk11/src/hotspot/share/oops/instanceKlass.cpp:669)
InterpreterRuntime::_new (/data/git/jdk11/src/hotspot/share/interpreter/interpreterRuntime.cpp:240)
?? (??:0)

I assume also there the initial value of node->signature is never freed, after the call to classTrack_addPreparedClass(). But attempts to make a similar change result in a crash:

Stack: [0x00007ffff7ecb000,0x00007ffff7fcc000],  sp=0x00007ffff7fc8d60,  free space=1015k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xb41fdc]  GuardedMemory::Guard::verify() const+0x22
V  [libjvm.so+0xb422cd]  GuardedMemory::verify_guards() const+0x2d
V  [libjvm.so+0x1039b91]  verify_memory(void*)+0x34
V  [libjvm.so+0x103a1ca]  os::free(void*)+0x8a
V  [libjvm.so+0xde9fa3]  JvmtiEnvBase::deallocate(unsigned char*)+0x23
V  [libjvm.so+0xde0cd5]  JvmtiEnv::Deallocate(unsigned char*)+0x23
V  [libjvm.so+0xd8c48b]  jvmti_Deallocate+0x1f8
C  [libjdwp.so+0x3abdf]  jvmtiDeallocate+0x51
C  [libjdwp.so+0x11cb1]  classTrack_addPreparedClass+0x175
C  [libjdwp.so+0x11f47]  classTrack_initialize+0xd8
C  [libjdwp.so+0x15cb2]  initialize+0x435
C  [libjdwp.so+0x14b06]  cbEarlyVMInit+0x10a
V  [libjvm.so+0xdf6a30]  JvmtiExport::post_vm_initialized()+0x18a
V  [libjvm.so+0x1242235]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x7eb
V  [libjvm.so+0xc9b92a]  JNI_CreateJavaVM_inner(JavaVM_**, void**, void*)+0x109
V  [libjvm.so+0xc9bc89]  JNI_CreateJavaVM+0x32
C  [libjli.so+0x781a]  InitializeJVM+0x13b
C  [libjli.so+0x432e]  JavaMain+0xd3

Comment 14 Roman Kennke 2022-02-02 13:06:39 UTC
Could you please test if this patch (vs jdk11, should also apply vs newer JDKs, possibly with a little fuzz) fixes the memory leak:

http://cr.openjdk.java.net/~rkennke/patch.txt

Thanks,
Roman

Comment 15 Simeon Andreev 2022-02-02 13:44:20 UTC
Created attachment 1858633 [details]
Crash log when using the patch from comment 14.

(In reply to Roman Kennke from comment #14)
> Could you please test if this patch (vs jdk11, should also apply vs newer
> JDKs, possibly with a little fuzz) fixes the memory leak:
> 
> http://cr.openjdk.java.net/~rkennke/patch.txt
> 
> Thanks,
> Roman

I got a JVM crash when using the reproduction steps, with a JDK 11 build that has the patch on top:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ffff2a9a093, pid=3880, tid=3922
#
# JRE version: OpenJDK Runtime Environment (11.0) (build 11-internal+0-adhoc.sandreev.jdk11)
# Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc.sandreev.jdk11, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libjdwp.so+0x1b093]  nextArgumentTypeTag+0x13
#
# Core dump will be written. Default location: /home/sandreev/temp_workspaces/contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/core.3880
#
# An error report file with more information is saved as:
# /home/sandreev/temp_workspaces/contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/hs_err_pid3880.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#


Did I somehow mess up the build?

I did:

git clean -fdx
git pull
bash configure
make images

Remote is:

https://github.com/openjdk/jdk11.git

I'm using the build at:

./build/linux-x86_64-normal-server-release/images/jdk/

Comment 16 Roman Kennke 2022-02-02 13:57:48 UTC
(In reply to Simeon Andreev from comment #15)
> Created attachment 1858633 [details]
> Crash log when using the patch from comment 14.
> 
> (In reply to Roman Kennke from comment #14)
> > Could you please test if this patch (vs jdk11, should also apply vs newer
> > JDKs, possibly with a little fuzz) fixes the memory leak:
> > 
> > http://cr.openjdk.java.net/~rkennke/patch.txt
> > 
> > Thanks,
> > Roman
> 
> I got a JVM crash when using the reproduction steps, with a JDK 11 build
> that has the patch on top:
> 
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00007ffff2a9a093, pid=3880, tid=3922
> #
> # JRE version: OpenJDK Runtime Environment (11.0) (build
> 11-internal+0-adhoc.sandreev.jdk11)
> # Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc.sandreev.jdk11,
> mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
> # Problematic frame:
> # C  [libjdwp.so+0x1b093]  nextArgumentTypeTag+0x13
> #
> # Core dump will be written. Default location:
> /home/sandreev/temp_workspaces/
> contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/core.3880
> #
> # An error report file with more information is saved as:
> #
> /home/sandreev/temp_workspaces/
> contributor_runtime_eclipse_jre11_jemalloc_customjre/Test/hs_err_pid3880.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> 
> 
> Did I somehow mess up the build?
> 
> I did:
> 
> git clean -fdx
> git pull
> bash configure
> make images
> 
> Remote is:
> 
> https://github.com/openjdk/jdk11.git
> 
> I'm using the build at:
> 
> ./build/linux-x86_64-normal-server-release/images/jdk/

It was probably my fault. I suspect that the methodSignature can be NULL for some requests. Please try the updated patch:
http://cr.openjdk.java.net/~rkennke/patch.txt

Thanks, Roman

Comment 17 Roman Kennke 2022-02-02 14:09:24 UTC
Nevermind the last patch. Please use this (i.e. deallocate the methodSignature in deleteGlobalArgumentRefs()):

http://cr.openjdk.java.net/~rkennke/patch.txt

Comment 18 Severin Gehwolf 2022-02-02 14:21:17 UTC
(In reply to Simeon Andreev from comment #15)
> Remote is:
> 
> https://github.com/openjdk/jdk11.git

This should be (11u):
https://github.com/openjdk/jdk11u

Comment 19 Simeon Andreev 2022-02-02 14:39:07 UTC
(In reply to Roman Kennke from comment #17)
> Nevermind the last patch. Please use this (i.e. deallocate the
> methodSignature in deleteGlobalArgumentRefs()):
> 
> http://cr.openjdk.java.net/~rkennke/patch.txt

No leak with the patch, vs leak with the Java 11 installation on my workstation:

openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.10+9-LTS, mixed mode, sharing)


Let me check also with https://github.com/openjdk/jdk11u, if that is where the patch will be landing.

Comment 20 Simeon Andreev 2022-02-02 15:05:41 UTC
OK, jemalloc doesn't report the memory leak with the patch on top of: https://github.com/openjdk/jdk11u (master branch)

Comment 21 Roman Kennke 2022-02-14 10:00:00 UTC
The bug has been fixed upstream in JDK19, and is going to be backported to JDK18, 17, 11 and perhaps 8 soon.

https://bugs.openjdk.java.net/browse/JDK-8276990

Comment 22 Roman Kennke 2022-03-09 13:03:27 UTC
(In reply to Roman Kennke from comment #21)
> The bug has been fixed upstream in JDK19, and is going to be backported to
> JDK18, 17, 11 and perhaps 8 soon.
> 
> https://bugs.openjdk.java.net/browse/JDK-8276990

I backported the fix to JDK 18, 17 and 11. It should become available in next OpenJDK release (April, I think).

Comment 23 Simeon Andreev 2022-03-09 13:07:26 UTC
(In reply to Roman Kennke from comment #22)
> (In reply to Roman Kennke from comment #21)
> > The bug has been fixed upstream in JDK19, and is going to be backported to
> > JDK18, 17, 11 and perhaps 8 soon.
> > 
> > https://bugs.openjdk.java.net/browse/JDK-8276990
> 
> I backported the fix to JDK 18, 17 and 11. It should become available in
> next OpenJDK release (April, I think).

Great, thanks!

Comment 24 Andrew John Hughes 2022-04-28 01:37:59 UTC
(In reply to Roman Kennke from comment #22)
> (In reply to Roman Kennke from comment #21)
> > The bug has been fixed upstream in JDK19, and is going to be backported to
> > JDK18, 17, 11 and perhaps 8 soon.
> > 
> > https://bugs.openjdk.java.net/browse/JDK-8276990
> 
> I backported the fix to JDK 18, 17 and 11. It should become available in
> next OpenJDK release (April, I think).

July. This went into 11.0.16 & 17.0.4.


Note You need to log in before you can comment on or make changes to this bug.