Hide Forgot
Description of problem: 2019-02-21 00:42:44,620+09 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error GC overhead limit exceeded The dump shows lots of objects sitting in an event deque for 2 hosts in an environment with just 2 hosts: Class Name | Shallow Heap | Retained Heap | Percentage ----------------------------------------------------------------------------------------------------------------- org.ovirt.vdsm.jsonrpc.client.events.SubscriptionHolder @ 0x6ce539218| 40 | 2,516,477,992 | 61.85% |- java.util.concurrent.ConcurrentLinkedDeque @ 0x6ce539338 | 24 | 2,516,477,536 | 61.85% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x70b617fb8 | 24 | 28,144 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x71d295f48 | 24 | 28,144 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x7a6b1c768 | 24 | 28,144 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fb3c9270 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fb3c92e8 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x701ff6518 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x701ff6590 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x70b617fe8 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x71d295f60 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x7a6b30eb8 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f0d4a080 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f0d4a0b0 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f0d4a170 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f0d4a1a0 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a909d0 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a909e8 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90ac0 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90ad8 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90b80 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90b98 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90c70 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90c88 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90d30 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f1a90d48 | 24 | 28,136 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d01dce88 | 24 | 28,128 | 0.00% | '- Total: 25 of 105,690 entries; 105,665 more | | | |- java.lang.String[4] @ 0x6ce8d8be8 | 32 | 128 | 0.00% |- java.lang.String @ 0x6ce8d8b70 removed | 24 | 96 | 0.00% |- java.util.ArrayList @ 0x6ce8d8ae0 | 24 | 80 | 0.00% |- java.lang.String @ 0x6ce8d8b30 VM_status | 24 | 64 | 0.00% |- java.util.concurrent.locks.ReentrantLock @ 0x6ce8d8ab0 | 16 | 48 | 0.00% '- Total: 6 entries | | | org.ovirt.vdsm.jsonrpc.client.events.SubscriptionHolder @ 0x6ce8d8398| 40 | 1,436,008,600 | 35.30% |- java.util.concurrent.ConcurrentLinkedDeque @ 0x6ce8d88e0 | 24 | 1,436,008,144 | 35.30% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x7bb691328 | 24 | 45,592 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f0f11c28 | 24 | 27,864 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x7753385c8 | 24 | 27,864 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x77533f2a0 | 24 | 27,864 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6f0f11c58 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fb63d848 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fb648a80 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fb661620 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fb6682f0 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fbacdb98 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6fbacdbb0 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x73881a650 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x73881a668 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x7389da8c8 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x7389da8f8 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x748f873b8 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x748f9b9e8 | 24 | 27,856 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d0a9d718 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d150ae98 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d2fd4688 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d31c8688 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d33ab6c0 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d37fbe40 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d3a17c80 | 24 | 27,848 | 0.00% | |- java.util.concurrent.ConcurrentLinkedDeque$Node @ 0x6d43f63d8 | 24 | 27,848 | 0.00% | '- Total: 25 of 58,946 entries; 58,921 more | | | |- java.lang.String[4] @ 0x6ce8d8910 | 32 | 128 | 0.00% |- java.lang.String @ 0x6ce8d89d0 removed | 24 | 96 | 0.00% |- java.util.ArrayList @ 0x6ce8d8a30 | 24 | 80 | 0.00% |- java.lang.String @ 0x6ce8d8960 VM_status | 24 | 64 | 0.00% |- java.util.concurrent.locks.ReentrantLock @ 0x6ce8d8a80 | 16 | 48 | 0.00% '- Total: 6 entries | | | Version-Release number of selected component (if applicable): rhvm-4.2.6.4-0.1.el7ev.noarch How reproducible: Unknown
Hosts run vdsm-4.20.39.1-1.el7ev
vdsm-jsonrpc-java-1.4.14-1.el7ev
Hi Ravi, Thank you for your work on this bugzilla. Do we still need to have logs in RHEV host side? Best Regards, Keigo Noha
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.3.z': '?'}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.3.z': '?'}', ] For more info please contact: rhv-devops
Looks good to me, thanks
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:1566
sync2jira