Bug 971894 - Sporadic qpid::types::Variant qpid::types::InvalidConversion exceptions (some even from string to string)
Sporadic qpid::types::Variant qpid::types::InvalidConversion exceptions (some...
Status: ON_QA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Ken Giusti
Irina Boverman
: OtherQA
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-07 10:03 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:18 EST (History)
4 users (show)

See Also:
Fixed In Version: qpid-cpp-0.22-14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Attempted reproducer - no failures. (624 bytes, text/plain)
2013-06-25 09:04 EDT, Ken Giusti
no flags Details

  None (edit)
Description Frantisek Reznicek 2013-06-07 10:03:43 EDT
Description of problem:

Sporadic qpid::types::Variant qpid::types::InvalidConversion exceptions (some even from string to string):

There are detected rare occurrences of unexpected qpid::types::InvalidConversion exceptions.

Some were triggered by sesame daemon see bug 772584, some triggered by qmf v1 qmf-agent example which are listed below:

a] Conversion from string to string (2 occurrences)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-rw------- 1 root root 21868544 Jun  7 05:43 core.19142

--- qmf-agent.log --- (A-beta)
terminate called after throwing an instance of 'qpid::types::InvalidConversion'
  what():  invalid conversion: Cannot convert from string to string (qpid/types/Variant.cpp:511)

[root@ibm-hs23-01 qpid_test_qmf_agent_bz470167]# gdb ./qmf-agent core.19142 
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-45.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /mnt/tests/distribution/MRG/Messaging/qpid_test_qmf_agent_bz470167/qmf-agent...done.
[New Thread 19144]
[New Thread 19143]
[New Thread 19142]
...
Core was generated by `./qmf-agent'.
Program terminated with signal 6, Aborted.
#0  0x00d4b410 in __kernel_vsyscall ()
(gdb) info thread
  3 Thread 0xb7f2b720 (LWP 19142)  0x00d4b410 in __kernel_vsyscall ()
  2 Thread 19143  0x00d4b410 in __kernel_vsyscall ()
* 1 Thread 0xb7508b90 (LWP 19144)  0x00d4b410 in __kernel_vsyscall ()
(gdb) t a a bt

Thread 3 (Thread 0xb7f2b720 (LWP 19142)):
#0  0x00d4b410 in __kernel_vsyscall ()
#1  0x007f58d6 in nanosleep () from /lib/libc.so.6
#2  0x007f56ff in sleep () from /lib/libc.so.6
#3  0x0039d81d in qpid::sys::sleep (secs=1) at qpid/sys/posix/Time.cpp:123
#4  0x0804b442 in CoreClass::doLoop (this=0xbfecc8d0) at example.cpp:126
#5  0x0804c9c9 in main_int (argc=1, argv=0xbfecca24) at example.cpp:226
#6  0x0804d127 in main (argc=0, argv=0x3b9898dc) at example.cpp:237

Thread 2 (Thread 19143):
#0  0x00d4b410 in __kernel_vsyscall ()
#1  0x007f58d6 in nanosleep () from /lib/libc.so.6
#2  0x007f56ff in sleep () from /lib/libc.so.6
#3  0x0039d81d in qpid::sys::sleep (secs=1) at qpid/sys/posix/Time.cpp:123
#4  0x006de545 in qpid::management::ManagementAgentImpl::ConnectionThread::run (this=0xb7f2a1cc) at qpid/agent/ManagementAgentImpl.cpp:1291
#5  0x0039df81 in qpid::sys::(anonymous namespace)::runRunnable (p=0xb7f2a1cc) at qpid/sys/posix/Thread.cpp:35
#6  0x008cd912 in start_thread () from /lib/libpthread.so.0
#7  0x0083760e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb7508b90 (LWP 19144)):
#0  0x00d4b410 in __kernel_vsyscall ()
#1  0x0078ae30 in raise () from /lib/libc.so.6
#2  0x0078c741 in abort () from /lib/libc.so.6
#3  0x001e0be0 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#4  0x001de5b5 in ?? () from /usr/lib/libstdc++.so.6
#5  0x001de5f2 in std::terminate() () from /usr/lib/libstdc++.so.6
#6  0x001de6ab in __cxa_rethrow () from /usr/lib/libstdc++.so.6
#7  0x08050405 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant>, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > >::_M_copy (this=0xb6a008b0, __x=0x8218bc8, __p=0xb6a008b4) at /usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:1250
#8  0x00568565 in _Rb_tree (this=0xb6a00898, m=std::map with 4 elements = {...}) at /usr/include/c++/4.1.2/bits/stl_tree.h:570
#9  map (this=0xb6a00898, m=std::map with 4 elements = {...}) at /usr/include/c++/4.1.2/bits/stl_map.h:175
#10 qpid::types::VariantImpl::VariantImpl (this=0xb6a00898, m=std::map with 4 elements = {...}) at qpid/types/Variant.cpp:153
#11 0x00568620 in qpid::types::Variant::operator= (this=0xb6a0088c, m=std::map with 4 elements = {...}) at qpid/types/Variant.cpp:783
#12 0x006ca7a4 in qpid::management::ManagementAgentImpl::getHeartbeatContent (this=0xb7f0a008, map=std::map with 1 elements = {...}) at qpid/agent/ManagementAgentImpl.cpp:1213
#13 0x006dd153 in qpid::management::ManagementAgentImpl::sendHeartbeat (this=0xb7f0a008) at qpid/agent/ManagementAgentImpl.cpp:494
#14 0x006e2234 in qpid::management::ManagementAgentImpl::periodicProcessing (this=0xb7f0a008) at qpid/agent/ManagementAgentImpl.cpp:1081
#15 0x006e47a8 in qpid::management::ManagementAgentImpl::PublishThread::run (this=0xb7f2a2f0) at qpid/agent/ManagementAgentImpl.cpp:1413
#16 0x0039df81 in qpid::sys::(anonymous namespace)::runRunnable (p=0xb7f2a2f0) at qpid/sys/posix/Thread.cpp:35
#17 0x008cd912 in start_thread () from /lib/libpthread.so.0
#18 0x0083760e in clone () from /lib/libc.so.6


b] Conversion from <unknown> to string
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-rw------- 1 root root 21868544 Jun  7 08:58 core.12901

[08:58:18] run(A) 96/100--------------------------[err_cnt:5]-
--- qmf-agent.log --- (A-alpha)
terminate called after throwing an instance of 'qpid::types::InvalidConversion'
  what():  invalid conversion: Cannot convert from <unknown> to string (qpid/types/Variant.cpp:511)


[root@ibm-hs23-01 qpid_test_qmf_agent_bz470167]# gdb ./qmf-agent core.12901 
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-45.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /mnt/tests/distribution/MRG/Messaging/qpid_test_qmf_agent_bz470167/qmf-agent...done.
[New Thread 12904]
[New Thread 12903]
[New Thread 12901]
...
Core was generated by `./qmf-agent'.
Program terminated with signal 6, Aborted.
#0  0x008b8410 in __kernel_vsyscall ()
(gdb) info thread
  3 Thread 0xb7f66720 (LWP 12901)  0x008b8410 in __kernel_vsyscall ()
  2 Thread 12903  0x008b8410 in __kernel_vsyscall ()
* 1 Thread 0xb7543b90 (LWP 12904)  0x008b8410 in __kernel_vsyscall ()
(gdb) t a a bt

Thread 3 (Thread 0xb7f66720 (LWP 12901)):
#0  0x008b8410 in __kernel_vsyscall ()
#1  0x001a38d6 in nanosleep () from /lib/libc.so.6
#2  0x001a36ff in sleep () from /lib/libc.so.6
#3  0x0044e81d in qpid::sys::sleep (secs=1) at qpid/sys/posix/Time.cpp:123
#4  0x0804b442 in CoreClass::doLoop (this=0xbfe0bcf0) at example.cpp:126
#5  0x0804c9c9 in main_int (argc=1, argv=0xbfe0be44) at example.cpp:226
#6  0x0804d127 in main (argc=0, argv=0x3b9846c0) at example.cpp:237

Thread 2 (Thread 12903):
#0  0x008b8410 in __kernel_vsyscall ()
#1  0x001a38d6 in nanosleep () from /lib/libc.so.6
#2  0x001a36ff in sleep () from /lib/libc.so.6
#3  0x0044e81d in qpid::sys::sleep (secs=1) at qpid/sys/posix/Time.cpp:123
#4  0x006de545 in qpid::management::ManagementAgentImpl::ConnectionThread::run (this=0xb7f651cc) at qpid/agent/ManagementAgentImpl.cpp:1291
#5  0x0044ef81 in qpid::sys::(anonymous namespace)::runRunnable (p=0xb7f651cc) at qpid/sys/posix/Thread.cpp:35
#6  0x008cd912 in start_thread () from /lib/libpthread.so.0
#7  0x001e560e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb7543b90 (LWP 12904)):
#0  0x008b8410 in __kernel_vsyscall ()
#1  0x00138e30 in raise () from /lib/libc.so.6
#2  0x0013a741 in abort () from /lib/libc.so.6
#3  0x00323be0 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#4  0x003215b5 in ?? () from /usr/lib/libstdc++.so.6
#5  0x003215f2 in std::terminate() () from /usr/lib/libstdc++.so.6
#6  0x003216ab in __cxa_rethrow () from /usr/lib/libstdc++.so.6
#7  0x08050337 in _M_create_node (this=0x9f53ef8, __x=0x9f537c8, __p=0x9f53efc) at /usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:373
#8  _M_clone_node (this=0x9f53ef8, __x=0x9f537c8, __p=0x9f53efc) at /usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:381
#9  std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant>, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, qpid::types::Variant> > >::_M_copy (
    this=0x9f53ef8, __x=0x9f537c8, __p=0x9f53efc) at /usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:1226
#10 0x00617565 in _Rb_tree (this=0x9f53ca0, m=std::map with 4 elements = {...}) at /usr/include/c++/4.1.2/bits/stl_tree.h:570
#11 map (this=0x9f53ca0, m=std::map with 4 elements = {...}) at /usr/include/c++/4.1.2/bits/stl_map.h:175
#12 qpid::types::VariantImpl::VariantImpl (this=0x9f53ca0, m=std::map with 4 elements = {...}) at qpid/types/Variant.cpp:153
#13 0x00617620 in qpid::types::Variant::operator= (this=0x9f53eec, m=std::map with 4 elements = {...}) at qpid/types/Variant.cpp:783
#14 0x006ca7a4 in qpid::management::ManagementAgentImpl::getHeartbeatContent (this=0xb7f45008, map=std::map with 1 elements = {...}) at qpid/agent/ManagementAgentImpl.cpp:1213
#15 0x006dd153 in qpid::management::ManagementAgentImpl::sendHeartbeat (this=0xb7f45008) at qpid/agent/ManagementAgentImpl.cpp:494
#16 0x006e2234 in qpid::management::ManagementAgentImpl::periodicProcessing (this=0xb7f45008) at qpid/agent/ManagementAgentImpl.cpp:1081
#17 0x006e47a8 in qpid::management::ManagementAgentImpl::PublishThread::run (this=0xb7f652f0) at qpid/agent/ManagementAgentImpl.cpp:1413
#18 0x0044ef81 in qpid::sys::(anonymous namespace)::runRunnable (p=0xb7f652f0) at qpid/sys/posix/Thread.cpp:35
#19 0x008cd912 in start_thread () from /lib/libpthread.so.0
#20 0x001e560e in clone () from /lib/libc.so.6


Version-Release number of selected component (if applicable):
qpid-0.18

How reproducible:
< 1%

Steps to Reproduce:
TEST_PARAM_RHTS_TEST_MRG_QPID_TEST_QMF_AGENT_BZ470167_LOOPS=100 make rb
or
1] compile the qmf-agent from sources (revision matching)
2] launch qmf-agent (no broker running)
3] stop qmf-agent by SIGINT signal
3.5] expect qmf-agent will cleanly shut down
4] run the broker
5] launch qmf-agent (broker running)
6] stop qmf-agent by SIGINT signal
6.5] expect qmf-agent will cleanly shut down
7] stop the broker
7.5] expect broker shuting down cleanly


Actual results:
qmf-agent uncovers that qpid::types::Variant sometimes does not convert obvious types.

Expected results:
qmf-agent should not trigger uqpid::types::InvalidConversion.

Additional info:
Comment 1 Justin Ross 2013-06-19 17:11:11 EDT
Ken, please assess.
Comment 2 Ken Giusti 2013-06-25 09:04:51 EDT
Created attachment 765075 [details]
Attempted reproducer - no failures.

I'm having difficulty reproducing this.

I've run the attached script against both trunk and 0.14-mrg-preview-2 for several hours on my F17 box (x86_64) with no luck.
Comment 3 Ken Giusti 2013-06-25 09:08:40 EDT
My guess would be some sort of corruption, but have been unable to repro (see previous comment).  This event is especially confusing:

terminate called after throwing an instance of 'qpid::types::InvalidConversion'
  what():  invalid conversion: Cannot convert from string to string (qpid/types/Variant.cpp:511)


Since the "string to string" conversion is actually a path through that method that would avoid throwing the exception!


Need to reproduce this - are there any specific environments that tend to reproduce this?  I've trunk trunk and 0.14-mrg-preview-2 on f17 64-bit, with no luck.
Comment 4 Frantisek Reznicek 2013-07-04 03:45:06 EDT
(In reply to Ken Giusti from comment #3)
> 
> Need to reproduce this - are there any specific environments that tend to
> reproduce this?  I've trunk trunk and 0.14-mrg-preview-2 on f17 64-bit, with
> no luck.

I'll reproduce that for you on RHEL, will stay in touch. (I'm gonna clear needinfo once reproduced)

Note You need to log in before you can comment on or make changes to this bug.