Bug 1005383 - gdb and valgrind integration not working correctly.
gdb and valgrind integration not working correctly.
Status: CLOSED WORKSFORME
Product: Fedora
Classification: Fedora
Component: gdb (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jan Kratochvil
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-06 15:19 EDT by Carlos O'Donell
Modified: 2013-09-06 18:01 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-06 16:00:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Carlos O'Donell 2013-09-06 15:19:13 EDT
Description of problem:
Connecting to a process under the control of valgrind and remote debugging is not working. This is an important use case since x86-64 hardware does not support unlimited hardware watchpoints. By using valgrind one should be able to do things like `awatch keys' where keys is an array of 1024 entries. This is incredibly useful when tracking down stray writes to an array of values.

Version-Release number of selected component (if applicable):
gdb-7.6-34.fc19.x86_64
valgrind-3.8.1-15.fc19.x86_64

How reproducible:
Reproduces every time.

Steps to Reproduce:
cat >> test.c <<EOF
#include <stdio.h>
#include <stdlib.h>

int keys[1024];

int main (void)
{
  keys[512] = 42;
  printf ("%d\n", keys[512]);
  return 0;
}
EOF
gcc -Wall -pedantic -O0 -g3 -o ./test ./test.c
valgrind --vgdb=full --vgdb-error=0  ./test


Actual results:
==7882== Memcheck, a memory error detector
==7882== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==7882== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==7882== Command: test
==7882== 
==7882== (action at startup) vgdb me ... 
==7882== 
==7882== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==7882==   /path/to/gdb test
==7882== and then give GDB the following command
==7882==   target remote | /usr/lib64/valgrind/../../bin/vgdb --pid=7882
==7882== --pid is optional if only one valgrind process is running
==7882== 

...

gdb ./test
GNU gdb (GDB) Fedora (7.6-34.fc19)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/carlos/test...done.
(gdb) target remote | vgdb --pid=7882
Remote debugging using | vgdb --pid=7882
relaying data between gdb and process 7882
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00000039dc201420 in _start () from /lib64/ld-linux-x86-64.so.2

(gdb) break main
Breakpoint 1 at 0x400530: file test.c, line 7.
(gdb) c
Continuing.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

...

==7882== 
==7882== 
==7882== HEAP SUMMARY:
==7882==     in use at exit: 0 bytes in 0 blocks
==7882==   total heap usage: 30 allocs, 30 frees, 3,681 bytes allocated
==7882== 
==7882== All heap blocks were freed -- no leaks are possible
==7882== 
==7882== For counts of detected and suppressed errors, rerun with: -v
==7882== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)

...
[Inferior 1 (Remote target) exited with code 01]
(gdb) 


Expected results:
I expect gdb to stop at main, then I can further issue `awatch keys' and then I expect a watchpoint to trigger when I write to the 513th entry in keys.

Additional info:
None.
Comment 1 Carlos O'Donell 2013-09-06 15:20:53 EDT
Please not that I don't know if this is a valgrind or gdb issue but I'm starting by assigning this to gdb.
Comment 2 Mark Wielaard 2013-09-06 15:37:08 EDT
This does work for me with:

$ rpm -q gdb valgrind
gdb-7.6-32.fc19.x86_64
valgrind-3.8.1-15.fc19.x86_64

(Note gdb is slightly older in my case. -32 vs original reporter -34)

[compile and valgrind part is the same]

$ gdb ./test 
GNU gdb (GDB) Fedora (7.6-32.fc19)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/test...done.
(gdb) target remote | vgdb
Remote debugging using | vgdb
relaying data between gdb and process 1270
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003414601420 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) break main
Breakpoint 1 at 0x400534: file ./test.c, line 8.
(gdb) c
Continuing.

Breakpoint 1, main () at ./test.c:8
8	  keys[512] = 42;
(gdb) quit
A debugging session is active.

	Inferior 1 [Remote target] will be detached.

Quit anyway? (y or n) y
Detaching from program: /tmp/test, Remote target
Ending remote debugging.
Comment 3 Mark Wielaard 2013-09-06 15:40:40 EDT
Same after I update to gdb-7.6-34.fc19

(gdb) target remote | vgdb
Remote debugging using | vgdb
relaying data between gdb and process 1338
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003414601420 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) break main
Breakpoint 1 at 0x400534: file ./test.c, line 8.
(gdb) c
Continuing.

Breakpoint 1, main () at ./test.c:8
8	  keys[512] = 42;
(gdb) 


Note that the original reporter gets this warning:

warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

Which I am not seeing.
Comment 4 Jan Kratochvil 2013-09-06 15:51:00 EDT
I also do not have the problem reproducible.
"Probes-based dynamic linker interface failed." really looks suspicious, I would say this Bug should be reassigned to the glibc maintainer. :-)
Comment 5 Carlos O'Donell 2013-09-06 15:59:49 EDT
(In reply to Mark Wielaard from comment #3)
> Same after I update to gdb-7.6-34.fc19
> 
> (gdb) target remote | vgdb
> Remote debugging using | vgdb
> relaying data between gdb and process 1338
> Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
> /usr/lib/debug/lib64/ld-2.17.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> 0x0000003414601420 in _start () from /lib64/ld-linux-x86-64.so.2
> (gdb) break main
> Breakpoint 1 at 0x400534: file ./test.c, line 8.
> (gdb) c
> Continuing.
> 
> Breakpoint 1, main () at ./test.c:8
> 8	  keys[512] = 42;
> (gdb) 
> 
> 
> Note that the original reporter gets this warning:
> 
> warning: Probes-based dynamic linker interface failed.
> Reverting to original interface.
> 
> Which I am not seeing.

(1) Warning.

The probes-based dynamic linker interface is in glibc. 

What glibc do you have?

I have glibc-2.17-14.fc19.x86_64.

It would be ironic if this were a glibc issue, but we definitely worked hard to get the stap probes into the dynamic linker and working for F19.

(2) Update to latest packages.

yum install glibc valgrind gdb gcc binutils
Loaded plugins: auto-update-debuginfo, langpacks, refresh-packagekit
Package glibc-2.17-14.fc19.x86_64 already installed and latest version
Package 1:valgrind-3.8.1-15.fc19.x86_64 already installed and latest version
Package gdb-7.6-34.fc19.x86_64 already installed and latest version
Package gcc-4.8.1-1.fc19.x86_64 already installed and latest version
Package binutils-2.23.52.0.1-9.fc19.x86_64 already installed and latest version
Nothing to do

Can you confirm these versions?

(3) Retest.

[carlos@koi ~]$ gdb ./test
GNU gdb (GDB) Fedora (7.6-34.fc19)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/carlos/test...done.
(gdb) target remote | vgdb
Remote debugging using | vgdb
relaying data between gdb and process 11031
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00000039dc201420 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) break main
Breakpoint 1 at 0x400534: file test.c, line 8.
(gdb) c
Continuing.

Breakpoint 1, main () at test.c:8
8	  keys[512] = 42;
Missing separate debuginfos, use: debuginfo-install valgrind-3.8.1-15.fc19.x86_64
(gdb) 

Appears to work now.

I didn't imagine it, I was getting "[Inferior 1 (Remote target) exited with code 01]" when debugging.

Debugging using emulated hardware watchpoints works perfectly now:
~~~
[carlos@koi ~]$ gdb ./test
GNU gdb (GDB) Fedora (7.6-34.fc19)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/carlos/test...done.
(gdb) target remote | vgdb --pid=11211
Remote debugging using | vgdb --pid=11211
relaying data between gdb and process 11211
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00000039dc201420 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) awatch keys
Hardware access (read/write) watchpoint 1: keys
(gdb) c
Continuing.
Hardware access (read/write) watchpoint 1: keys

Old value = {0 <repeats 1024 times>}
New value = {0 <repeats 512 times>, 42, 0 <repeats 511 times>}
main () at test.c:9
9	  printf ("%d\n", keys[512]);
Missing separate debuginfos, use: debuginfo-install valgrind-3.8.1-15.fc19.x86_64
(gdb) c
Continuing.
Hardware access (read/write) watchpoint 1: keys

Value = {0 <repeats 512 times>, 42, 0 <repeats 511 times>}
main () at test.c:9
9	  printf ("%d\n", keys[512]);
(gdb) c
Continuing.
[Inferior 1 (Remote target) exited normally]
(gdb) 
~~~

Closing as CLOSED/WORKSFORME.

I don't know what might have caused gdb/valgrind issues.
Comment 6 Carlos O'Donell 2013-09-06 16:00:29 EDT
I didn't imagine the failure though so we should keep our eyes open.

The failure might have been a systemtap kernel issue?
Comment 7 Carlos O'Donell 2013-09-06 16:10:58 EDT
(In reply to Carlos O'Donell from comment #6)
> I didn't imagine the failure though so we should keep our eyes open.
> 
> The failure might have been a systemtap kernel issue?

Notes:
jkratoch says gdb uses stap probe point directly so no kernel involvement. Therefore this might be an intermittent gdb issue with target_write_memory.
Comment 8 Mark Wielaard 2013-09-06 18:01:24 EDT
(In reply to Carlos O'Donell from comment #5)
> (2) Update to latest packages.
> 
> yum install glibc valgrind gdb gcc binutils
> Loaded plugins: auto-update-debuginfo, langpacks, refresh-packagekit
> Package glibc-2.17-14.fc19.x86_64 already installed and latest version
> Package 1:valgrind-3.8.1-15.fc19.x86_64 already installed and latest version
> Package gdb-7.6-34.fc19.x86_64 already installed and latest version
> Package gcc-4.8.1-1.fc19.x86_64 already installed and latest version
> Package binutils-2.23.52.0.1-9.fc19.x86_64 already installed and latest
> version
> Nothing to do
> 
> Can you confirm these versions?

Yes:

$ rpm -q glibc valgrind gdb gcc binutils
glibc-2.17-14.fc19.x86_64
glibc-2.17-14.fc19.i686
valgrind-3.8.1-15.fc19.x86_64
gdb-7.6-34.fc19.x86_64
gcc-4.8.1-1.fc19.x86_64
binutils-2.23.52.0.1-9.fc19.x86_64

Note You need to log in before you can comment on or make changes to this bug.