Bug 1186918

Currently gdb does not work well in containers.  I would say this is more of a gdb issue then an issue with docker or atomic.

The process on the host is in a separate namespace, which GDB doesn't recognize.  GDB should work if you use nsenter to enter that namespace.  To borrow your example:

bash-4.2# ps -ef | grep journal
root        530      1  0 14:24 ?     00:00:01 /usr/lib/systemd/systemd-journald
bash-4.2# sudo nsenter --target 530 --mount --uts --ipc --net --pid
bash-4.2# gdb -p 1

It would be nice if GDB did this automatically, but this should work for now.

Right but this would only work if gdb is in the container.

Is that not the most natural way to work?
You can't install debuginfo on Atomic can you? On the host?
So that has to be in the container, presumably in some debug layer on top of your production layers.
Forgive me if I'm totally off-track, I've only been working with Docker for a week.

Well the goal is to have lots of containers, and then to allow an admin/developer to be able to debug them.  Eventually gdb will grow the ability to handle this, so you might have a gdb container that could look at processes in other containers and debug them.  Whether the debuginfo in in the debugged container, on the host or on a server somewhere.

Bottom line most containers will not ship with gdb embedded in them.

Suggest investigating installing gdbserver (but not gdb) in the
containers, and have an external gdb invoke it via
"target extended-remote | nsenter .... gdbserver -"
or such.

(In reply to Frank Ch. Eigler from comment #6)
> Suggest investigating installing gdbserver (but not gdb) in the
> containers, and have an external gdb invoke it via
> "target extended-remote | nsenter .... gdbserver -"
> or such.

Agreed.

We should make no distinction between container on host, or container somewhere else in the cloud. The gdbserver binary is tiny, consider it the minimal agent required to debug applications.

Created attachment 995093 [details]
Broken backtraces with nsenter...gdbserver

Frank, nsenter...gdbserver does not seem to work.  GDB shows the same backtrace with nsenter...gdbserver as it does if you attach directly (see attached).

I assume the shared libraries in the container are not the same as on the host.

What does "info sharedlibrary" show?

You'll also need to point GDB at the correct copy of the container's shared libraries and their debug info.

If the container's filesystem is visible on some directory on the host,
try "set sysroot /container-root/" and "set debug-file-directory /container-root/usr/lib/debug/"

If not, try "set sysroot remote:", which has GDB read the files directly out of the remote target.  But given separate debug info files, I think you'd also need "set debug-file-directory remote:", which IIRC doesn't work today.

That's what's happening, "info sharedlibrary" shows different "To" addresses for
/lib64/libc.so.6 and /lib64/ld-linux-x86-64.so.2, and looking in the respective
filesystems the files are different.  After "set sysroot remote:" "info sharedlibrary" shows the correct addresses, but the backtrace is still messed up (possibly because it's still loading the symbols for /bin/bash from the debugger container rather than the app container).

(In reply to Gary Benson from comment #10)
> That's what's happening, "info sharedlibrary" shows different "To" addresses
> for
> /lib64/libc.so.6 and /lib64/ld-linux-x86-64.so.2, and looking in the
> respective
> filesystems the files are different.  After "set sysroot remote:" "info
> sharedlibrary" shows the correct addresses, but the backtrace is still
> messed up (possibly because it's still loading the symbols for /bin/bash
> from the debugger container rather than the app container).

What normally happens though is that gdb complains bitterly about section mismatches when it tries to line up the host and target libraries (Which are not the same)? Did this not happen?

A big part of this is educating our audience about how remote debugging works. And container debugging is exactly like remote debugging. Then automating and making this easier.

Pedro, Does `set sysroot` get the right libthread_db.so? I expect the next thing to fail is thread debugging since you've not used `set libthread-db-search-path`?

There were no complaints about section mismatches, but possibly because the host and target libraries are pretty similar:

  rhel7:      glibc-2.17-55.el7.x86_64
  rhel-tools: glibc-2.17-78.el7.x86_64

In a way that's more insidious than having very different libraries.

I disagree about educating our audience.  Sure, if we find a workaround for a not-yet-fixed bug we can educate people, but remote debugging is just something we're exploring in a "well, direct debugging doesn't work, lets see if we can make it work using remote."  Container debugging doesn't *have* to be like remote debugging: the debugger and the inferior are on the same system, users should be able to run "gdb -p PID" and have GDB do the legwork.  I don't want to go down the road of remote debugging being the preferred solution because right now remote debugging does not have the same feature set as native.  And it's more awkward.

> Pedro, Does `set sysroot` get the right libthread_db.so?

No, it's unrelated.

> I expect the next thing to fail is thread debugging since you've not used `set libthread-db-search-path`?

With remote debugging, it's actually gdbserver that loads libthread_db.so, not GDB.

Given that in general we don't know what system is running inside the container, the user may indeed in some cases need to tweak the libthread_db.so search path.
I expect that to be rare though, given how containers tend to be very minimal and not include e.g., multiple glibc versions in odd paths.  In any case, if the user needs that, there's an equivalent monitor command to tweak it:
  "monitor set libthread-db-search-path"

(In reply to Gary Benson from comment #12)
> There were no complaints about section mismatches, but possibly because the
> host and target libraries are pretty similar:
> 
>   rhel7:      glibc-2.17-55.el7.x86_64
>   rhel-tools: glibc-2.17-78.el7.x86_64
> 
> In a way that's more insidious than having very different libraries.

Agreed.

> I disagree about educating our audience.  Sure, if we find a workaround for
> a not-yet-fixed bug we can educate people, but remote debugging is just
> something we're exploring in a "well, direct debugging doesn't work, lets
> see if we can make it work using remote."  Container debugging doesn't
> *have* to be like remote debugging: the debugger and the inferior are on the
> same system, users should be able to run "gdb -p PID" and have GDB do the
> legwork.  I don't want to go down the road of remote debugging being the
> preferred solution because right now remote debugging does not have the same
> feature set as native.  And it's more awkward.

You're right, container debugging doesn't have to be like remote debugging. It's just the only useful framework I have in my mind to describe this. I agree that the more gdb does the better.

(In reply to Carlos O'Donell from comment #14)
> (In reply to Gary Benson from comment #12)
> > There were no complaints about section mismatches, but possibly because the
> > host and target libraries are pretty similar:
> > 
> >   rhel7:      glibc-2.17-55.el7.x86_64
> >   rhel-tools: glibc-2.17-78.el7.x86_64
> > 
> > In a way that's more insidious than having very different libraries.
> 
> Agreed.

Ideally GDB would compare the GNU buildid of the files found on host's filesystem with the buildid of the loaded DSOs in target memory and warn if a mismatch is found.  Patches have been posted upstream a while ago that make this work.  I believe Jan picked them up and is stewarding/improving them.

(In reply to Pedro Alves from comment #13)
> > Pedro, Does `set sysroot` get the right libthread_db.so?
> 
> No, it's unrelated.
> 
> > I expect the next thing to fail is thread debugging since you've not used `set libthread-db-search-path`?
> 

Considering docker + random systems running inside the containers, the libthread_db.so aspect is worth thinking about some more.  It's "interesting" for a couple of reasons:

- security concerns

libthread_db.so is a library that is loaded and executed in GDB's address space and because of that basically can do anything GDB can.  The libthread_db.so that is being loaded is the one from the container, not our validated version shipped on the "host".

- ABI mismatches

Hopefully the ABIs inside and outside the container are compatible.  If they're subtly incompatible and the "host"'s loader manages to dlopen libthread_db.so into GDB, gdb will crash when it calls into libthread_db.so, if we're lucky.
We may never see this in practice, but it could happen.

- Stability

If the libthread_db.so that is loaded into GDB is buggy, GDB misbehaves/crashes, but that is beyond our control.  The GDB team then gets to try to figure out what went wrong from a bug report, which may not be obvious at all.

> With remote debugging, it's actually gdbserver that loads libthread_db.so,
> not GDB.

One thing to keep in mind here I thought to mention, related to the proposal to make gdbserver a static binary: gdbserver needs to dlopen libthread_db.so, not link with it, which conflicts with the static linking.  AFAIK, glibc doesn't support that really well.

libthread_db.so really needs to die :-/

(In reply to Pedro Alves from comment #16)

All of your previous concerns apply...

> One thing to keep in mind here I thought to mention, related to the proposal
> to make gdbserver a static binary: gdbserver needs to dlopen
> libthread_db.so, not link with it, which conflicts with the static linking. 
> AFAIK, glibc doesn't support that really well.

... except this one. This is something which glibc suppoorts perfectly fine.

However, because you have two coordinating runtimes, the static one, and the dynamic one being loaded, they must be identical runtimes to ensure the coordination goes as planned. There is no direct isolation between the two runtimes. Any failures are bugs and we'll fix them.

However, this goes against my recommendation which is not to make gdbserver a static application, but to bundle it and run the bundle inside the target container e.g. -Wl,--dynamic-loader=$ORIGIN/ld.so -Wl,-rpath=$ORIGIN/libs etc. etc. You might even have a self-contained executable.

> libthread_db.so really needs to die :-/

No, you still have NSS, ICONV and IDN all which require dlopen in glibc. While libthread_db.so can be converted to a data-driven design, NSS can't, and is required for you to lookup permissions and other things. Therefore a static gdbserver is simply not an option.

The fundamental problem remains: The remote host in the container can be anything. Either you build to run on anything (compile with an old-enough sysroot), compile fully statically (rewriting glibc to provide NSS, ICONV, and IDN via data-based isolation e.g. sockets to servers with services whose protocols are known and documented), bundle your dependencies.

So while libthread_db.so=>python or dwarf-description solves one problem, it doesn't solve the others.

(In reply to Carlos O'Donell from comment #17)
> (In reply to Pedro Alves from comment #16)
> 
> All of your previous concerns apply...
> 
> > One thing to keep in mind here I thought to mention, related to the proposal
> > to make gdbserver a static binary: gdbserver needs to dlopen
> > libthread_db.so, not link with it, which conflicts with the static linking. 
> > AFAIK, glibc doesn't support that really well.
> 
> ... except this one. This is something which glibc suppoorts perfectly fine.

Well, except it's practically useless for the use case static linking would most be useful for: isolation from whatever runtime is in the system.

> However, because you have two coordinating runtimes, the static one, and the
> dynamic one being loaded, they must be identical runtimes to ensure the
> coordination goes as planned. There is no direct isolation between the two
> runtimes.

Yes, that's specifically the issue that I was trying to recall but it was too late be able to put the finger on --- a statically build gdbserver linked against an old RHEL sysroot for compatility can not dlopen a random libthread_db.so inside the container, as that will load a second, incompatible runtime.  Actually, isn't that a problem even in the dynamically linked case?

> Any failures are bugs and we'll fix them.

Good, thanks.  :-)

> 
> However, this goes against my recommendation which is not to make gdbserver
> a static application, but to bundle it and run the bundle inside the target
> container e.g. -Wl,--dynamic-loader=$ORIGIN/ld.so -Wl,-rpath=$ORIGIN/libs
> etc. etc.

Ah, yes, somehow I got confused.  Sorry about that.

> You might even have a self-contained executable.

If we need to open libthread_db outside the box/sysroot for TLS debugging, we'll still have a problem, right?

> 
> > libthread_db.so really needs to die :-/
> 
> No, you still have NSS, ICONV and IDN all which require dlopen in glibc.
> While libthread_db.so can be converted to a data-driven design, NSS can't,
> and is required for you to lookup permissions and other things. Therefore a
> static gdbserver is simply not an option.

*NOD*

But, does a gdbserver linked with an old sysroot need to load the NSS, etc. modules in the container (not those in the sysroot)?  If it does, can it be done in a safe, defined way, or do we run into two-runtimes-in-same-process issues?

> The fundamental problem remains: The remote host in the container can be
> anything. Either you build to run on anything (compile with an old-enough
> sysroot), compile fully statically (rewriting glibc to provide NSS, ICONV,
> and IDN via data-based isolation e.g. sockets to servers with services whose
> protocols are known and documented), bundle your dependencies.
> 

> So while libthread_db.so=>python or dwarf-description solves one problem, it
> doesn't solve the others.

It would seem to solve the problem in the solution that Gary's now prototyping, which is to make GDB itself enter/exit the namespaces as needed.  nss, etc. are no longer a problem here, as gdb is running outside the container.  In this solution, the container's (potentially untrusted) libthread_db.so is _still_ what needs to be loaded into the host's GDB in order to support TLS.

(In reply to Pedro Alves from comment #18)
> (In reply to Carlos O'Donell from comment #17)
> > (In reply to Pedro Alves from comment #16)
> > 
> > All of your previous concerns apply...
> > 
> > > One thing to keep in mind here I thought to mention, related to the proposal
> > > to make gdbserver a static binary: gdbserver needs to dlopen
> > > libthread_db.so, not link with it, which conflicts with the static linking. 
> > > AFAIK, glibc doesn't support that really well.
> > 
> > ... except this one. This is something which glibc suppoorts perfectly fine.
> 
> Well, except it's practically useless for the use case static linking would
> most be useful for: isolation from whatever runtime is in the system.

Agreed.

Sorry, I was being nit picky here, and defending glibc. Static linking is useless for providing *complete* isolation, as is desired in this case. I merely wanted to clarify that glibc does support dlopen from statically linked applications. Supportability has nothing to do with the fact that the  feature doesn't do what you want. We could make it do what you want, but then it wouldn't work on every system. For example: a customer using SSSD/IdM for user and group management, without NSS and plugins, we would need a gdbserver that could tolerate not being able to do user lookups. It might not be that big of a problem in reality? We could build a static-only glibc that disabled these uses of dlopen and used only the built-in NSS modules, perhaps only if LD_STATIC was 1, but I don't know if that would work correctly.

> > However, because you have two coordinating runtimes, the static one, and the
> > dynamic one being loaded, they must be identical runtimes to ensure the
> > coordination goes as planned. There is no direct isolation between the two
> > runtimes.
> 
> Yes, that's specifically the issue that I was trying to recall but it was
> too late be able to put the finger on --- a statically build gdbserver
> linked against an old RHEL sysroot for compatility can not dlopen a random
> libthread_db.so inside the container, as that will load a second,
> incompatible runtime.  Actually, isn't that a problem even in the
> dynamically linked case?

No, the dynamically linked case is different, let me explain.

In the static case you have runtime A statically linked into the application. When you go to load runtime B there could be a conflict.

In a dynamically linked application you always start with runtime B being used, you start with runtime B's dynamic loader, and coordination follows naturally, and the runtime provides backwards compatibility.

Hopefully that clarifies the situation.

I discuss below what happens if you are running runtime A and trying to load NSS modules from runtime B.

> > However, this goes against my recommendation which is not to make gdbserver
> > a static application, but to bundle it and run the bundle inside the target
> > container e.g. -Wl,--dynamic-loader=$ORIGIN/ld.so -Wl,-rpath=$ORIGIN/libs
> > etc. etc.
> 
> Ah, yes, somehow I got confused.  Sorry about that.
> 
> > You might even have a self-contained executable.
> 
> If we need to open libthread_db outside the box/sysroot for TLS debugging,
> we'll still have a problem, right?

No. The design for libthread_db is such that you can load it from anywhere, and it will use the ps_* symbols your implementation provides to introspect the inferior. Therefore it never cares about which runtime gdbserver is running, just the runtime it's inspecting with say ps_pglobal_lookup as it tries to verify if the inferior has a correctly matching libpthread.so.

If libthread_db did any introspection into the runtime being used by gdbserver, which it should not, then it might fail. We keep it simple enough and isolated that we guarantee you'll always be able to load it. If you can't, then it's a bug in the container's libthread_db.so and we need to look into it.

> > > libthread_db.so really needs to die :-/
> > 
> > No, you still have NSS, ICONV and IDN all which require dlopen in glibc.
> > While libthread_db.so can be converted to a data-driven design, NSS can't,
> > and is required for you to lookup permissions and other things. Therefore a
> > static gdbserver is simply not an option.
> 
> *NOD*

Unless gdbserver and all dependent libraries don't care about NSS, ICONV, and IDN? I doubt it though.
 
> But, does a gdbserver linked with an old sysroot need to load the NSS, etc.
> modules in the container (not those in the sysroot)?  If it does, can it be
> done in a safe, defined way, or do we run into two-runtimes-in-same-process
> issues?

Just to be clear, linking against the old sysroot gives you maximum runtime compatibility, but without bundling it might not work e.g. the system doesn't provide a required library like liblzma.so (xz-libs).

The gdbserver will always need to load the NSS modules in the container, it has to use the container configuration to work properly. The backwards compatibility requirements for glibc prevent us from breaking the NSS interface, but it has certainly not been bug-for-bug compatible, so version skew might have interesting consequences we may need to workaround.

At present glibc implements NSS v2 API. In some hypothetical future we may switch to NSS v3 API at which point the gdbserver's bundled runtimes may not know how to use NSS v3 APIs. The NSS v3 API change requires a SONAME bump for all NSS modules. Such a system though is required to also provide NSS v2 API interfaces in the form of *.so.2 modules for older runtimes to use. So gdbserver would continue to operate correctly.

> > The fundamental problem remains: The remote host in the container can be
> > anything. Either you build to run on anything (compile with an old-enough
> > sysroot), compile fully statically (rewriting glibc to provide NSS, ICONV,
> > and IDN via data-based isolation e.g. sockets to servers with services whose
> > protocols are known and documented), bundle your dependencies.
> > 
> 
> > So while libthread_db.so=>python or dwarf-description solves one problem, it
> > doesn't solve the others.
> 
> It would seem to solve the problem in the solution that Gary's now
> prototyping, which is to make GDB itself enter/exit the namespaces as
> needed.  nss, etc. are no longer a problem here, as gdb is running outside
> the container.  In this solution, the container's (potentially untrusted)
> libthread_db.so is _still_ what needs to be loaded into the host's GDB in
> order to support TLS.

Yes, entering the namespace in a limited way will resolve the need to run *in* the container.

Yes, you still need to load the untrusted libthrad_db.so, which poses a *massive* security risk. So large in fact that the running gdb must not be privileged or you have immediate privilege escalation issues. At which point the gdbserver solution starts looking more and more promising with hardening and isolation at the remote serial protocol (RSP) layer.

Hopefully my answers show that you can build a gdbserver with an old sysroot, and if the target container has all the required libs, it will run correctly. Alternatively you can bundle gdbserver with most of the runtime you need, minus NSS libraries, and still function correctly as you load the system NSS libraries.

Cheers,
Carlos.

Here is a workaround:

(gdb) set sysroot /proc/PID/root
(gdb) file /proc/PID/exe 
(gdb) attach PID

Obviously this only allows you to attach to one inferior, but it's a start.

Hi Milan, 

The contents of the Doc Text field live in the following commit in the 7.1_BETA branch of the Release Notes:

4c9a1bf Added the BZ#1186918 KI to Atomic.

Thanks!

Bug 1208444 is related to this issue.

What are the use cases of this request?  Isn't DTS (Developer Toolset) sufficient for that?
Backporting this patch is not much feasible, all the pathname handling in GDB has been reworked in the meantime, such a huge 60+ parts rebase would be be more regression risky than to use a rebased GDB (which is what DTS is).  Reimplementing the container support just for RHEL-7 would be even more regression risky.

Currently it looks RHEL-7.3 gdbserver will get some extension so that DTS GDB can attach to the processes fine.  Still for multi-threaded processes one will need to run gdbserver inside the container.  There is no possibility to make stock RHEL-7 GDB being able to attach to processes running in a container.

[patch] Suggest running gdbserver for a PID in container
https://sourceware.org/ml/gdb-patches/2016-03/msg00246.html

Test: GDB: Message for containers with inappropriate gdb -p:
docker run -ti docker.io/centos sleep 1h
+
/opt/.../gdb -p `pidof sleep`
->
warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable.  Connect to gdbserver inside the container.
This version of GDB needs local access to files from the container. It is easier to access the files from the container itself. This RHEL-7 version of GDB does not support such direct container files access.
The following versions of GDB support the direct access:
- Upstream version of GDB (unsupported) 7.10 or later
- Red Hat Developer Toolset (DTS) version of GDB from DTS 4.0 or later (available only for the x86_64 architecture)
The direct container file access needs to be supported also by gdbserver inside the container:
- Upstream version of gdbserver (unsupported) 7.10 or later
- Red Hat Developer Toolset (DTS) version of gdbserver from DTS 4.0 or later (only on x86_64)
- RHEL-7.3 version of gdbserver (on any architecture)

Test: gdbserver: One needs DTS 4.1 (or 4.0) GDB to test that.
RHEL-7.3 - in some different filesystem/container: gdbserver :1234 true
DTS-4.1: gdb -ex 'target remote :1234' -ex 'b main' -ex c -ex 'info shared'
FAIL:
Remote debugging using :1234
warning: Could not load vsyscall page because no executable was specified
try using the "file" command first.
0x00007ffff7ddc430 in ?? ()
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) _
PASS:
Breakpoint 1, 0x0000000000401320 in main ()
From                To                  Syms Read   Shared Object Library
0x00007ffff7ddbae0  0x00007ffff7df627a  Yes (*)     target:/lib64/ld-linux-x86-64.so.2
0x00007ffff7a393e0  0x00007ffff7b7cc10  Yes (*)     target:/lib64/libc.so.6
(*): Shared library is missing debugging information.
(gdb) _

(In reply to Jan Kratochvil from comment #35)
> Test: GDB: Message for containers with inappropriate gdb -p:
> docker run -ti docker.io/centos sleep 1h
> +
> /opt/.../gdb -p `pidof sleep`
> ->
> warning: Target and debugger are in different PID namespaces; thread lists
> and other data are likely unreliable.  Connect to gdbserver inside the
> container.
> This version of GDB needs local access to files from the container. It is
> easier to access the files from the container itself. This RHEL-7 version of
> GDB does not support such direct container files access.
> The following versions of GDB support the direct access:
> - Upstream version of GDB (unsupported) 7.10 or later
> - Red Hat Developer Toolset (DTS) version of GDB from DTS 4.0 or later
> (available only for the x86_64 architecture)
> The direct container file access needs to be supported also by gdbserver
> inside the container:
> - Upstream version of gdbserver (unsupported) 7.10 or later
> - Red Hat Developer Toolset (DTS) version of gdbserver from DTS 4.0 or later
> (only on x86_64)
> - RHEL-7.3 version of gdbserver (on any architecture)

This RHEL-7.3 extension has been dropped, there should be only the message:

warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable.  Connect to gdbserver inside the container.


> Test: gdbserver: One needs DTS 4.1 (or 4.0) GDB to test that.
> RHEL-7.3 - in some different filesystem/container: gdbserver :1234 true
> DTS-4.1: gdb -ex 'target remote :1234' -ex 'b main' -ex c -ex 'info shared'
> FAIL:
> Remote debugging using :1234
> warning: Could not load vsyscall page because no executable was specified
> try using the "file" command first.
> 0x00007ffff7ddc430 in ?? ()
> No symbol table is loaded.  Use the "file" command.
> Make breakpoint pending on future shared library load? (y or [n]) _
> PASS:
> Breakpoint 1, 0x0000000000401320 in main ()
> From                To                  Syms Read   Shared Object Library
> 0x00007ffff7ddbae0  0x00007ffff7df627a  Yes (*)    
> target:/lib64/ld-linux-x86-64.so.2
> 0x00007ffff7a393e0  0x00007ffff7b7cc10  Yes (*)     target:/lib64/libc.so.6
> (*): Shared library is missing debugging information.
> (gdb) _

This test is still valid/required/applies.

Hi Jan,

Could you please check the Release Notes text?

Thank you.

Fine with the Release Notes text, thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2328.html