Bug 1304106 - dlopen deadlocks during thread cancellation
dlopen deadlocks during thread cancellation
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: glibc (Show other bugs)
6.9
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: glibc team
qe-baseos-tools
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-02 17:10 EST by Anit Lulla
Modified: 2017-08-29 16:27 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-29 16:27:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
C++ source code that reproduces the bug (1.70 KB, text/plain)
2016-02-02 17:10 EST, Anit Lulla
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Sourceware 22034 None None None 2017-08-29 16:27 EDT

  None (edit)
Description Anit Lulla 2016-02-02 17:10:19 EST
Created attachment 1120575 [details]
C++ source code that reproduces the bug

Description of problem:
dlopen() deadlocks when a prior dlopen() call is cancelled via pthread_cancel()

Version-Release number of selected component (if applicable):
All versions

How reproducible:
Always (see reproducer, which is attached)

#include <iostream>
#include <unistd.h>
#include <pthread.h>
#include <dlfcn.h>

using namespace std;

// to reproduce (copy contents to dl_deadlock.cpp)
//
// $ g++ -D FOO_SHARE_LIB -shared -o libfoo.so -fPIC dl_deadlock.cpp
// $ g++ -g dl_deadlock.cpp -ldl -lpthread
// $ ./a.out

#ifdef FOO_SHARE_LIB
class Foo
{
  public:
  Foo() {
    cout << "construct foo" << endl;
    usleep(500000);
    cout << "done construct foo" << endl;
  }
};
Foo g_foo;

#else

extern "C" void *thrfxn_init(void *arg)
{ usleep(500000); return NULL; }

extern "C" void *thrfxn(void *arg)
{
  cout << "before dlopen" << endl;
  void *h = dlopen("./libfoo.so",  RTLD_GLOBAL | RTLD_NOW);
  cout << "after dlopen" << endl;
  if (h);
    dlclose(h);
}

int main()
{
  pthread_t thr;

  // need to call cancel so pthread_cancel_init is called,
  // which calls into libdl (needed since first pthread_cancel waits on dlopen)
  pthread_create(&thr, NULL, thrfxn_init, NULL);
  pthread_cancel(thr);
  pthread_join(thr, NULL);
  
  cout << "before thread create" << endl;
  pthread_create(&thr, NULL, thrfxn, NULL);
  cout << "after thread create" << endl;

  // sleep 5ms
  usleep(5000);

  // cancel thread that is waiting in dlopen
  cout << "before cancel" << endl;
  pthread_cancel(thr);
  cout << "after cancel" << endl;

  // this dlopen will now deadlock
  void *h = dlopen("/lib64/libc.so.6", RTLD_GLOBAL | RTLD_NOW);
  if (h)
    dlclose(h);

  //find_symbol("PA_REPORT_CALC_DISPATCH", PA_REPORT_CALC_DISPATCH);
  //FDSA_FQL_SERVICE, CREATE_FDSA_FQL_SERVICE;

  cout << "before join" << endl;
  pthread_join(thr, NULL);
  cout << "after join" << endl;

  return 0;
}

#endif

Steps to Reproduce:
1. g++ -D FOO_SHARE_LIB -shared -o libfoo.so -fPIC dl_deadlock.cpp
2. g++ -g dl_deadlock.cpp -ldl -lpthread
3. ./a.out

Actual results:
before thread create
after thread create
before dlopen
construct foo
before cancel
after cancel
-- deadlock --

Expected results:
before thread create
after thread create
before dlopen
construct foo
before cancel
after cancel
done construct foo
after dlopen
before join
-- program completion --

Additional info:
Comment 3 Carlos O'Donell 2016-11-22 12:11:56 EST
I'm moving this to rhel-6.10 for review. We aren't going to look at this for rhel-6.9. There are several cases which we have fixed where the error paths out of dlopen failed to successfully rollback the changes for intermediate loads, but it's likely we haven't done this for all the cases possible. It would be interesting if this reproduces in upstream master.
Comment 5 Carlos O'Donell 2017-08-29 16:27:21 EDT
Constructors are foreign function calls which happen while the dynamic loader is loading the shared library, as such, there are some things which don't work during this particular stage of loading. In this case the foreign function has triggered deferred cancellation and has left the internal dynamic loader locks taken. One fix would be to push a cleanup function in dlopen which releases the locks if the foreign functions call a cancellation point. Unfortunately this kind of fix is invasive for RHEL6, and upstream still doesn't pass this test.

I've opened upstream bug 22034 for this to be tracked upstream.

Note You need to log in before you can comment on or make changes to this bug.