Bug 436453

Michael: thanks

I haven't seen your "doesn't always see the disks" issue before, so please do
log a bug on that

Adding to PvOpsTracker ..

I have logged the disk issue as bug 436493 . I am guessing that for the current
bug that yum isn't directly involved, it just produces sort of conditions for
the problem to show itself. It has also only occurred so far during one yum run,
though the crash was repeated a few times, but I might get a better idea of how
reproducible it is when the next batch of updates appears.

Confirmed; just saw this myself during a yum update of an x86_32 guest

Okay, I now have a reliable reproducer with a fully up to date guest.

Just running:

  /usr/lib/nss/unsupported-tools/shlibsign -i /lib/libsoftokn3.so 

seems to trigger it for me

Both armbru and I also saw crond trigger it when running cron.hourly

Further narrowed it down to any dlopen() on a library whose first segment is to
be loaded at a reasonably high virtual address

i.e. running this:

#include <dlfcn.h>

int main(int argc, char **argv)
{
  return dlclose(dlopen(argv[1], RTLD_LAZY));
}

against any of the libraries returned by:

for iii in /lib/lib*.so.* /usr/lib/lib*.so.*; do [ -L $iii ] && continue;
v=$(eu-readelf -l $iii | awk '/LOAD/ { if (match($3, "[^0x]")) {print $3}
exit}') ; [ "$v" ] && echo $iii $v; done | sort -n

Further notes:

  - This is perfectly reproducible on stock 2.6.25-rc6 pv_ops xen; so
    we can rule out the x86_64 xen patches as the cause

  - prelink is what's causing these libs to have a non-zero base load
    address; if I "prelink -u" a lib first, then it can be dlopen()ed
    without a problem

  - the oops occurs during dlclose() when we try to munmap() the base
    load address

  - I've only reproduced this on x86_32 so far, but I can't seem to get
    prelink to relocate libs on x86_64, so perhaps it is really an issue
    there too

An even simpler test case:

--
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

#define MMAP_ADDR_GOOD (void *)0x3ffff000
#define MMAP_ADDR_BAD  (void *)0x40000000
#define MMAP_LEN 0x1000

int
main(int argc, char **argv)
{
  int fd = open("/dev/zero", O_RDONLY);
  munmap(mmap(MMAP_ADDR_GOOD, MMAP_LEN, PROT_READ, MAP_PRIVATE, fd, 0), MMAP_LEN);
  printf("Mapping to %p succeeded\n", MMAP_ADDR_GOOD);
  munmap(mmap(MMAP_ADDR_BAD, MMAP_LEN, PROT_READ, MAP_PRIVATE, fd, 0), MMAP_LEN);
  printf("Mapping to %p succeeded\n", MMAP_ADDR_BAD);
  close(fd);
  return 0;
}
--

The BUG occurs on the second munmap()

Posted a fix for this to lkml, see:

  http://lkml.org/lkml/2008/3/28/286

Awaiting some upstream feedback before building in rawhide.

Should be fixed now in kernel-xen-2.6-2.6.25-0.12.rc7.git6.fc9