Bug 699189

I see the same versions of cronie in dist-f16 and dist-f15. What exact cronie rpm version was segfaulting and what works for you?

Also a backtrace would be nice.

I get this too, it segfaults every time it wants to execute a crontab. If I schedule something to run every minute, it segfaults every minute. If I schedule something to run every 5, it segfaults every 5 and so on.

Packages installed: 
cronie-debuginfo-1.4.7-1.fc15.x86_64
cronie-1.4.7-1.fc15.x86_64
cronie-anacron-1.4.7-1.fc15.x86_64

Cleanly installed Fedora 15 system, fully updated as of 5 minutes ago.

Backtrace: 

#0  0x00007f6c49b0d82e in __pthread_initialize_minimal_internal () at nptl-init.c:295
#1  0x00007f6c49b0c1f9 in _init () from /lib64/libpthread-2.13.90.so
#2  0x00007f6c49f36868 in ?? () from /lib64/security/pam_env.so
#3  0x00007f6c519d89eb in call_init (l=0x1c57e90, argc=2, argv=0x7fff45d62b88, env=0x7fff45d62ba0) at dl-init.c:68
#4  0x00007f6c519d8be9 in call_init (env=0x7fff45d62ba0, argv=0x7fff45d62b88, argc=2, l=<optimized out>) at dl-init.c:50
#5  _dl_init (main_map=0x1c56940, argc=2, argv=0x7fff45d62b88, env=0x7fff45d62ba0) at dl-init.c:98
#6  0x00007f6c519dcdc2 in dl_open_worker (a=<optimized out>) at dl-open.c:464
#7  0x00007f6c519d88a6 in _dl_catch_error (objname=0x7fff45d61848, errstring=0x7fff45d61850, mallocedp=0x7fff45d6185f, operate=0x7f6c519dca30 <dl_open_worker>, args=0x7fff45d61800) at dl-error.c:178
#8  0x00007f6c519dd30a in _dl_open (file=0x1c55430 "/lib64/security/pam_env.so", mode=-2147483646, caller_dlopen=0x7f6c515a2ab6, nsid=-2, argc=2, argv=<optimized out>, env=0x7fff45d62ba0) at dl-open.c:555
#9  0x00007f6c5139af26 in dlopen_doit (a=0x7fff45d61a20) at dlopen.c:67
#10 0x00007f6c519d88a6 in _dl_catch_error (objname=0x7f6c5159d0d0, errstring=0x7f6c5159d0d8, mallocedp=0x7f6c5159d0c8, operate=0x7f6c5139aec0 <dlopen_doit>, args=0x7fff45d61a20) at dl-error.c:178
#11 0x00007f6c5139b52f in _dlerror_run (operate=0x7f6c5139aec0 <dlopen_doit>, args=0x7fff45d61a20) at dlerror.c:164
#12 0x00007f6c5139afc1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:88
#13 0x00007f6c515a2ab6 in _pam_load_module (pamh=0x1c51020, mod_path=0x1c55430 "/lib64/security/pam_env.so", handler_type=0) at pam_handlers.c:680
#14 0x00007f6c515a2da8 in _pam_add_handler (pamh=0x1c51020, handler_type=0, other=0, stack_level=0, type=1, actions=0x7fff45d61b90, mod_path=0x7fff45d61c4a "pam_env.so", argc=0, argv=0x0, argvlen=0) at pam_handlers.c:766
#15 0x00007f6c515a38d8 in _pam_parse_conf_file (pamh=<optimized out>, f=0x1c55180, known_service=0x1c511a0 "crond", requested_module_type=1, stack_level=0) at pam_handlers.c:263
#16 0x00007f6c515a3ab2 in _pam_load_conf_file (stack_level=0, module_type=1, service=0x1c511a0 "crond", config_name=<optimized out>, pamh=0x1c51020) at pam_handlers.c:315
#17 _pam_parse_conf_file (pamh=<optimized out>, f=0x1c5d9f0, known_service=0x1c511a0 "crond", requested_module_type=0, stack_level=0) at pam_handlers.c:207
#18 0x00007f6c515a3f28 in _pam_init_handlers (pamh=0x1c51020) at pam_handlers.c:414
#19 0x00007f6c515a581c in pam_start (service_name=<optimized out>, user=0x1c5df30 "root", pam_conversation=0x60d900, pamh=0x64e270) at pam_start.c:127
#20 0x0000000000409a53 in cron_start_pam (pw=0x1c5df00) at security.c:175
#21 0x0000000000409c6c in cron_set_job_security_context (e=0x1c5df70, u=0x1c5c240, jobenv=0x7fff45d62948) at security.c:106
#22 0x0000000000406930 in do_command (e=0x1c5df70, u=0x1c5c240) at do_command.c:52
#23 0x0000000000405a3b in job_runqueue () at job.c:60
#24 0x0000000000403845 in main (argc=808, argv=0x7fff014b7ac1) at cron.c:414

What is the output of 'rpm -q glibc kernel pam' ?

Erik, do you have updates-testing repository enabled?

And output of 'rpm -q libselinux' as well.

Also do the segfaults happen only immediately after yum upgrade or also after the reboot?

Yes, updates-testing is enabled on this machine.

Here's the output of the above command:
glibc-2.13.90-9.x86_64
glibc-2.13.90-9.i686
kernel-2.6.38.2-9.fc15.x86_64
kernel-2.6.38.3-18.fc15.x86_64
pam-1.1.3-8.fc15.x86_64
pam-1.1.3-8.fc15.i686
libselinux-2.0.99-4.fc15.x86_64
libselinux-2.0.99-4.fc15.i686

Following a reboot, this has stopped segfaulting.

For clarity, although it may be unrelated; I should mention that I was running the -9 kernel, and booted into the -18 one now when it stopped crashing.

OK, this is clear now. It was caused by upgrade of glibc where it changed some pthreads internals. The crond was running with the old glibc but the dlopened pam modules loaded new libpthread which made the cron child process to crash.

The only remedy as I see it is to put crond restart in a %trigger on glibc, pam, and libselinux upgrades.

I've added the trigger in cronie-1.4.7-2.fc16.