Bug 819158 - bacula-fd crashing
bacula-fd crashing
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: bacula (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Simone Caronni
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-05 05:44 EDT by Tomasz Torcz
Modified: 2012-06-11 20:36 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-11 20:36:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
bacula-fd.conf (958 bytes, text/plain)
2012-06-05 14:03 EDT, Tomasz Torcz
no flags Details
bacula-dir.conf (11.26 KB, text/plain)
2012-06-05 14:03 EDT, Tomasz Torcz
no flags Details

  None (edit)
Description Tomasz Torcz 2012-05-05 05:44:01 EDT
Description of problem:
Ever since upgrading to F17, bacula-fd on my laptop crashes during backup.

Version-Release number of selected component (if applicable):
bacula-client-5.2.6-2.fc17.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Start any backup job.
2.
3.
  
Actual results:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff13a0700 (LWP 15539)]
sm_free (file=0x4277c8 "xattr.c", line=1323, fp=0x5555555555555555) at smartall.c:192
192	   if (!head->abin_use) {

(gdb) bt
#0  sm_free (file=0x4277c8 "xattr.c", line=1323, fp=0x5555555555555555) at smartall.c:192
#1  0x0000000000420a44 in generic_xattr_build_streams (jcr=0x7fffec0008e8, ff_pkt=<optimized out>) at xattr.c:1323
#2  0x000000000040dc0b in save_file (jcr=0x7fffec0008e8, ff_pkt=0x7fffec0a62f8, top_level=<optimized out>) at backup.c:732
#3  0x0000003ec5c0655f in find_one_file (jcr=jcr@entry=0x7fffec0008e8, ff_pkt=ff_pkt@entry=0x7fffec001278, handle_file=handle_file@entry=
    0x3ec5c03c90 <our_callback(JCR*, FF_PKT*, bool)>, fname=fname@entry=0x7fffec003448 "/home/fisia/.config/dconf", parent_device=140737153268920, 
    parent_device@entry=18, top_level=top_level@entry=false) at find_one.c:780
#4  0x0000003ec5c064dd in find_one_file (jcr=jcr@entry=0x7fffec0008e8, ff_pkt=ff_pkt@entry=0x7fffec001278, handle_file=handle_file@entry=
    0x3ec5c03c90 <our_callback(JCR*, FF_PKT*, bool)>, fname=fname@entry=0x7fffec003658 "/home/fisia/.config", parent_device=140737153489224, parent_device@entry=18, 
    top_level=top_level@entry=false) at find_one.c:763
#5  0x0000003ec5c064dd in find_one_file (jcr=jcr@entry=0x7fffec0008e8, ff_pkt=ff_pkt@entry=0x7fffec001278, handle_file=handle_file@entry=
    0x3ec5c03c90 <our_callback(JCR*, FF_PKT*, bool)>, fname=fname@entry=0x7fffec003928 "/home/fisia", parent_device=140737153378264, parent_device@entry=18, 
    top_level=top_level@entry=false) at find_one.c:763
#6  0x0000003ec5c064dd in find_one_file (jcr=0x7fffec0008e8, ff_pkt=0x7fffec001278, handle_file=0x3ec5c03c90 <our_callback(JCR*, FF_PKT*, bool)>, fname=
    0x7fffec002028 "/home", parent_device=140737153342968, top_level=<optimized out>) at find_one.c:763
#7  0x0000003ec5c042fc in find_files (jcr=<optimized out>, ff=0x7fffec001278, file_save=<optimized out>, plugin_save=0x410420 <plugin_save(JCR*, FF_PKT*, bool)>)
    at find.c:217
#8  0x000000000040b46e in blast_data_to_storage_daemon (jcr=0x7fffec0008e8, addr=<optimized out>) at backup.c:188
#9  0x0000000000415bd3 in backup_cmd (jcr=0x7fffec0008e8) at job.c:1937
#10 0x0000000000417934 in handle_client_request (dirp=0x853948) at job.c:289
#11 0x0000003ec443ff7b in workq_server (arg=0x62ce20) at workq.c:346
#12 0x0000003ec2c07d14 in start_thread (arg=0x7ffff13a0700) at pthread_create.c:309
#13 0x0000003ec28f194d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Various backtraces show it crashes on different files.


Expected results:
Working backup.

Additional info:
Comment 1 Simone Caronni 2012-05-21 11:38:45 EDT
Hello,

can you test this build?

http://koji.fedoraproject.org/koji/buildinfo?buildID=319894

No changes, but it's a rebuild in more recent f17 times.

Thanks,
--Simone
Comment 2 Tomasz Torcz 2012-05-21 15:02:35 EDT
No changes, the same crash.
Comment 3 Simone Caronni 2012-05-23 03:10:07 EDT
Hello,

this is the answer on the bacula mailing list:

http://marc.info/?l=bacula-devel&m=133775686222165&w=2

Can you try what's written there (turning off acl/xattr support) and test again? If that's the case I will ask on fedora-devel how we can circumvent stack protector options.

Thanks,
--Simone
Comment 4 Simone Caronni 2012-05-23 03:13:02 EDT
...and file the bug on bacula, as suggested.

Please attach your bacula-dir.conf and bacula-fd.conf files.
Comment 5 Tomasz Torcz 2012-05-23 16:46:05 EDT
Disabling xattrs support makes the backup succeed, however this is suboptimal.
I see this issue on one machine only. The filesystem to be backed up is btrfs, if that matters.
From bacula log file I see that previous successful backup was at 02-Feb-12. It was around time bacula-5.2.6 replaced bacula-5.0.x in Fedora.
Comment 6 Simone Caronni 2012-05-24 03:02:53 EDT
Hello,

bacula 5.2 replaced bacula 5.0 in Fedora (rawhide and then 17) many many months ago, so probably that's the date you upgraded your system.

Bacula 5.2 introduced a lot of changes related to ACLs and Extended Attributes; including the option to backup file capabilities in 5.2.4.

http://www.bacula.org/en/?page=news
http://bacula.org/5.2.x-manuals/en/main/main/Bacula_Main_Reference.html

The fact that you experience the crash only on one system and that system is btrfs can be a pointer, maybe bacula has not been tested in such an environment. I'll forward such information on the mailing list; please attach here your bacula-dir.conf and bacula-fd.conf.

If you miss file capabilities upon a restore; you can use this command to see which need to be restored.

rpm --qf "[%{FILECAPS} %{FILENAMES}\n]" -qa | grep '^='
Comment 7 Simone Caronni 2012-05-24 08:22:56 EDT
Hello,

can you test the build here?

http://slaanesh.fedorapeople.org/819158/

This contains a small patch provided directly by the Bacula developers. If it works I'll push the update.

Thanks,
--Simone
Comment 8 Fedora Update System 2012-05-28 09:36:25 EDT
bacula-5.2.6-5.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/bacula-5.2.6-5.fc17
Comment 9 Simone Caronni 2012-05-28 09:37:44 EDT
Please test if the update fixes your problem and submit a karma feedback if it does.

yum --enablerepo=updates-testing update bacula*


Thanks,
--Simone
Comment 10 Fedora Update System 2012-05-29 06:25:19 EDT
Package bacula-5.2.6-5.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing bacula-5.2.6-5.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-8532/bacula-5.2.6-5.fc17
then log in and leave karma (feedback).
Comment 11 Tomasz Torcz 2012-06-02 16:41:30 EDT
Hi,

sorry for lack of feedback - I was vacationing last week.

Unfortunately, bacula-client-5.2.6-5.fc17.x86_64 do not fix the issue. Traceback is following:

(gdb) thread apply all bt

Thread 4 (Thread 0x7fffebfff700 (LWP 30903)):
#0  0x00000032450ead03 in select () at ../sysdeps/unix/syscall-template.S:82
#1  0x0000003c3a01b28e in BSOCK::wait_data_intr (this=0x7fffe40008e8, sec=<optimized out>, usec=<optimized out>) at bsock.c:891
#2  0x0000000000413ae9 in sd_heartbeat_thread (arg=0x7fffec0008e8) at heartbeat.c:77
#3  0x0000003245407d14 in start_thread (arg=0x7fffebfff700) at pthread_create.c:309
#4  0x00000032450f199d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7ffff119b700 (LWP 30902)):
#0  sm_free (file=0x4277c8 "xattr.c", line=1324, fp=0x5555555555555555) at smartall.c:192
#1  0x0000000000420a44 in generic_xattr_build_streams (jcr=0x7fffec0008e8, ff_pkt=<optimized out>) at xattr.c:1324
#2  0x000000000040dc0b in save_file (jcr=0x7fffec0008e8, ff_pkt=0x7fffec003488, top_level=<optimized out>) at backup.c:732
#3  0x0000003c3980655f in find_one_file (jcr=jcr@entry=0x7fffec0008e8, ff_pkt=ff_pkt@entry=0x7fffec001278, handle_file=handle_file@entry=0x3c39803c90 <our_callback(JCR*, FF_PKT*, bool)>, 
    fname=fname@entry=0x7fffec002e78 "/etc/rpm", parent_device=140737153304216, parent_device@entry=18, top_level=top_level@entry=false) at find_one.c:780
#4  0x0000003c398064dd in find_one_file (jcr=0x7fffec0008e8, ff_pkt=0x7fffec001278, handle_file=0x3c39803c90 <our_callback(JCR*, FF_PKT*, bool)>, fname=0x7fffec001fd8 "/etc", 
    parent_device=140737153268920, top_level=<optimized out>) at find_one.c:763
#5  0x0000003c398042fc in find_files (jcr=<optimized out>, ff=0x7fffec001278, file_save=<optimized out>, plugin_save=0x410420 <plugin_save(JCR*, FF_PKT*, bool)>) at find.c:217
#6  0x000000000040b46e in blast_data_to_storage_daemon (jcr=0x7fffec0008e8, addr=<optimized out>) at backup.c:188
#7  0x0000000000415bd3 in backup_cmd (jcr=0x7fffec0008e8) at job.c:1937
#8  0x0000000000417934 in handle_client_request (dirp=0x853958) at job.c:289
#9  0x0000003c3a03ff7b in workq_server (arg=0x62ce20) at workq.c:346
#10 0x0000003245407d14 in start_thread (arg=0x7ffff119b700) at pthread_create.c:309
#11 0x00000032450f199d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7ffff199c700 (LWP 30890)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:218
#1  0x0000003c3a03f79d in watchdog_thread (arg=<optimized out>) at watchdog.c:321
#2  0x0000003245407d14 in start_thread (arg=0x7ffff199c700) at pthread_create.c:309
#3  0x00000032450f199d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7ffff7dcb800 (LWP 30887)):
#0  0x00000032450ead03 in select () at ../sysdeps/unix/syscall-template.S:82
#1  0x0000003c3a0178d4 in bnet_thread_server (addrs=0x837478, max_clients=20, client_wq=0x62ce20, handle_client_request=0x4176d0 <handle_client_request(void*)>) at bnet_server.c:160
#2  0x000000000040903d in main (argc=<optimized out>, argv=<optimized out>) at filed.c:278
Comment 12 Simone Caronni 2012-06-04 12:16:53 EDT
Hello,

I've pushed Bacula 5.2.7 to testing, if you want to give it a try but I doubt it will solve the problems.

On the bacula bug tracker I was asked if the File Daemon is running unprivileged.

Are you're running it with "-k" in the command line?

Can you also please attach your config files as requested on comment #4 and comment #9? Asking questions and waiting for answers makes the debugging more complicated.

I need these files:

/etc/bacula/bacula-dir.conf
/etc/bacula/bacula-fd.conf
/etc/sysconfig/bacula-fd

Thanks,
--Simone
Comment 13 Simone Caronni 2012-06-04 12:27:30 EDT
I've made a note about the privileges option in the config files so it will get in into the next version:

http://pkgs.fedoraproject.org/gitweb/?p=bacula.git;a=commitdiff;h=9dc3c9b46e8a63450cb805bf76157e6a5c511706
Comment 14 Tomasz Torcz 2012-06-05 14:03:23 EDT
Created attachment 589595 [details]
bacula-fd.conf

bacula-fd runs as root (i.e. no "-k" option used)
Comment 15 Tomasz Torcz 2012-06-05 14:03:51 EDT
Created attachment 589599 [details]
bacula-dir.conf
Comment 16 Simone Caronni 2012-06-06 14:27:43 EDT
Hello,

I've included a patch from the Bacula developers that overhaul the xattr code, can you please test this build? The developers have made tests with F17 on a btrfs root and say they fixed the issue:

http://koji.fedoraproject.org/koji/buildinfo?buildID=322378

There's a change in this package, the symlink library name has changed because we discovered a bug during version upgrades.

Please issue the following before upgrading:

alternatives --remove /usr/lib64/libbaccats-mysql-5.2.6.so
alternatives --remove /usr/lib64/libbaccats-sqlite3-5.2.6.so
alternatives --remove /usr/lib64/libbaccats-postgresql-5.2.6.so

And then set again your preference database with:

# alternatives --config libbaccats.so

There are 3 programs which provide 'libbaccats.so'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib64/libbaccats-mysql.so
 + 2           /usr/lib64/libbaccats-sqlite3.so
*  3           /usr/lib64/libbaccats-postgresql.so

Enter to keep the current selection[+], or type selection number:

Thanks,
--Simone
Comment 17 Tomasz Torcz 2012-06-06 15:18:11 EDT
Success! With this build, backup with xattrs enabled finished without errors.
Thank you for all the time spent on this case.
Comment 18 Fedora Update System 2012-06-08 04:37:13 EDT
bacula-docs-5.2.7-1.fc17,bacula-5.2.7-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/bacula-docs-5.2.7-1.fc17,bacula-5.2.7-4.fc17
Comment 19 Fedora Update System 2012-06-08 20:08:09 EDT
Package bacula-docs-5.2.7-1.fc17, bacula-5.2.7-4.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing bacula-docs-5.2.7-1.fc17 bacula-5.2.7-4.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9159/bacula-docs-5.2.7-1.fc17,bacula-5.2.7-4.fc17
then log in and leave karma (feedback).
Comment 20 Simone Caronni 2012-06-11 05:55:27 EDT
5.2.8 has come out upstream with many other bugs fixed and includes already the latest xattr patch.

I introduced a change in the spec file that checks for the obsolete symlinks for bacula-dir, bacula-sd (>=F16) and libbaccats-backend-version.so (=F17) and sets the new one accordingly so the change should not be done manually.

So even an upgrade from F16 to F17 should be safe and retain the preferences.

Excerpt from the spec file:

# Fix for automatic selection of backends during upgrades
if readlink /etc/alternatives/libbaccats.so | grep --silent mysql || \
   readlink /etc/alternatives/bacula-dir | grep --silent mysql || \
   readlink /etc/alternatives/bacula-sd | grep --silent mysql; then
   /usr/sbin/alternatives --set libbaccats.so %{_libdir}/libbaccats-mysql.so
elif readlink /etc/alternatives/libbaccats.so | grep --silent sqlite || \
   readlink /etc/alternatives/bacula-dir | grep --silent sqlite || \
   readlink /etc/alternatives/bacula-sd | grep --silent sqlite; then
   /usr/sbin/alternatives --set libbaccats.so %{_libdir}/libbaccats-sqlite3.so
else
   /usr/sbin/alternatives --set libbaccats.so %{_libdir}/libbaccats-postgresql.so
fi

Regards,
--Simone
Comment 21 Fedora Update System 2012-06-11 06:17:59 EDT
bacula-docs-5.2.8-1.fc17,bacula-5.2.8-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/bacula-docs-5.2.8-1.fc17,bacula-5.2.8-1.fc17
Comment 22 Fedora Update System 2012-06-11 20:36:18 EDT
bacula-docs-5.2.8-1.fc17, bacula-5.2.8-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.