Bug 830358

Summary: bacula-dir fails as upgrade installs PostgreSQL version on MySQL based system
Product: [Fedora] Fedora Reporter: Wolfgang Denk <wd>
Component: baculaAssignee: Simone Caronni <negativo17>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 17CC: andreas, fschwarz, limburgher, lnykryn, negativo17, vanmeeuwen+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-11 20:36:11 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Wolfgang Denk 2012-06-08 18:40:06 EDT
Description of problem:

After upgrading from Fedora 16 tp Fedora 17, bacula-dir fails to start.
It complains:

bacula-dir[5650]: bacula-dir: dird.c:959 postgresql.c:248 Unable to connect to PostgreSQL server. Database=bacula User=bacula

This is to be expected, as the system is using MySQL as DB.  For some reason, the upgrade installs the PostgreSQL version of the tools, rsp. fails to configure these as needed.

Version-Release number of selected component (if applicable):

bacula-client-5.2.6-2.fc17.x86_64
bacula-common-5.2.6-2.fc17.x86_64
bacula-console-5.2.6-2.fc17.x86_64
bacula-console-bat-5.2.6-2.fc17.x86_64
bacula-director-5.2.6-2.fc17.x86_64
bacula-docs-5.2.6-1.fc17.noarch
bacula-libs-5.2.6-2.fc17.x86_64
bacula-libs-sql-5.2.6-2.fc17.x86_64
bacula-storage-5.2.6-2.fc17.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Start with a working installation using a MySQL database on a Fedora 16
2. update to Fedora Fedora 17
  
Actual results:

Bacula fails because it is assuming a PostgreSQL DB.

Expected results:

Bacula should continue to work, using the existing MySQL DB.

Additional info:

This is quite a killer as it totally ruins your backup servers...
Comment 1 Wolfgang Denk 2012-06-08 18:59:39 EDT
Adding a line

      dbdriver = "dbi:mysql"; dbaddress = 127.0.0.1; dbport = 3306

to bacula-dir.conf does not help.

I notice that the bacula domumentation (http://www.bacula.org/manuals/en/concepts/concepts/New_Features.html) says:

    It is important to remember, when compiling Bacula with libdbi, the following packages are needed:

    libdbi version 1.0.0, http://libdbi.sourceforge.net/
    libdbi-drivers 1.0.0, http://libdbi-drivers.sourceforge.net/

However, Fedora 17 ships with older versions:

libdbi-dbd-mysql-0.8.3-9.fc17.x86_64
libdbi-drivers-0.8.3-9.fc17.x86_64
libdbi-0.8.3-5.fc17.x86_64


Don't know if this is the cause of the problem here.
Comment 2 Wolfgang Denk 2012-06-08 19:12:49 EDT
OK, the relevant information can be found here:

http://repos.fedorapeople.org/repos/slaanesh/bacula/README.txt

See section notes.  The trichk is that you need to run "alternatives --config libbaccats.so" to configure for the correct database.

This should be done automatically when updating to the new packages that require such configuration.  At least, a clear error message in the system log would have been helpful - add at least some comment to the bacula-dir.conf config file, please.


Arghhh... this is a step forward, leading into the next breakage:

dird.c:959 Version error for database "bacula". Wanted 14, got 12

Why has the DB version not be auto-upgraded???

Has *anybody* ever tested an update from F16 to F17? Hello, do we have any QA here?
Comment 3 Simone Caronni 2012-06-11 04:08:19 EDT
Hello,

starting from 5.0.3, there were multiple binaries built from the same source inside the package, lots of symlinks in the alternatives system and lot of patches to support the various database backends.

Starting from 5.2.x, upstream developers moved all the common code into libraries and made the selection of the database through a single symlink on a versioned library.

At the beginning this was done with multiple versioned libraries offering a single shared object name and tricks like that. This made packaging almost impossible (packages requiring libraries contained in themselves, yum going crazy) and ended up with the developers changing everything a couple of times (and the spec file followed, with %ghost, etc.).

Up to Fedora 16 release the package was not in a very good shape and an overhaul was introduced with rawhide; you can see how complex the package is by looking at the spec file and the changes introduced at:

http://pkgs.fedoraproject.org/gitweb/?p=bacula.git;a=summary

In the latest incarnation of the package that was released with Fedora 17 (5.2.6-2.fc17 - unchaged for a long time), the backend was chosen with an alternative (+ a slave):

# alternatives --config libbaccats.so

There are 3 programs which provide 'libbaccats.so'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib64/libbaccats-mysql-5.2.6.so
 + 2           /usr/lib64/libbaccats-sqlite3-5.2.6.so
*  3           /usr/lib64/libbaccats-postgresql-5.2.6.so

Enter to keep the current selection[+], or type selection number:

In updates testing you can find 5.2.7-4.fc17; it closes a few Fedora specific bugs and many upstream ones, one of which is #829219; regarding the selection of libraries, so it has changed again.

Please remove your old links:

alternatives --remove libbaccats.so /usr/lib64/libbaccats-mysql-5.2.6.so
alternatives --remove libbaccats.so /usr/lib64/libbaccats-sqlite3-5.2.6.so
alternatives --remove libbaccats.so /usr/lib64/libbaccats-postgresql-5.2.6.so

And then set again your preference database with:

# alternatives --config libbaccats.so

There are 3 programs which provide 'libbaccats.so'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib64/libbaccats-mysql.so
 + 2           /usr/lib64/libbaccats-sqlite3.so
*  3           /usr/lib64/libbaccats-postgresql.so

Enter to keep the current selection[+], or type selection number:

Regarding the database upgrade, this is done by hand upstream and I don't think is a good idea to have it automated; you should perform a backup before and you could have permission problems and such. And historically, Fedora never performed the upgrade automatically, also from Fedora 12' Bacula 3.0.x to Fedora 13 Bacula 5.0.x.

To update your database, as pointed out in the Bacula Docs, you just need to:

# cd /usr/libexec/bacula/
# ./update_bacula_tables mysql

There's no change in your configuration files at /etc/bacula/*.conf for the upgrade.

btw, after all of this, I agree with you that a mention in the release notes should have been added prior to release.

Regards,
--Simone
Comment 4 Simone Caronni 2012-06-11 05:07:47 EDT
Hello,

5.2.8 has come out upstream with many other bugs fixed.

I introduced a change in the spec file that checks for the obsolete symlinks for bacula-dir, bacula-sd (>=F16) and libbaccats-backend-version.so (=F17) and sets the new one accordingly so the change should not be done manually.

So even an upgrade from F16 to F17 should be safe and retain the preferences.

Regards,
--Simone
Comment 5 Wolfgang Denk 2012-06-11 05:51:26 EDT
(In reply to comment #3)
> 
> starting from 5.0.3, there were multiple binaries built from the same source
> inside the package, lots of symlinks in the alternatives system and lot of
> patches to support the various database backends.
> 
> Starting from 5.2.x, upstream developers moved all the common code into
> libraries and made the selection of the database through a single symlink on
> a versioned library.

I understand where we are coming from, and that the changes actually
make sense.

> In updates testing you can find 5.2.7-4.fc17; it closes a few Fedora
> specific bugs and many upstream ones, one of which is #829219; regarding the
> selection of libraries, so it has changed again.

Thanks for pointing out - when will this be pushed into the normal
repos?

> Please remove your old links:
> 
> alternatives --remove /usr/lib64/libbaccats-mysql-5.2.6.so
> alternatives --remove /usr/lib64/libbaccats-sqlite3-5.2.6.so
> alternatives --remove /usr/lib64/libbaccats-postgresql-5.2.6.so

This doesn't work - I get a usage message because the "libbaccats.so"
argument is missing...

> And then set again your preference database with:

It is this what I'm complaining about - the user is supposed to
perform a number of manual actions, which are (1) poorly or not
documented and (2) not integrated into the update process.

Please keep in mind how the normal process for Fedora packages works:
yous ee there are new versions available, you think the updates are
good for you, you install them.  If there are no errors reported, you
assume everything is fine.

Bacula breaks with these expectations: the update is started without
any indication that it might be dangerous, and it completes without
any warnings or error messages.  But it silently breaks your backup
server.

This is something that should never happen.

I think there are several areas of problems:

1) The update procedure is not complete and has never actually been
   tested with automatic updates in mind.  Why should I manually have
   to delete any old links, or manually ran commands to select the
   right alternatives?  Why isn't this done automatically as part of
   the pre- resp. postinstall scripts?

> Regarding the database upgrade, this is done by hand upstream and I don't
> think is a good idea to have it automated; you should perform a backup
> before and you could have permission problems and such. And historically,

2) In cases like this it is IMO mandatory to inform the user. Ideally
   this should be done _before_ performing such a critical action.
   But in any case this _must_ be done after performing such an
   incompatible modification.

   I'm not sure what would be the best way to do this - either check
   for the existence of certain flag files or similar that indiate
   that the user did perform the needed preparations, and/or send
   e-mail to root as part of the installtion pointing out what needs
   to be done, and/or somethign else.


> btw, after all of this, I agree with you that a mention in the release notes
> should have been added prior to release.

3) Yes, things like this should be mentioned (with big red exclamation
   marks) in the release notes, and automatic updates should not be
   attempted if you cannot make sure that the result of the update is
   a running system.

Thanks.
Comment 6 Wolfgang Denk 2012-06-11 05:56:33 EDT
(In reply to comment #4)
> 
> 5.2.8 has come out upstream with many other bugs fixed.

Can you please drop me a note when it hits the testing repo (or is available anywhere else)?

As is, I'm stuck with two dead backup servers :-(

I tried to follow your description, and installed 5.2.7-4 from testing.

Now I see:

# alternatives --config libbaccats.so

There are 3 programs which provide 'libbaccats.so'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib/libbaccats-sqlite3.so
*  2           /usr/lib/libbaccats-postgresql.so
 + 3           /usr/lib/libbaccats-mysql.so

Enter to keep the current selection[+], or type selection number: 

and:

# alternatives --display libbaccats.so
libbaccats.so - status is manual.
 link currently points to /usr/lib/libbaccats-mysql.so
/usr/lib/libbaccats-sqlite3.so - priority 40
 slave libbaccats-5.2.6.so: (null)
/usr/lib/libbaccats-postgresql.so - priority 60
 slave libbaccats-5.2.6.so: (null)
/usr/lib/libbaccats-mysql.so - priority 50
 slave libbaccats-5.2.6.so: (null)
Current `best' version is /usr/lib/libbaccats-postgresql.so.

but any attempt to start the director fails:

bacula-dir[20688]: bacula-dir: dird.c:993 Could not open Catalog "MyCatalog", database "bacula".
bacula-dir[20688]: bacula-dir: dird.c:998 sqlite.c:182 Database /var/bacula/bacula.db does not exist, please create it.
bacula-dir[20688]: 11-Jun 11:46 bacula-dir ERROR TERMINATION

I have no idea why it now tries to access a sqlite DB - the alternates appear to be set for mysql??
Comment 7 Simone Caronni 2012-06-11 06:09:10 EDT
Hello, there are some leftovers from the various packages, to fix your servers please proceed as following.

I made a typo in one of the above comments, but an "alternatives --help" should have pointed out the issues.

These actions are to be performed on all the servers where you have the bacula-libs-sql used (i.e. only directors and storages).

1) Fix alternatives

yum --enablerepo=updates-testing update bacula\*
alternatives --remove libbaccats.so /usr/lib64/libbaccats-mysql-5.2.6.so
alternatives --remove libbaccats.so /usr/lib64/libbaccats-sqlite3-5.2.6.so
alternatives --remove libbaccats.so /usr/lib64/libbaccats-postgresql-5.2.6.so
alternatives --set libbaccats.so /usr/lib64/libbaccats-mysql.so

2) Restore configuration files

If you have not changed anything in your config files, you should have various /etc/bacula/<program>.conf.rpmsave and /etc/bacula/<program>.conf.rpmnew from the various updates.

Please restore the one you were using before the upgrade and restart the service, it should work with no change

-------------------------------------------------------------------------------

I'm building 5.2.8 now (waiting on a BuildRoot override for the docs), you'll see the update in this bug when it is complete and available.

Version 5.2.8-1.fc17 will take care of setting and preserving the user choice from the beginning, so hopefully we will save all of the other users upgrading. Here is an excerpt from the spec file:

# Fix for automatic selection of backends during upgrades
if readlink /etc/alternatives/libbaccats.so | grep --silent mysql || \
   readlink /etc/alternatives/bacula-dir | grep --silent mysql || \
   readlink /etc/alternatives/bacula-sd | grep --silent mysql; then
   /usr/sbin/alternatives --set libbaccats.so %{_libdir}/libbaccats-mysql.so
elif readlink /etc/alternatives/libbaccats.so | grep --silent sqlite || \
   readlink /etc/alternatives/bacula-dir | grep --silent sqlite || \
   readlink /etc/alternatives/bacula-sd | grep --silent sqlite; then
   /usr/sbin/alternatives --set libbaccats.so %{_libdir}/libbaccats-sqlite3.so
else
   /usr/sbin/alternatives --set libbaccats.so %{_libdir}/libbaccats-postgresql.so
fi

Regards,
--Simone
Comment 8 Fedora Update System 2012-06-11 06:17:53 EDT
bacula-docs-5.2.8-1.fc17,bacula-5.2.8-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/bacula-docs-5.2.8-1.fc17,bacula-5.2.8-1.fc17
Comment 9 Wolfgang Denk 2012-06-11 07:49:21 EDT
(In reply to comment #7)
> Hello, there are some leftovers from the various packages, to fix your
> servers please proceed as following.
> 
> I made a typo in one of the above comments, but an "alternatives --help"
> should have pointed out the issues.

Yes, I saw this, but it didn't work.

> These actions are to be performed on all the servers where you have the
> bacula-libs-sql used (i.e. only directors and storages).
> 
> 1) Fix alternatives
> 
> yum --enablerepo=updates-testing update bacula\*

Done:

# rpm -qa | grep bacula
bacula-libs-sql-5.2.7-4.fc17.i686
bacula-director-5.2.7-4.fc17.i686
bacula-docs-5.2.7-1.fc17.noarch
webacula-5.0.3-3.fc17.noarch
bacula-storage-5.2.7-4.fc17.i686
bacula-console-bat-5.2.7-4.fc17.i686
bacula-libs-5.2.7-4.fc17.i686
bacula-client-5.2.7-4.fc17.i686
bacula-console-5.2.7-4.fc17.i686
bacula-common-5.2.7-4.fc17.i686

> alternatives --remove libbaccats.so /usr/lib64/libbaccats-mysql-5.2.6.so
> alternatives --remove libbaccats.so /usr/lib64/libbaccats-sqlite3-5.2.6.so
> alternatives --remove libbaccats.so /usr/lib64/libbaccats-postgresql-5.2.6.so
> alternatives --set libbaccats.so /usr/lib64/libbaccats-mysql.so

Done, I even retry: for verification:

# alternatives --remove libbaccats.so /usr/lib/libbaccats-mysql-5.2.6.so
/usr/lib/libbaccats-mysql-5.2.6.so has not been configured as an alternative for libbaccats.so
# alternatives --remove libbaccats.so /usr/lib/libbaccats-sqlite3-5.2.6.so
/usr/lib/libbaccats-sqlite3-5.2.6.so has not been configured as an alternative for libbaccats.so
# alternatives --remove libbaccats.so /usr/lib/libbaccats-postgresql-5.2.6.so
/usr/lib/libbaccats-postgresql-5.2.6.so has not been configured as an alternative for libbaccats.so
# alternatives --set libbaccats.so /usr/lib/libbaccats-mysql.so

So this looks ok, but nevertheless I see references to "*-5.2.6.so":

# alternatives --display libbaccats.so
libbaccats.so - status is manual.
 link currently points to /usr/lib/libbaccats-mysql.so
/usr/lib/libbaccats-sqlite3.so - priority 40
 slave libbaccats-5.2.6.so: (null)
/usr/lib/libbaccats-postgresql.so - priority 60
 slave libbaccats-5.2.6.so: (null)
/usr/lib/libbaccats-mysql.so - priority 50
 slave libbaccats-5.2.6.so: (null)
Current `best' version is /usr/lib/libbaccats-postgresql.so.

And even though I configure for mysql:

# alternatives --config libbaccats.so

There are 3 programs which provide 'libbaccats.so'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib/libbaccats-sqlite3.so
*  2           /usr/lib/libbaccats-postgresql.so
 + 3           /usr/lib/libbaccats-mysql.so

Enter to keep the current selection[+], or type selection number:
#

I still get this when trying to start bacula-dir:

bacula-dir[21215]: bacula-dir: dird.c:993 Could not open Catalog "MyCatalog", database "bacula".
bacula-dir[21215]: bacula-dir: dird.c:998 sqlite.c:182 Database /var/bacula/bacula.db does not exist, please create it.
bacula-dir[21215]: 11-Jun 12:23 bacula-dir ERROR TERMINATION


I have no idea why this insistes on accessing a sqlite DB ?

> 2) Restore configuration files
> 
> If you have not changed anything in your config files, you should have
> various /etc/bacula/<program>.conf.rpmsave and
> /etc/bacula/<program>.conf.rpmnew from the various updates.

I checked this.  There are .rpmnew files, but no .rpmsave, and my
.conf files are all as they should be (and have been before)

> Please restore the one you were using before the upgrade and restart the
> service, it should work with no change

Unfortunatley, it doesn't.


Looking at the resulting links it seems obvious that this cannot work:

1) System 1, 32 bit config:

# ls -l /usr/lib/libbaccats* /etc/alternatives/libbaccats*
lrwxrwxrwx 1 root root    28 Jun 11 12:21 /etc/alternatives/libbaccats.so -> /usr/lib/libbaccats-mysql.so
lrwxrwxrwx 1 root root    21 Jun 11 11:44 /usr/lib/libbaccats-5.2.7.so -> libbaccats-sqlite3.so
-rwxr-xr-x 1 root root 22032 Jun  8 10:21 /usr/lib/libbaccats-mysql-5.2.7.so
lrwxrwxrwx 1 root root    25 Jun 11 11:44 /usr/lib/libbaccats-mysql.so -> libbaccats-mysql-5.2.7.so
-rwxr-xr-x 1 root root 30276 Jun  8 10:21 /usr/lib/libbaccats-postgresql-5.2.7.so
lrwxrwxrwx 1 root root    30 Jun 11 11:44 /usr/lib/libbaccats-postgresql.so -> libbaccats-postgresql-5.2.7.so
lrwxrwxrwx 1 root root    31 Jun 11 12:21 /usr/lib/libbaccats.so -> /etc/alternatives/libbaccats.so
-rwxr-xr-x 1 root root 22020 Jun  8 10:21 /usr/lib/libbaccats-sqlite3-5.2.7.so
lrwxrwxrwx 1 root root    27 Jun 11 11:44 /usr/lib/libbaccats-sqlite3.so -> libbaccats-sqlite3-5.2.7.so


2) System 2, 64 bit config:

# ls -l /usr/lib64/libbaccats* /etc/alternatives/libbaccats*
lrwxrwxrwx 1 root root    30 Jun 11 12:14 /etc/alternatives/libbaccats.so -> /usr/lib64/libbaccats-mysql.so
lrwxrwxrwx 1 root root    21 Jun 11 11:11 /usr/lib64/libbaccats-5.2.7.so -> libbaccats-sqlite3.so
-rwxr-xr-x 1 root root 26984 Jun  8 10:19 /usr/lib64/libbaccats-mysql-5.2.7.so
lrwxrwxrwx 1 root root    25 Jun 11 11:11 /usr/lib64/libbaccats-mysql.so -> libbaccats-mysql-5.2.7.so
-rwxr-xr-x 1 root root 35272 Jun  8 10:19 /usr/lib64/libbaccats-postgresql-5.2.7.so
lrwxrwxrwx 1 root root    30 Jun 11 11:11 /usr/lib64/libbaccats-postgresql.so -> libbaccats-postgresql-5.2.7.so
lrwxrwxrwx 1 root root    31 Jun 11 12:14 /usr/lib64/libbaccats.so -> /etc/alternatives/libbaccats.so
-rwxr-xr-x 1 root root 26952 Jun  8 10:19 /usr/lib64/libbaccats-sqlite3-5.2.7.so
lrwxrwxrwx 1 root root    27 Jun 11 11:11 /usr/lib64/libbaccats-sqlite3.so -> libbaccats-sqlite3-5.2.7.so


It seems the key problem is that despite any settings with the
"alternatives" tool /usr/lib{64,}/libbaccats-5.2.7.so links to
libbaccats-sqlite3.so - this is obviously wrong here.  The link does
not belong to any RPM, so it's unclear to me who created it, or how
to fixit (except manually).
Comment 10 Simone Caronni 2012-06-11 08:29:36 EDT
Your library has been overwritten by a symlink. To verify packages you can use this command:

rpm -qVV bacula-libs-sql

If you don't have any output that means the files in the package have not been overwritten. If you have some, you can see the "corrupted" files.

Your problem is simply fixed by reinstalling the bacula-libs-sql package; it thrashes away all the alternatives and libraries and reinstall everything:

# rpm -e --nodeps bacula-libs-sql
# yum reinstall bacula-libs-sql
# alternatives --config libbaccats.so

Please let me know.

--Simone
Comment 11 Fedora Update System 2012-06-11 20:36:11 EDT
bacula-docs-5.2.8-1.fc17, bacula-5.2.8-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 12 Wolfgang Denk 2012-06-12 05:08:00 EDT
(In reply to comment #11)
> bacula-docs-5.2.8-1.fc17, bacula-5.2.8-1.fc17 has been pushed to the Fedora
> 17 stable repository.  If problems still persist, please make note of it in
> this bug report.

After manually fixing the issues with the previous 5.2.7-4 based setup (I needed to get the machines running again) the update to 5.2.8-1 worked without problems.

Thanks for all your help.
Comment 13 Simone Caronni 2012-06-12 06:15:36 EDT
Many thanks for the feedback!

Regards,
--Simone