Bug 1460077

Summary: Executing db_hotbackup as user root instead of the user owning the db-files causes issues
Product: Red Hat Enterprise Linux 7 Reporter: Christian Horn <chorn>
Component: libdbAssignee: Matej Mužila <mmuzila>
Status: CLOSED ERRATA QA Contact: Vaclav Danek <vdanek>
Severity: medium Docs Contact: Lenka Špačková <lkuprova>
Priority: unspecified    
Version: 7.4CC: chorn, hhorak, mmuzila, mskalick, ovasik, pkubat, vdanek
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
"db_hotbackup -c" should be used with caution The "db_hotbackup" command with the "-c" option must be run by the user that owns the database. If the user is different and the log file reaches its maximal size, a new log file is created with an ownership of the user that ran the command, which consequently makes the database unusable for its owner. This note has been added to the `db_hotbackup(1)` manual page.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 17:44:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1420851, 1465891, 1466363    

Description Christian Horn 2017-06-09 02:40:18 UTC
Description of problem:
openldap uses libdb as database backend.  When writes are done on the openldap (resulting in writes on the underlying libdb files), and db_hotbackup is executed as root instead of the user owning the db-files, this can result in db-files beeing owned by the root-user, and as a result the ldap-service going down.
This is not intuitive for users, they run db_hotbackup as root and then face corruptions.

Version-Release number of selected component (if applicable):
current

How reproducible:
always, after some minutes

Steps to Reproduce:
1. create a simple openldap setup
2. perform write operations, for example in a loop create/remove an ldap object
3. at the same time run db_hotbackup in a loop, as root user

Actual results:
After some time
db_hotbackup: BDB0060 PANIC: fatal region error detected; run recovery
db_hotbackup: DB_ENV->log_archive: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
db_hotbackup: BDB1581 File handles still open at environment close
db_hotbackup: BDB1582 Open file handle: /var/lib/ldap/__db.001
db_hotbackup: BDB1582 Open file handle: /var/lib/ldap/__db.002
db_hotbackup: BDB1582 Open file handle: /var/lib/ldap/__db.003
db_hotbackup: BDB1582 Open file handle: /var/lib/ldap/./log.0000000130
db_hotbackup: BDB0060 PANIC: fatal region error detected; run recovery
db_hotbackup: dbenv->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
db_hotbackup: BDB5043 HOT BACKUP FAILED!

Expected results:
I suggest that db_hotbackup verifies if it gets started as the user who owns the db-files.  

Additional info:

Comment 5 Christian Horn 2017-08-16 00:32:21 UTC
Hi,

(In reply to Matej Mužila from comment #4)
> Workaround to this problem is to run db_hotbackup not by user root but by
> user ldap.
Thank you, we in support are aware, and this is also documented in the linked kbase article.  
This is however not preventing the customer though to run the backup with a different user.  This bz is to get more bulletproof tools and lower chances that people have to experience the issue first while running as incorrect user, risking loss of data and waste of time, before they understand that they use the incorrect user.  Checking this in the tool should be simple, and the gains are huge.

Comment 8 Christian Horn 2017-09-05 02:03:32 UTC
(In reply to Matej Mužila from comment #7)
> I'm afraid that there is no bulletproof way to get the owner of database.
> For example when the database is not initialized, there are no files that
> could be used to determine owner of the database.
Yes, we are not worried about that case here as the files do not exist.  Whoever executes the backup then and has write permissions can write the first backup (excluding unlikely things like disk full etc.).
 
> Even if there are database files present, determining owner of the database
> from them is a bit tricky. 
Do you mean the database files?  If the filesystem is aware of ownerships, we should be able to see who owns the file?

> There may exist a situation, where database files
> are not owned by user that runs the database.
To my understanding, the requirement is that the db_hotbackup is executed as the same user who owns the files, correct?
If that is the case, shouldn't we just need 
a) the user-id under which db_hotbackup is started
b) the owner of the existing files
and then compare these?

> Determining the right user from a previous log file is insufficient, because
> previous log file may be nonexistent.
Yes, but I think it's ok to not consider that situation.

Comment 11 Christian Horn 2017-09-22 00:43:43 UTC
(In reply to Matej Mužila from comment #10)
> (In reply to Christian Horn from comment #8)
> > > Even if there are database files present, determining owner of
> > > the database
> > > from them is a bit tricky. 
> > Do you mean the database files?  If the filesystem is aware of ownerships,
> > we should be able to see who owns the file?
> 
> Yes, we are able to find out who owns those files, but imagine a situation
> when files are not owned by the user that runs database. They may be owned
> by different user and have set read/write permissions to a group or anybody.
> This is obviously not the case of ldap, however ldap is not the only thing
> that uses libdb.

I think you refer to the situation which we want to make more bulletproof with this bz.  Saying it in pseudo-code:

if (no databasefiles do yet exist) {
  just write the files with the permissions of the user
  executing the db_hotbackup script
}
else {
  if ( (user who started db_hotbackup) == (user who owns existing files) ) {
    just do the backup, permissions seem to be in order.
  }
  else {
    do some appropriate(tm) action.
    - For example, if started as root, write files as user who owns
      existing files.
    - or atleast output a big fat warning message if this situation
      is detected
    optional: blindly try to write the backup files anyway,
    this is the action we currently undertake.
  }
}


> > To my understanding, the requirement is that the db_hotbackup is executed as
> > the same user who owns the files, correct?
> > If that is the case, shouldn't we just need 
> > a) the user-id under which db_hotbackup is started
> > b) the owner of the existing files
> > and then compare these?
> 
> db_hotbackup should be ran by the same user, that runs database (in this
> case user ldap).

Agreed.  The request in this bz is to better deal with the situation
when that is not the case.  Even outputting "this looks fishy, the 
user who started the script does not own existing backup files" would
help.


> > > Determining the right user from a previous log file is
> > > insufficient, because
> > > previous log file may be nonexistent.
> > Yes, but I think it's ok to not consider that situation.
> 
> I don't think so. This situation is real and may occur. I think that making
> db_hotbackup to "guess" the right user is much worse than not doing so, if
> it is impossible to do so in all cases. It would make usage of db_hotbackup
> even less predictable and tricky than it is now.

I have not understood the situation here referred to well enough to comment, I can not imagine a situation where we have to guess.  Getting less predictable is not good of course.  How about just trying to detect the situation then and additionally to what we currently do (simply trying to write the files) additionally outputting a warning?

Comment 13 Christian Horn 2017-09-28 00:15:10 UTC
(In reply to Matej Mužila from comment #12)
> We may print warning when we detect potentially dangerous situation (user
> different to user owning files). Most of dangerous situations would be
> detected (and announced to admin) and behavior of db_hotbackup would not
> change.
> 
> What do you think about it?

Great, lets do that.

Comment 17 Christian Horn 2017-11-16 01:40:38 UTC
(In reply to Matej Mužila from comment #16)
> We think, it would be better to mention in man page, that if db_hotbackup
> is run with -c parameter, it is necessary to run it by the user that owns
> databse files.
> 
> The other thing is that the problem occurs quite rarely (1 of 40 000 cases)
> and it is not caused by the program itsefl, but by running the program
> incorrectly.
> 
> Do you think it is OK?

Customer will continue to run into this, and just then when having run into the issue investigate and hit the error description - so warning would be beneficial over just documenting, IMHO.
We have to take only documentation if we are not getting the warning, though.  We have this in kbase for customers, having it in the manpage will document it for all users.

Comment 30 errata-xmlrpc 2018-04-10 17:44:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0952