RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1376453 - bacula-sd reports "Permission denied" on device though running as root
Summary: bacula-sd reports "Permission denied" on device though running as root
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: bacula
Version: 6.8
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Josef Ridky
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-15 13:08 UTC by André Schramm
Modified: 2017-11-03 08:44 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-20 11:45:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description André Schramm 2016-09-15 13:08:34 UTC 
Description of problem:

bacula-sd reports "Permission denied" on the device though it runs as root and all the devices are read-/writable for root. When trying to read the tape barcode labels, 0 slots instead of 40 are detected. But if one simply copies and renames the init script, everything works fine.


Setup:

Server "bacula-dir" (Gentoo): running bacula-dir
Server "bacula-sd" (CentOS 6.8): running bacula-sd, has tape lib "Quantum-Changer" attached

The bacula versions are not identical, but this should be no problem acc. to the bacula compatibility wiki and does not explain the effect.


Version-Release number of selected component (if applicable):

bacula-storage-common-5.0.0-13.el6.x86_64


How reproducible:

Always


Steps to Reproduce:

1. Load a tape

[root@bacula-sd ~]# /usr/libexec/bacula/mtx-changer /dev/sg5 load 1 /dev/nst0 0
Loading media from Storage Element 1 into drive 0...done

2. Run bconsole and check the status

[root@bacula-dir ~]# bconsole
Connecting to Director bacula-dir:9101
1000 OK: bacula-dir Version: 5.2.12 (12 September 2012)
Enter a period to cancel a command.
*status storage=Quantum-Changer
Connecting to Storage daemon Quantum-Changer at 192.168.99.160:9103

bacula-sd Version: 5.0.0 (26 January 2010) x86_64-redhat-linux-gnu redhat
Daemon started 14-Sep-16 16:12, 0 Jobs run since started.
 Heap: heap=135,168 smbytes=17,871 max_bytes=115,156 bufs=70 max_bufs=72
Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name
===================================================================
 15441  Incr          0         0   Cancel   01-Sep-16 16:59 bacula-4
 15451  Full     57,660    27.12 G  Cancel   02-Sep-16 14:24 bacula-sd-2
 15452  Full          0         0   Error    02-Sep-16 14:54 bacula-sd-2
 15453  Full          0         0   Error    02-Sep-16 14:57 bacula-sd-2
 15454  Full          0         0   Error    02-Sep-16 15:34 bacula-sd-2
 15455  Full          0         0   Cancel   02-Sep-16 16:00 bacula-sd-2
====

Device status:
Autochanger "Quantum-Changer" with devices:
   "LTO5" (/dev/nst0)
Device "LTO5" (/dev/nst0) open but no Bacula volume is currently mounted.
    Drive 0 status unknown.
    Total Bytes Read=0 Blocks Read=0 Bytes/block=0
    Positioned at File=0 Block=0
====

Used Volume status:
====

====


3. Try to label or unmount the tape


Actual results:

*label storage=Quantum-Changer pool=LTO5-Pool slots=1-3 barcodes
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
Connecting to Storage daemon Quantum-Changer at 192.168.99.160:9103 ...
3306 Issuing autochanger "slots" command.
Device "LTO5" has 0 slots.
No slots in changer to scan.

*unmount storage=Quantum-Changer
3301 Issuing autochanger "loaded? drive 0" command.
3991 Bad autochanger "loaded? drive 0" command: ERR=Child exited with code 1.
Results=cannot open SCSI device '/dev/sg5' - Permission denied

3002 Device "LTO5" (/dev/nst0) unmounted.


Now doing the following:

[root@bacula-sd ~]# /etc/init.d/bacula-sd stop
Shutting down bacula-sd:                                   [  OK  ]

[root@bacula-sd ~]# cp /etc/init.d/bacula-sd /etc/init.d/bacula-sdd

[root@bacula-sd init.d]# /etc/init.d/bacula-sdd start
Starting bacula-sd:                                        [  OK  ]


This leads to the expected results (detecting 40 slots and no permission errors):

*label storage=Quantum-Changer pool=LTO5-Pool slots=1-3 barcodes
Connecting to Storage daemon Quantum-Changer at 192.168.99.160:9103 ...
3306 Issuing autochanger "slots" command.
Device "LTO5" has 40 slots.
Connecting to Storage daemon Quantum-Changer at 192.168.99.160:9103 ...
3306 Issuing autochanger "list" command.
The following Volumes will be labeled:
Slot  Volume
==============
   1  000001
   2  000002
   3  000003
Do you want to label these Volumes? (yes|no): no

*unmount storage=Quantum-Changer
3307 Issuing autochanger "unload slot 1, drive 0" command.
3002 Device "LTO5" (/dev/nst0) unmounted.


Additional info:

[root@bacula-sd ~]# rpm -qa | egrep '(bacula|mt-st|mtx)' | sort
bacula-common-5.0.0-13.el6.x86_64
bacula-console-5.0.0-13.el6.x86_64
bacula-director-common-5.0.0-13.el6.x86_64
bacula-director-mysql-5.0.0-13.el6.x86_64
bacula-storage-common-5.0.0-13.el6.x86_64
bacula-storage-mysql-5.0.0-13.el6.x86_64
mt-st-1.1-5.el6.x86_64
mtx-1.3.12-5.el6.x86_64


[root@bacula-sd ~]# rpm -vV bacula-storage-common
S.5....T.  c /etc/bacula/bacula-sd.conf
.........    /etc/rc.d/init.d/bacula-sd
.........    /usr/libexec/bacula/disk-changer
.........    /usr/libexec/bacula/dvd-handler
.........    /usr/libexec/bacula/mtx-changer
.........    /usr/sbin/bacula-sd
.........    /usr/sbin/bextract
.........    /usr/sbin/bls
.........    /usr/sbin/btape
.........  d /usr/share/man/man8/bacula-sd.8.gz
.........  d /usr/share/man/man8/bcopy.8.gz
.........  d /usr/share/man/man8/bextract.8.gz
.........  d /usr/share/man/man8/bls.8.gz
.........  d /usr/share/man/man8/bscan.8.gz
.........  d /usr/share/man/man8/btape.8.gz


[root@bacula-sd ~]# ps aux | grep bacula-sd
root     32116  0.0  0.0 246808  1920 ?        Ssl  14:44   0:00 /usr/sbin/bacula-sd -v -d 10 -c /etc/bacula/bacula-sd.conf -u root -g root


[root@bacula-sd init.d]# ll /dev/nst* /dev/sg* /dev/st* /dev/sch*
crw-rw----. 1 root tape   9, 128 Sep  2 16:15 /dev/nst0
crw-rw----. 1 root tape   9, 224 Sep  2 16:15 /dev/nst0a
crw-rw----. 1 root tape   9, 160 Sep  2 16:15 /dev/nst0l
crw-rw----. 1 root tape   9, 192 Sep  2 16:15 /dev/nst0m
crw-rw----. 1 root disk  86,   0 Sep  2 16:15 /dev/sch0
crw-rw----. 1 root disk  21,   0 Jul  8 17:14 /dev/sg0
crw-rw----. 1 root disk  21,   1 Jul  8 17:14 /dev/sg1
crw-rw----. 1 root disk  21,  10 Sep 10 16:15 /dev/sg10
crw-rw----. 1 root disk  21,  11 Sep 10 16:18 /dev/sg11
crw-rw----. 1 root disk  21,  12 Sep 10 16:18 /dev/sg12
crw-rw----. 1 root disk  21,  13 Sep 10 16:18 /dev/sg13
crw-rw----. 1 root disk  21,  14 Sep 10 16:18 /dev/sg14
crw-rw----. 1 root disk  21,  15 Sep 10 16:18 /dev/sg15
crw-rw----. 1 root disk  21,   2 Jul  8 17:14 /dev/sg2
crw-rw----. 1 root disk  21,   3 Jul  8 17:14 /dev/sg3
crw-rw----. 1 root tape  21,   4 Sep  2 16:15 /dev/sg4
crw-rw----. 1 root tape  21,   5 Sep  2 16:15 /dev/sg5
crw-rw----. 1 root cdrom 21,   6 Jul  8 17:14 /dev/sg6
crw-rw----. 1 root disk  21,   7 Sep 10 16:15 /dev/sg7
crw-rw----. 1 root disk  21,   8 Sep 10 16:15 /dev/sg8
crw-rw----. 1 root disk  21,   9 Sep 10 16:15 /dev/sg9
crw-rw----. 1 root tape   9,   0 Sep  2 16:15 /dev/st0
crw-rw----. 1 root tape   9,  96 Sep  2 16:15 /dev/st0a
crw-rw----. 1 root tape   9,  32 Sep  2 16:15 /dev/st0l
crw-rw----. 1 root tape   9,  64 Sep  2 16:15 /dev/st0m
Comment 2 Josef Ridky 2016-09-20 11:45:26 UTC 
Thank you for this report.

This issue has been discussed with upstream authors with following results:

1) The main problem is in different versions of the Director and the SD. Bacula does not support running with different versions of the Director and SD.

Versions supported
   DIR == SD  (only)
   FD <= DIR/FD

So different DIR and SD is not supported and in general will not work. And the FD must be the same version as the DIR/SD or older, but not newer.

2) When is first point resolved and if the permission problem still persist
on the SD which is running with root, it is probably because the SELINUX is enabled and the rules have not been appropriately adapted. The Bacula project does not use or support SELINUX.

While Red Hat welcomes bug reports on Red Hat products here in our public bugzilla database, please keep in mind that bugzilla is not a support tool or means of accessing support.  If you would like technical support please visit our support portal at access.redhat.com or call us for information on subscription offerings to suit your needs.
Comment 3 Josef Ridky 2016-09-27 06:08:58 UTC 
I have got one more highlight to this topic from upstream.

I have put the message from upstream below.

----  UPSTREAM MESSAGE ----

I think that this behavior is caused by wrong Bacula usage by the user.

Please look at following steps that user did (with my comments):

1) [root@bacula-sd ~]# /usr/libexec/bacula/mtx-changer /dev/sg5 load 1
/dev/nst0 0

To load/unload volume in storage managed by Bacula, the user has to
umount the storage first. Otherwise Bacula SD will have wrong tape
drive state remembered and from this reason Bacula can try load volume
already loaded or can try unload volume not loaded.

2) [root@bacula-dir ~]# bconsole
*status storage=Quantum-Changer
...
Device status:
Autochanger "Quantum-Changer" with devices:
   "LTO5" (/dev/nst0)
Device "LTO5" (/dev/nst0) open but no Bacula volume is currently mounted.
    Drive 0 status unknown.
    Total Bytes Read=0 Blocks Read=0 Bytes/block=0
    Positioned at File=0 Block=0
====
...

In this output the "no Bacula volume is currently mounted" state means
that Bacula SD doesn't know about loaded tape from slot 1 to tape
drive /dev/nst0. So, before loading volume by mtx-changer the storage
wasn't unmounted (it was mounted).

3) *label storage=Quantum-Changer pool=LTO5-Pool slots=1-3 barcodes

It can fail because of above problem.

4) *unmount storage=Quantum-Changer

It can fail because of above problem.

5) [root@bacula-sd ~]# /etc/init.d/bacula-sd stop

All Bacula SD devices are freed.

6) [root@bacula-sd ~]# cp /etc/init.d/bacula-sd /etc/init.d/bacula-sdd

I am not sure if it is required.

7) [root@bacula-sd init.d]# /etc/init.d/bacula-sdd start

After starting Bacula SD, all devices are mounted again and tape
drives (with volumes or without) are initialized in Bacula SD.

*label storage=Quantum-Changer pool=LTO5-Pool slots=1-3 barcodes

Devices were mounted and initliazed correctly, so no error.

*unmount storage=Quantum-Changer

Devices were mounted and initliazed correctly, so no error.

---- END OF MESSAGE ----


Please let us know if the message was helpful for you.
Comment 4 André Schramm 2016-09-28 13:37:22 UTC 
Thanks for your reply. To put it short: We don't think this is a bug in Bacula, but rather in the CentOS init script or sth. related. Otherwise we would have directly reported it to the Bacula team. And we did not use the CentOS bug tracker as it would have ended up here anyway.

A short summary would be: the error "Permission denied" goes away if one simply renames /etc/init.d/bacula-sd to /etc/init.d/bacula-sdd and starts the latter. Nothing else is changed in any of the configuration files, only the name of the init script.

Basically, we were trying to set up a new/second tape lib in an existing scenario. The snippets from bconsole do not show everything we did, but should only show the difference between the actual and expected command outputs. Let aside the tape loading behind Bacula's back (which would have confused it) and the unmount on a not mounted tape (which nevertheless flawlessly worked in the second example).
The label scenario was Bacula with the new tape lib and all tapes unlabeled. Running the label command after starting /etc/init.d/bacula-sd gives "Permission denied" and detects no slots in the changer. Simply running /etc/init.d/bacula-sdd works as expected: 40 slots are detected and the tapes are labeled correctly. So it looks like the name of the init script somehow interferes with something else and/or causes "Permission denied".

But what was important was the "DIR == SD" constraint. Though I checked up the versions compatibility list [1] beforehand, I only focused on the FD. So our current setup with different versions of DIR and SD might be working (as it does with /etc/init.d/bacula-sdd), but is not officially supported and we'll have to change it anyway. It thus is unclear whether it is worth debugging this bug within the current setup.


[1] http://wiki.bacula.org/doku.php?id=versions_compatibility
Comment 5 Yves Mulleneers 2017-06-07 14:18:05 UTC 
Bacula Storage Deamon is executed as user bacula.  On my system this user must be added to group tape.  This command solved the 'permission denied' problem.

# usermod -a -G tape bacula

Obviously, tape devices are owned by group tape.
Comment 6 André Schramm 2017-11-03 08:44:50 UTC 
Thanks for your reply. As the topic states, the SD was running as root when the error occurred. I came across "Bug #905530 - bacula-director runs as root not bacula user" and it was the same for the SD at that time. I tried the solution from #905530 to run SD as user bacula, which was added to the tape group before. As this did not work, I changed SD_USER and SD_GROUP back to root.

This was done on a customer system I no longer have access to, so I cannot provide any further information. Thanks for your help and this Bugzilla account is hereby closed.
Note You need to log in before you can comment on or make changes to this bug.