RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2000179 - Amanda index files are not being created correctly for xfsdump backups on RHEL8 clients
Summary: Amanda index files are not being created correctly for xfsdump backups on RHE...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: amanda
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Pavel Cahyna
QA Contact: Evgeny Fedin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-01 14:14 UTC by Dan Astoorian
Modified: 2023-03-01 07:27 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-03-01 07:27:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-95891 0 None None None 2021-09-01 14:18:27 UTC

Description Dan Astoorian 2021-09-01 14:14:57 UTC
Description of problem:
When backing up an XFS filesystem on a Red Hat Enterprise Linux 8 Amanda client using the DUMP method, index files are not being created.  The .gz files under /var/lib/amanda/DailySet1/index/ which should contain the list of files contained in the dump instead contains text of the form "//sbin/xfsrestore: ERROR: unable to create /xfsrestorehousekeepingdir.108902: Permission denied".

As a result, amrecover does not list any files when attempting to restore

Version-Release number of selected component (if applicable):
amanda-3.5.1-13.el8.x86_64
amanda-client-3.5.1-13.el8.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Configure an Amanda installation to back up an XFS filesystem on a RHEL8 client;
2. Perform an amdump.

Actual results:
"amrecover" for the dumped filesystem does not list any files, and the index files under /var/lib/amanda/DailySet1/index/ contain a "Permission denied" error instead of a file list.

Expected results:
"amrecover" should be able to restore files.

Additional info:
Earlier releases of Red Hat Enterprise Linux did not exhibit the problem.  For some reason, the permissions on /var/log/amanda were widened from 02700 in earlier releases to 02770 in RHEL 8.  Because the directory is no longer restricted to be accessible to the amandadump user only, Amanda's safe_cd() function causes amandad to set its working directory to / (which is not writable by amandabackup) instead of /var/log/amanda; I strongly suspect that this is why xfsrestore is unable to run in order to create the index.

Is there a specific reason /var/log/amanda was changed to include g=rwx permissions in RHEL8?  Would anything break if the permissions were restored to drwx--S--- instead of drwxrws---?

If the permissions on /var/log/amanda cannot be tightened up to their previous values for some reason, an alternative fix might be to patch Amanda to provide an "-a" option to xfsrestore to relocate the xfsrestorehousekeepingdir to a directory writable by amandabackup.

Note that the backups themselves are still being performed, and it is still possible to restore files using lower-level utilities (e.g., amfetchdump), but amrecover and any other facilities that depend on the index files will be unable to function.

Comment 1 Pavel Cahyna 2021-09-01 14:33:51 UTC
Hello, the permissions were changed due to bz1572696 (the original permissions caused SElinux denials when starting kamanda). By the way, is there a customer case for the problem?

Comment 2 Dan Astoorian 2021-09-01 14:42:51 UTC
If there's a customer case, I'm not aware of it.  The problem was observed on a derived distribution (AlmaLinux 8), not directly on a Red Hat Enterprise Linux system.  However, note that Red Hat customers are not likely to become aware of the problem until they need to try to recover a file from their Amanda backups and discover that amrecover cannot retrieve them, so it is very much in Red Hat's interest to be proactive about this issue.

Comment 3 Dan Astoorian 2021-09-01 15:50:16 UTC
I don't have access to bz1572696.  By "kamanda", do you just mean Amanda configured to use a daemon running at port 10081 (i.e., with "amandad -auth=krb5)?

Is it possible that the reason widening the permissions on /var/log/amanda fixed the SElinux problems with kamanda was not because group permissions on the directory are actually required, but that, as a side effect, it caused amandad to refuse to use /var/log/amanda as a safe working directory (and thus SElinux denying access to that location was no longer relevant)?  

As far as you are aware, would removing group permissions from /var/log/amanda be a practical workaround if a site is not using Kerberos, and/or if SELinux is disabled?  Or would this result in other known problems?

Comment 4 Pavel Cahyna 2021-09-01 16:50:56 UTC
(In reply to Dan Astoorian from comment #2)
> However, note that Red Hat customers are not
> likely to become aware of the problem until they need to try to recover a
> file from their Amanda backups and discover that amrecover cannot retrieve
> them, so it is very much in Red Hat's interest to be proactive about this
> issue.

I see, thanks for pointing out this and for the report! Yeah, bugs that show up at restore time and not at backup time are nasty. One really needs to test backups before using them. At least I am glad that the data are actually there.

Concerning bz1299761, sorry, it is private, I'll try to summarize what I understood: yes, kamanda is amanda configured to use Kerberos authentication. There is a service file for this usage: kamanda@.service. If you look inside, you will see that it runs amandad as root, not as the usual user (not sure why off the top of my head, presumably to access a keytab?) which is apparently the root of the problem - selinux does not like root accessing this directory using its "appropriate privileges". Yes, I think removing group permissions would be an acceptable workaround if you don't use kamanda nor SELinux in enforcing mode.

Comment 5 Dan Astoorian 2021-09-01 18:02:23 UTC
Thanks for the clarification.

Hopefully the permissions can be restored to what Amanda's safe_cd() is expecting by fixing the SElinux policy to allow amandad to access what it needs when run as root (since changing amandad's working directory from /var/log/amanda to / as a side effect of the permission change may have other unforeseen consequences).

Is it known why this was not an issue on Red Hat Enterprise 7 and earlier?

Comment 6 Pavel Cahyna 2021-09-02 09:31:26 UTC
(In reply to Pavel Cahyna from comment #4)

> it runs amandad as root, not as the usual user (not sure why off the top 
> of my head, presumably to access a keytab?)

indeed, that's the case - see amanda-auth(7): "Note that you're running this as root, rather than as your dump user. AMANDA will set its UID down to the dump user at times it doesn't need to read the keytab file, and give up root permissions entirely before it goes off and runs dump. Alternately you can change your keytab files to be readable by user amanda. You should understand the security implications of this before changing the permissions on the keytab."

Support for kerberized Amanda was added in Fedora in bz1077642 and RHEL then inherited it.

Comment 7 Dan Astoorian 2021-09-02 14:51:02 UTC
It looks like the permissions on /var/log/amanda were changed sometime between amanda-3.5.1-9.fc28.x86_64.rpm and amanda-3.5.1-14.fc29.x86_64.rpm, so Fedora 29 may have had the same deficiency.

Fedora 30 included a patch ("patch-xfsrestore-housekeeping") which avoided the problem by specifying "-a /var/lib/amanda" as parameters to the xfsrestore command, but this fix apparently has not made its way into Red Hat Enterprise Linux 8.  It may not be as clean a fix as making /var/log/amanda "safe" for the purposes of safe_cd(), but it would fix the problem at hand.

Comment 8 Pavel Cahyna 2021-09-02 16:04:19 UTC
Thanks a lot for the analysis and finding the solution! The Fedora bugreport is bz1671117.

Comment 9 Pavel Cahyna 2021-09-21 13:55:41 UTC
I've built packages with the same change as in Fedora, available at https://people.redhat.com/pcahyna/amanda/. Please test.

Comment 13 Dan Astoorian 2021-09-22 15:24:15 UTC
Unfortunately, I don't currently have any systems appropriate for testing; I have only production systems available (running AlmaLinux 8, a derived distribution), which are using the workaround above, i.e., removing group permissions from /var/log/amanda.

Comment 15 RHEL Program Management 2023-03-01 07:27:45 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.


Note You need to log in before you can comment on or make changes to this bug.