Bug 996483 - Restoring F19 tar archive using Live or Rescue Media removes all extended file attributes & corrupts user and group names
Restoring F19 tar archive using Live or Rescue Media removes all extended fil...
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: tar (Show other bugs)
19
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Pavel Raiskup
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-13 05:05 EDT by Craig A Martin
Modified: 2013-08-14 13:09 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-14 02:37:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Craig A Martin 2013-08-13 05:05:19 EDT
Description of problem:
Restoring a F19 tar archive using either F19 Live Media or Rescue Media removes all extended file attributes and corrupts many file user and group names.

Version-Release number of selected component (if applicable):
/usr/bin/tar-2:1.26-24.fc19.x86_64
/usr/lib64/libacl.so.1.1.0.fc19.x86_64
/usr/lib64/libc-2.17.so.fc19.x86_64 (glibc-2.17.11.fc19.x86_64)

How reproducible:
Always happens in the same way.

Steps to Reproduce:
1.Create a tar archive of F19 (tar -cv -f $TAPE -X $EXCLUDE --one-file-system --acls --xattrs / /boot /home /var/lib/pgsql/data /tmp 2> /tmp/tarerr.out)
2.Boot Live or Rescue Media
3.tar xvf /dev/st0 to restore F19

Actual results:
F19 boots but functions poorly due to file attribute corruption and disfunction.
Absolutely all extended file attributes (acls and xattrs) are lost.  For example:

# rpm -Va | grep "\.""\."P (caPability change)
........P    /usr/libexec/pt_chown (glibc-common)
........P    /usr/bin/gnome-keyring-daemon (gnome-keyring)
........P    /usr/bin/rcp (rsh)
........P    /usr/bin/rlogin (rsh)
........P    /usr/bin/rsh (rsh)
........P    /usr/bin/systemd-detect-virt (systemd)
........P    /usr/bin/ping (iputils)
........P    /usr/bin/ping6 (iputils)
........P    /usr/sbin/arping (iputils)
........P    /usr/sbin/clockdiff (iputils)
........P    /usr/sbin/suexec (httpd)
........P    /usr/sbin/mtr (mtr)

In addition, many file/directory user names and groups are corrupted:

Eg:

# lt -lt -d /var/lib/chrony
drwxr-xr-x.  2 977       977        4096 Aug 13 16:07 chrony

instead of:
drwxr-xr-x.  2 chrony    chrony     4096 Aug 13 16:07 chrony

Where 977 is a non-existant user and group.  This results in messages in /var/log/messages that chrony does have have permission to access /var/lib/977/driftfile.  Other similar exmples are to numerous to mention.
Expected results:


Additional info:

The only "work around" is to scan all file attributes and repair those effected (user names, group names, suids, guids, acls and xattrs).
Comment 1 Craig A Martin 2013-08-13 05:11:22 EDT
The files tar, libacl.so.1.1.0 and libc-2.17.so are all smaller in the F19 Live and Rescue Media than in the full F19 OS.  The libacl file is actually 7% smaller.  I used the F19-x86_64-DESK-20130806.iso respin from http://alt.fedoraproject.org/pub/alt/live-respins/. The 3 files are identical in size in the Live Media when compared to those in the Rescue Media
Comment 2 Pavel Raiskup 2013-08-13 06:31:49 EDT
Craig, thanks for the report.  Try to restore with --xattrs --acls please.
Extended attributes are not restored by default in F19+ (as the
acls/selinux/xattr patch became upstream).  If you don't like this behavior,
you still may use the TAR_OPTIONS='--xattrs --selinux --acls' environment
variable.

According to group/user name corruptions.  Be careful that the same
users/groups exist on the running system and that they have the _same_
UIDs/GIDs.

--

I doubt the problem is in libacl.so* size difference (it is probably different
stripping level).
Comment 3 Pavel Raiskup 2013-08-13 07:15:59 EDT
> According to group/user name corruptions.  Be careful that the same
> users/groups exist on the running system and that they have the _same_
> UIDs/GIDs.

That was not 100% correct statement, sorry.  The username and groupname should
have precedence before user ID and group ID during archive extraction.

Thus the UID/GID change among system should not be big a problem .. the
problem with 'chrony' user is probably that its account is not present on the
live media.  If this is not the case, we should really observe why the
user/group information are not restored properly.

Pavel
Comment 4 Craig A Martin 2013-08-13 16:11:36 EDT
Hi Pavel
I tried (and I think I tried this once before):

tar --acls --xattrs xvf /dev/st0

but with the same result, namely:

[root@server lib]# rpm -Va | grep "\.""\."P
........P    /usr/bin/rcp
........P    /usr/bin/rlogin
........P    /usr/bin/rsh
........P    /usr/sbin/suexec
........P    /usr/bin/systemd-detect-virt
........P    /usr/bin/ping
........P    /usr/bin/ping6
........P    /usr/sbin/arping
........P    /usr/sbin/clockdiff
........P    /usr/libexec/pt_chown
........P    /usr/sbin/mtr
........P    /usr/bin/gnome-keyring-daemon

However, with the user/group names I found that for user/groupnames that
existed on both the live media and the full OS:
where the userid or groupid on the live media were different to those in
the the full OS, the user/groupnames on the files restored to the full
OS were replaced with the user/groupids from the live media. So, before:

[root@server ~]# ls -lt /var/lib  (full OS)
total 260
drwxr-xr-x+  2 root      root       4096 Aug 14 02:18 alsa
drwxr-x---+  2 root      slocate    4096 Aug 14 02:04 mlocate
drwxr-xr-x+  2 root      root       4096 Aug 14 01:49 PackageKit
drwxr-xr-x+  2 root      root       4096 Aug 14 01:49 xkb
drwxr-xr-x+  2 root      root       4096 Aug 14 01:49 NetworkManager
drwxrwx--T+ 11 gdm       gdm        4096 Aug 14 01:49 gdm
drwxr-xr-x+  6 root      root       4096 Aug 14 01:49 nfs
drwx------+  2 rpc       rpc        4096 Aug 14 01:49 rpcbind
drwxr-xr-x+  2 root      root       4096 Aug 14 01:49 rpm
-rw-------.  1 root      root       4096 Aug 14 01:49 random-seed
drwxr-xr-x+  2 chrony    chrony     4096 Aug 14 01:48 chrony
drwxr-xr-x+  6 root      root       4096 Aug 14 01:47 yum
-rw-r--r--   1 root      root        210 Aug 14 01:29 error
-rw-r--r--.  1 root      root       1355 Aug 13 11:32 logrotate.status
drwx------+  2 postfix   root       4096 Aug 10 17:41 postfix
drwxr-xr-x+  2 root      root       4096 Aug 10 17:41 misc
drwxr-xr-x+  2 root      root       4096 Aug 10 17:41 alternatives
drwx------+  2 root      root       4096 Aug 10 11:02 udisks
drwxr-xr-x+  3 root      root       4096 Aug  8 09:58 var
drwxr-xr-x+  3 root      root       4096 Aug  8 09:55 usr
drwxr-xr-x+  2 unbound   unbound    4096 Aug  1 03:10 unbound
drwx------+  2 apache    apache     4096 Jul 31 14:50 dav
drwxr-xr-x+  2 root      root       4096 Jul 26 01:13 initramfs
drwxr-xr-x+  5 root      root       4096 Jul 25 21:01 pear
drwxr-xr-x+  2 root      root       4096 Jul 23 23:36 bluetooth
drwxr-xr-x+  2 root      root       4096 Jul 23 19:12 hp
drwxr-xr-x+  3 root      root       4096 Jul 22 15:04 php
drwxr-xr-x+  3 colord    colord     4096 Jul 19 06:32 colord
drwxr-xr-x+  6 root      root       4096 Jul 19 00:06 sss
drwxr-xr-x+  4 root      root       4096 Jul 15 09:54 texmf
drwxr-xr-x+  2 root      root       4096 Jul 15 09:34 prelink
drwxr-xr-x+  2 root      root       4096 Jul 14 14:47 freeipmi
drwxr-xr-x+  4 root      root       4096 Jul 14 14:41 hsqldb
drwxr-xr-x+  2 root      root       4096 Jul 14 14:39 sepolgen
drwx------+  4 postgres  postgres   4096 Jul 14 14:38 pgsql
drwxr-x---+  3 root      polkitd    4096 Jul 14 14:36 polkit-1
drwxr-xr-x+  3 root      root       4096 Jul  8 16:56 games
drwxr-xr-x+  3 root      root       4096 Jul  8 16:56 rpm-state
drwxr-xr-x+  2 root      root       4096 Jul  3 01:02 os-prober
drwxr-xr-x+  2 root      root       4096 Jul  2 20:14 dhclient
drwxr-xr-x+  5 root      root       4096 Jun 24 23:31 systemd
drwxr-xr-x+  2 root      root       4096 Jun 22 09:15 selinux
drwxr-xr-x+  2 root      root       4096 Jun 18 06:23 dbus
drwx------+  3 root      root       4096 Jun 12 23:17 authconfig
drwxr-xr-x+  4 root      root       4096 Jun 12 01:16 AccountsService
drwxr-xr-x+  2 root      root       4096 Jun 11 21:27 dnsmasq
drwx------+  2 tss       tss        4096 Jun  2 21:42 tpm
drwxr-xr-x+  4 root      root       4096 May 31 16:07 stateless
drwx------+  2 pulse     pulse      4096 May 31 12:36 pulse
drwxr-xr-x+  2 root      root       4096 May 21 22:32 setroubleshoot
drwxr-xr-x+  4 root      root       4096 May  6 16:09 net-snmp
drwxr-xr-x+  3 root      root       4096 Apr 30 03:27 dirmngr
drwxr-xr-x+  3 root      root       4096 Apr 17 19:48 samba
drwx------+  2 root      root       4096 Apr  3 23:47 rsyslog
drwxr-x---+  6 backuppc  root       4096 Apr  1 03:50 BackupPC
drwx------+  2 root      root       4096 Mar 28 21:11 udisks2
drwxr-xr-x+  2 root      root       4096 Mar 27 00:14 plymouth
drwxr-xr-x+  2 root      root       4096 Mar 20 22:10 upower
drwxr-xr-x+  3 root      root       4096 Mar  9 02:19 color
drwxr-xr-x+  2 webalizer root       4096 Feb 19 09:38 webalizer
drwxr-xr-x+  2 root      root       4096 Feb 14 13:53 cs
drwxrwx---+  2 root      root       4096 Sep  9  2010 DeviceKit-disks
drwxr-xr-x+  2 root      root       4096 Sep  9  2010 DeviceKit-power
drwxr-xr-x+  2 root      root       4096 Mar 25  2010 readahead
drwxrwx---+  2 root      polkituser 4096 Aug 18  2009 PolicyKit


But now after running tar from the live media:

[root@server lib]# ls  -lt  (full OS)
total 256
drwxr-xr-x+  2 root      root       4096 Aug 14 00:34 alsa
drwxr-xr-x+  2 root      root       4096 Aug 14 00:33 rpm
drwxr-xr-x+  2 root      root       4096 Aug 14 00:29 PackageKit
drwxr-xr-x+  2 root      root       4096 Aug 14 00:28 xkb
drwxrwx--T+ 11 gdm       gdm        4096 Aug 14 00:28 gdm
drwxr-xr-x+  2 root      root       4096 Aug 14 00:28 NetworkManager
drwxr-xr-x+  6 root      root       4096 Aug 14 00:28 nfs
-rw-------.  1 root      root       4096 Aug 14 00:28 random-seed
drwxr-xr-x+  2       997        996 4096 Aug 13 19:34 chrony
drwx------+  2 rpc       rpc        4096 Aug 13 13:49 rpcbind
drwxr-x---+  2 root      slocate    4096 Aug 13 11:32 mlocate
-rw-r--r--.  1 root      root       1355 Aug 13 11:32 logrotate.status
drwxr-xr-x+  6 root      root       4096 Aug 13 07:52 yum
drwx------+  2 postfix   root       4096 Aug 10 17:41 postfix
drwxr-xr-x+  2 root      root       4096 Aug 10 17:41 misc
drwxr-xr-x+  2 root      root       4096 Aug 10 17:41 alternatives
drwx------+  2 root      root       4096 Aug 10 11:02 udisks
drwxr-xr-x+  3 root      root       4096 Aug  8 09:58 var
drwxr-xr-x+  3 root      root       4096 Aug  8 09:55 usr
drwxr-xr-x+  2       996        995 4096 Aug  1 03:10 unbound
drwx------+  2 apache    apache     4096 Jul 31 14:50 dav
drwxr-xr-x+  2 root      root       4096 Jul 26 01:13 initramfs
drwxr-xr-x+  5 root      root       4096 Jul 25 21:01 pear
drwxr-xr-x+  2 root      root       4096 Jul 23 23:36 bluetooth
drwxr-xr-x+  2 root      root       4096 Jul 23 19:12 hp
drwxr-xr-x+  3 root      root       4096 Jul 22 15:04 php
drwxr-xr-x+  3       998        998 4096 Jul 19 06:32 colord
drwxr-xr-x+  6 root      root       4096 Jul 19 00:06 sss
drwxr-xr-x+  4 root      root       4096 Jul 15 09:54 texmf
drwxr-xr-x+  2 root      root       4096 Jul 15 09:34 prelink
drwxr-xr-x+  2 root      root       4096 Jul 14 14:47 freeipmi
drwxr-xr-x+  4 root      root       4096 Jul 14 14:41 hsqldb
drwxr-xr-x+  2 root      root       4096 Jul 14 14:39 sepolgen
drwx------+  4 postgres  postgres   4096 Jul 14 14:38 pgsql
drwxr-x---+  3 root             999 4096 Jul 14 14:36 polkit-1
drwxr-xr-x+  3 root      root       4096 Jul  8 16:56 games
drwxr-xr-x+  3 root      root       4096 Jul  8 16:56 rpm-state
drwxr-xr-x+  2 root      root       4096 Jul  3 01:02 os-prober
drwxr-xr-x+  2 root      root       4096 Jul  2 20:14 dhclient
drwxr-xr-x+  5 root      root       4096 Jun 24 23:31 systemd
drwxr-xr-x+  2 root      root       4096 Jun 22 09:15 selinux
drwxr-xr-x+  2 root      root       4096 Jun 18 06:23 dbus
drwx------+  3 root      root       4096 Jun 12 23:17 authconfig
drwxr-xr-x+  4 root      root       4096 Jun 12 01:16 AccountsService
drwxr-xr-x+  2 root      root       4096 Jun 11 21:27 dnsmasq
drwx------+  2 tss       tss        4096 Jun  2 21:42 tpm
drwxr-xr-x+  4 root      root       4096 May 31 16:07 stateless
drwx------+  2       992        992 4096 May 31 12:36 pulse
drwxr-xr-x+  2 root      root       4096 May 21 22:32 setroubleshoot
drwxr-xr-x+  4 root      root       4096 May  6 16:09 net-snmp
drwxr-xr-x+  3 root      root       4096 Apr 30 03:27 dirmngr
drwxr-xr-x+  3 root      root       4096 Apr 17 19:48 samba
drwx------+  2 root      root       4096 Apr  3 23:47 rsyslog
drwxr-x---+  6 backuppc  root       4096 Apr  1 03:50 BackupPC
drwx------+  2 root      root       4096 Mar 28 21:11 udisks2
drwxr-xr-x+  2 root      root       4096 Mar 27 00:14 plymouth
drwxr-xr-x+  2 root      root       4096 Mar 20 22:10 upower
drwxr-xr-x+  3 root      root       4096 Mar  9 02:19 color
drwxr-xr-x+  2 webalizer root       4096 Feb 19 09:38 webalizer
drwxr-xr-x+  2 root      root       4096 Feb 14 13:53 cs
drwxrwx---+  2 root      root       4096 Sep  9  2010 DeviceKit-disks
drwxr-xr-x+  2 root      root       4096 Sep  9  2010 DeviceKit-power
drwxr-xr-x+  2 root      root       4096 Mar 25  2010 readahead
drwxrwx---+  2 root      polkituser 4096 Aug 18  2009 PolicyKit

and you can see the user/groupids in the 900s corresponding to those
belonging to the use/groupnames on the live media rather than those on
the full OS.  This seems to be the behavior that you were alluding to in
your first message.  I did not expect it because I thought all the file
meta data would come from the tar archive.

Do you know of any way of getting the file user/groupnames off the tar
archive instead of picking up the user/groupids from the live media in
these cases?  Or is there no way of preventing this from happening.

Just to be clear, all of the files (including chrony) in /var/lib that
picked up the 'strange' user/groupids do exist in the live media.  In my
original bug report I said chrony had a user and group of 977 (from
memory), that should have been 997 996 (getting senile!)

The /var/lib files from the live media are listed below:
ls -lt 
total 196 
drwxr-xr-x.  2 root    root    4096 Aug 13 10:01 NetworkManager 
drwxr-xr-x.  2 root    root    4096 Aug 13 10:01 xkb 
drwx------.  2 rpc     rpc     4096 Aug 13 10:01 rpcbind 
-rw-------.  1 root    root     512 Aug 13 10:01 random-seed 
drwxr-xr-x.  2 root    root    4096 Aug 13 10:01 dhclient 
drwxr-xr-x.  2 root    root    4096 Aug  9 10:41 alsa 
drwxr-xr-x.  2 chrony  chrony  4096 Aug  9 10:41 chrony 
drwxr-xr-x.  3 colord  colord  4096 Aug  9 10:36 colord 
drwxr-xr-x.  2 root    root    4096 Aug  6 08:26 rpm 
drwx------.  3 root    root    4096 Aug  6 08:26 authconfig 
drwxr-xr-x.  6 root    root    4096 Aug  6 08:26 yum 
drwxr-xr-x.  2 root    root    4096 Aug  6 08:17 alternatives 
drwxrwx--T.  3 gdm     gdm     4096 Aug  6 08:17 gdm 
drwxr-xr-x.  2 root    root    4096 Aug  6 08:16 prelink 
drwxr-xr-x. 12 root    root    4096 Aug  6 08:14 libvirt 
drwxr-xr-x.  5 root    root    4096 Aug  6 08:14 nfs 
drwxr-xr-x.  3 root    root    4096 Aug  6 08:13 samba 
drwxr-xr-x.  2 root    root    4096 Aug  6 08:13 plymouth 
drwxr-xr-x.  4 root    root    4096 Aug  6 08:13 stateless 
drwxr-xr-x.  6 root    root    4096 Aug  6 08:13 sss 
drwxr-xr-x.  2 unbound unbound 4096 Aug  6 08:13 unbound 
drwxr-xr-x.  8 root    root    4096 Aug  6 08:12 iscsi 
drwxr-xr-x.  4 root    root    4096 Aug  6 08:12 AccountsService 
drwxr-xr-x.  3 root    root    4096 Aug  6 08:12 rpm-state 
drwxr-x---.  3 root    polkitd 4096 Aug  6 08:12 polkit-1 
drwxr-xr-x.  4 root    root    4096 Aug  6 08:12 systemd 
drwxr-xr-x.  4 root    root    4096 Aug  6 08:12 net-snmp 
drwxr-xr-x.  3 root    root    4096 Aug  6 08:12 color 
drwxr-xr-x.  2 root    root    4096 Jul 25 13:13 initramfs 
drwxr-xr-x.  2 root    root    4096 Jul 24 23:44 glusterd 
drwxr-xr-x.  2 root    root    4096 Jul 23 11:36 bluetooth 
drwxr-xr-x.  2 root    root    4096 Jul 10 07:34 corosync 
-rw-r--r--.  1 root    root       0 Jul 10 03:39 logrotate.status 
drwxr-xr-x.  2 root    root    4096 Jul  8 04:56 games 
drwxr-xr-x.  2 root    root    4096 Jul  8 04:56 misc 
drwxr-xr-x.  2 root    root    4096 Jul  2 13:02 os-prober 
drwxr-xr-x.  2 root    root    4096 Jul  2 09:40 lldpad 
drwxr-xr-x.  2 root    root    4096 Jun 22 09:33 PackageKit 
drwxr-xr-x.  2 root    root    4096 Jun 21 21:15 selinux 
drwxr-xr-x.  2 root    root    4096 Jun 17 18:23 dbus 
drwxr-xr-x.  2 root    root    4096 Jun 11 09:27 dnsmasq 
drwx------.  2 tss     tss     4096 Jun  2 09:42 tpm 
drwx------.  2 pulse   pulse   4096 May 31 00:36 pulse 
drwxr-xr-x.  2 root    root    4096 May 21 10:32 setroubleshoot 
drwxr-x---.  2 root    slocate 4096 Apr 10 12:57 mlocate 
drwx------.  2 root    root    4096 Apr  3 11:47 rsyslog 
drwx------.  2 root    root    4096 Mar 28 09:11 udisks2 
drwxr-xr-x.  2 root    root    4096 Mar 20 10:10 upower 
drwxr-xr-x.  2 root    root    4096 Mar  5 10:21 fprint 
drwxr-xr-x.  2 root    root    4096 Feb 18 03:19 sheepdog 


Thanks so much for your help  -  Craig
Comment 5 Pavel Raiskup 2013-08-13 17:43:01 EDT
> I tried (and I think I tried this once before):
>
> tar --acls --xattrs xvf /dev/st0

Well, I see — for restoring of capabilities, you need to specify also
--xattrs-include='security.capability'.

The problem I re-observed here is that the --xattrs & related options are
still not documented in Fedora (as I backported original upstream changes not
containing original documentation).  Thanks for this - I filled a bug #996753.

> [...]
>
> Do you know of any way of getting the file user/groupnames off the tar
> archive instead of picking up the user/groupids from the live media in
> these cases?  Or is there no way of preventing this from happening.

Unpack only files you need again on target system?  Fix the UID for 'chrony'
by usermod/groupmod (if there is no clash)?

Overall, there seems to be no easy solution for this.  [OT] Most "efficient"
advice from me is either (a) don't reinstall, just upgrade Fedora or (b)
reinstall & restore only what you really need to (data & configuration, not
system). [/OT]

Pavel
Comment 6 Craig A Martin 2013-08-13 23:23:14 EDT
> Well, I see — for restoring of capabilities, you need to specify also
--xattrs-include='security.capability'.

Yes, that worked for me.  Thanks - secret information.

>  Most "efficient" advice from me is either (a) don't reinstall, just upgrade Fedora or (b) reinstall & restore only what you really need to (data & configuration, not system).

The problem here is that I am reformatting and partitioning the HDD, and restoring the full F19 OS from a tar tape for disaster recovery eg HDD failure.  I would really rather have the file user and group straight off the tape than have tar making use of the /etc/group and similar files on the active system.  Was this not the former behavior tar?

Craig
Comment 7 Pavel Raiskup 2013-08-14 02:37:43 EDT
> The problem here is that I am reformatting and partitioning the HDD, and
> restoring the full F19 OS from a tar tape for disaster recovery eg HDD
> failure.  I would really rather have the file user and group straight off
> the tape than have tar making use of the /etc/group and similar files on the
> active system.

Then the --numeric-owner is your friend.  But note that if there is a UID/GID
clash between source/target systems, this will make even more mess.

> Was this not the former behavior tar?

Can't tell you.  At least latest upstream git repository shows that the
numeric-owner option was there in that time.


I am closing this bugreport as it is not a bug but feel free to continue
discussing or reopen if you feel this is really bug.
Comment 8 Pavel Raiskup 2013-08-14 02:58:04 EDT
> At least latest upstream git repository shows

I meant the oldest commit :) in upstream repo.
Comment 9 Craig A Martin 2013-08-14 13:09:16 EDT
OK, thanks Pavel.

With your guidance, I used: tar --numeric-owner --acls --xattrs --xattrs-include='security.capability' -xvf /dev/st0, to perform a full system restore that preserved all of the original file attributes.  I realize that I will have take care that the u/gids correspond to the right u/g names.

Hope it was not wasted time for you.  Cheers  -  Craig

Note You need to log in before you can comment on or make changes to this bug.