Bug 771927

At the moment, there is not even acl/xattrs support in upstream tar ... and we use our patch for it - using posix tar format.

However - as POSIX.1-2001 format has extended header, it is probably doable within it. It would still be good to be compatible with star ( http://cdrecord.berlios.de/private/man/star/star.4.html ) if using this format.

Interestingly, in Fedora 16, strace of "tar -cf foo.tar --xattrs /bin/ping" shows this:

fgetxattr(4, "system.posix_acl_access", 0x7fff81e827f0, 132) = -1 ENODATA (No data available)
fgetxattr(4, "security.selinux", "system_u:object_r:ping_exec_t:s0", 255) = 33
flistxattr(4, 0x1bbd590, 1024)          = 37
fgetxattr(4, "security.capability", "\x01\x00\x00\x02\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00", 1024) = 20
fgetxattr(4, "security.selinux", "system_u:object_r:ping_exec_t:s0", 1024) = 33


But the corresponding extract only shows this:

setxattr("ping", "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x07\x00\xff\xff\xff\xff\x04\x00\x05\x00\xff\xff\xff\xff \x00\x05\x00\xff\xff\xff\xff", 28, 0) = 0
setxattr("ping", "security.selinux", "system_u:object_r:ping_exec_t:s0", 33, 0) = 

This looks odd because acls are being set when none are read and the security.capability xattr is read but not saved.

Perhaps the xattrs patch just has a straightforward bug?  (I haven't checked yet.)

(In reply to comment #2)
> Perhaps the xattrs patch just has a straightforward bug?

Fedora tar does not save/restore all extended attributes.  There is a hard-wired list of namespaces we are interested in.  It may be worth to consider a more generic approach as taken e.g. in Gentoo Linux:

https://bugs.gentoo.org/show_bug.cgi?id=382067#c9

*** Bug 832494 has been marked as a duplicate of this bug. ***

We certainly need the security.capability since we turn this on by default in Fedora 16/17.  	
Ondrej can you at least change the tar patch to support both SELinux and capability security attributes?

As Kamil said, we will probably completely remove the namespace limits, similar to the gentoo approach. This is not the first time someone needs some category to be added.
For F16/F17, just adding security.capability to stored namespaces is probably the right way. I'll make that change now...

tar-1.26-6.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/tar-1.26-6.fc17

tar-1.26-3.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/tar-1.26-3.fc16

Package tar-1.26-3.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing tar-1.26-3.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9561/tar-1.26-3.fc16
then log in and leave karma (feedback).

tar-1.26-6.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

tar-1.26-7.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/tar-1.26-7.fc17

tar-1.26-4.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/tar-1.26-4.fc16

tar-1.26-3.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

tar-1.26-7.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

I just tested tar-1.26-3.fc16.x86_64, and it does not fix the bug.

It seems as if the patch was omitted from the rpm.  If I unpack the
source rpm for tar-1.26-2.fc16 and the new rpm for tar-1.26-3.fc16, here
is the diff:

bash$ diff -qr tar*
Only in tar-1.26-3.fc16.src.rpm: tar-1.26-update-with-change-directory.patch
Files tar-1.26-2.fc16.src.rpm/tar.1 and tar-1.26-3.fc16.src.rpm/tar.1 differ
Files tar-1.26-2.fc16.src.rpm/tar.spec and tar-1.26-3.fc16.src.rpm/tar.spec differ

As you can see, there is no patch related to the file capabilities.
Did somebody forget to put it in the rpm?

Regards,
Andy

Capability support is part of tar-1.24-xattrs.patch distribution patch, so no new patch was required. However, due to order of fchown() and xattrs restoring, capabilities were reset by fchown(), you need to update to https://admin.fedoraproject.org/updates/tar-1.26-4.fc16 - as you can see in comment #12

Ah, damned, in f16 update the relevant part of the xattr patch was not modified. Will make an update soon.

tar-1.26-5.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/tar-1.26-5.fc16

tar-1.26-5.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

It stopped working with tar-1.26-27.fc19.x86_64:

[root@hermes ~]# mkdir /tmp/rhbz771927
[root@hermes ~]# cd /tmp/rhbz771927
[root@hermes rhbz771927]# mkdir src dst
[root@hermes rhbz771927]# touch src/foo
[root@hermes rhbz771927]# setcap cap_net_admin,cap_net_raw+ep src/foo
[root@hermes rhbz771927]# getcap src/foo
src/foo = cap_net_admin,cap_net_raw+ep
[root@hermes rhbz771927]# tar -c --selinux --acls --xattrs -f - -C src foo | \
  tar -x --selinux --acls --xattrs -f - -C dst
[root@hermes rhbz771927]# getcap dst/foo
[root@hermes rhbz771927]#

Hi Cristian, tar does not extract extended attributes other than user.*
by default.  This are usually not really "extended attributes", but other
features which use extended attributes as its storage place.  Usually,
non-root users are unable to set these.  Thus, capabilities are stored under
'security.capabilities', if you are curious - try to run
`getfattr -m.  /usr/bin/ping`.

The correct extraction should be (in f19+) like this:
tar -x --selinux --acls --xattrs-include='*' -f - -C dst

Feel free to reopen if that does not work for you,
Pavel

This behaviour is totally unexpected for the user. The man page does
not provide any help here; it does not include any example either.

OK, the info page says:

	By default, all extended attributes are stored into the
	archive. ... On the other hand, during extracting only
	the 'user.*' domain is extracted by default. 

Such asymmetric behaviour is always a pain in the neck. Why cannot we
have symmetric behaviour, such that when run as root user the tar
restore will just do "the right thing" and restore all settings that
were archived? When running as non-root user, proper "Cannot setcap:
Operation not permitted" (or similar) errors should be emitted.
This way we at least would know that certain parts of the original
file settings were not restored.

It make sense to me to explicitly enable support for acls and xattrs,
but anything beyond this level should be optional and only be needed
if you want to tweak the behaviour.  The default behaviour should not
only be to "make the backup process as complete as possible by
default" (see info page in the '--xattrs' section) - we should also do
this upon restore.  Needing just "--xattrs" for one operation and
another set of options for the reverse operation is really poor
design.



Note that the info page also fails to provide an example how to
correctly restore an archive with all potentially contained
information and settings.

(In reply to Wolfgang Denk from comment #22)
> This behaviour is totally unexpected for the user. The man page does
> not provide any help here; it does not include any example either.

That is truth.  Manual page (which is a little bit enhanced --help output, jfyi)
should shortly say what will be extracted (I'll mention the user.* prefix).

> OK, the info page says:
> 
> 	By default, all extended attributes are stored into the
> 	archive. ... On the other hand, during extracting only
> 	the 'user.*' domain is extracted by default. 
> 
> Such asymmetric behaviour is always a pain in the neck. Why cannot we
> have symmetric behaviour, such that when run as root user the tar
> restore will just do "the right thing" and restore all settings that
> were archived?

I don't see extracting capabilities by default as "doing the right thing".  User
should not expect capabilities to be restored;  the relation to --xattrs is JUST
implementation detail.  Once something like --capabilities option comes into
tar, the relation to --xattrs disappears.  But, here and ....

> When running as non-root user, proper "Cannot setcap:
> Operation not permitted" (or similar) errors should be emitted.
> This way we at least would know that certain parts of the original
> file settings were not restored.

... even here:  It seems to me you are asking for some --extract-all option
(which is asking for separate bugreport, good point btw).

> It make sense to me to explicitly enable support for acls and xattrs,
> but anything beyond this level should be optional and only be needed
> if you want to tweak the behaviour.

And what about capabilities?  How these differ from acls?  We need to
incorporate libcap.so (blind idea) and use --capabilities..

> The default behaviour should not only be to "make the backup process as
> complete as possible by default" (see info page in the '--xattrs' section) -
> we should also do this upon restore.

The reason we store capabilities also in --xattrs input is to not cause
catastrophic scenarios for users.  Just to not loose data.

> Needing just "--xattrs" for one operation and another set of options for the
> reverse operation is really poor design.

Don't expect --xattrs store/restore capabilities and it will be fine.  Consider
storing *all* xattrs (including capabilities) as safety belt (you are still able
to restore everything, right?).

> Note that the info page also fails to provide an example how to correctly
> restore an archive with all potentially contained information and settings.

That is again good point which should result in separate bugzilla request
for something like --extract-all.

Pavel

(In reply to Pavel Raiskup from comment #23)
>
> > Such asymmetric behaviour is always a pain in the neck. Why cannot we
> > have symmetric behaviour, such that when run as root user the tar
> > restore will just do "the right thing" and restore all settings that
> > were archived?
> 
> I don't see extracting capabilities by default as "doing the right thing". 
> User should not expect capabilities to be restored;  the relation to --xattrs is

What makes you think so?  Users expect to have "exact copies"
restored, including all properties the file had when it was backed up.

I think there is a pretty significan majority of user (eventually even an
overwhelming one) who just expect that tar backup and tar restore are
truely inverse operations, and that they can be used to identically
restore a copy of file archived before.

I don't know how many documents exist that claim that something like

	tar -cf - | ( cd somewhere; tar -xpf - )

can be used for reliable copies.  I don;t remember how many sysadmins
I've seen using this method to copy root file systems - and it has
been working perfectly fine for decades.

Now we have ACLs and XATTRs and Capabilities and whatever, and things
break down.  This would still be somewhat acceptable if there were
clear warnings like "this file has additional properties that
are not backed up" and especially error message like "unable to
restore this or that property".  But what we have now is something
that silently works, but the result is not what the user expects.


> > When running as non-root user, proper "Cannot setcap:
> > Operation not permitted" (or similar) errors should be emitted.
> > This way we at least would know that certain parts of the original
> > file settings were not restored.
> 
> ... even here:  It seems to me you are asking for some --extract-all option
> (which is asking for separate bugreport, good point btw).

No.  This is actually NOT what I'm asking for.  I'm asking to make
this behaviour the DEFAULT..

> > It make sense to me to explicitly enable support for acls and xattrs,
> > but anything beyond this level should be optional and only be needed
> > if you want to tweak the behaviour.
> 
> And what about capabilities?  How these differ from acls?  We need to
> incorporate libcap.so (blind idea) and use --capabilities..

In this context I don't care what the name of such properties is. 
I'm playing the role of an end user here, who just wants to archive
and restore a file with all properties it has - the user normally does
not even care if there are ACLs or XATTRs or Capabilities assigned for
the file - he just expects that he can make a backup, nd that
restoring from the backup will give him an axact copy with the very
same behaviour.

It is already a major pain that we explicitly have to enable support
for such features, and it would be indeed nice if "--preserve" would
automatically preserve ALL properties, not only time and owner.  And
restore ALL properties, of course.

> > The default behaviour should not only be to "make the backup process as
> > complete as possible by default" (see info page in the '--xattrs' section) -
> > we should also do this upon restore.
> 
> The reason we store capabilities also in --xattrs input is to not cause
> catastrophic scenarios for users.  Just to not loose data.

But this is the very problem: we do lose date on restore, and
silently, without even a warning.  Restoring a file from an archive
will NOT create an identical copy any more.

> > Needing just "--xattrs" for one operation and another set of options for the
> > reverse operation is really poor design.
> 
> Don't expect --xattrs store/restore capabilities and it will be fine. 
> Consider
> storing *all* xattrs (including capabilities) as safety belt (you are still
> able
> to restore everything, right?).

As mentioned before, I'm not focussed on capabilities here - for the
average user, this is just one of the rarely used, usually not even
noticed properties a file may have.  Archiving and restoring should
always preserve such information to the extend possible, and issue
clear messages when any such information is lost (at archival) or
cannot be restored (at restore).

Just my $ 0.02, of course...

This bug is "RFE: tar should support file capabilities" and our default tar
(GNU tar) supports it.  I see your concerns regarding the default behavior,
Wolfgang.  Sadly, there is IMO not much we can do for your request via this
resolved bugreport.

> tar -cf - | ( cd somewhere; tar -xpf - )

100% expectation that this will work the same way among all systems is bad.
You always must take into account where the 'tar -p' command is run.

The problem is that what must be stored/restored by default is not defined
(even the -p option is not defined), so you can't expect anything.  As tar is
marked LEGACY by POSIX, I would say that there is high chance that this will
never get changed.

My 2 cents, as Fedora & GNU tar user:  I don't want to produce tarballs having
non standard attributes inside BY DEFAULT (it causes unnecessary troubles to
users from other distributions, or non-Linux users - they must deal with
warnings, etc.).  We have a options for that if I know what I am doing.

If you wan't to store/extract literally everything by default, please could
you raise a bug for it?  Whether we should or should not do it is more like
principal question we may not answer in isolation.  I can raise an upstream
question (I mean for all tar implementations like GNU tar, star, libarchive
and others?) though I am not sure I am good person to protect that idea..  So
probably somebody more interested to protect that should raise it..?

Pavel

(In reply to Pavel Raiskup from comment #21)
> Hi Cristian, tar does not extract extended attributes other than user.*
> by default.  This are usually not really "extended attributes", but other
> features which use extended attributes as its storage place.  Usually,
> non-root users are unable to set these.  Thus, capabilities are stored under
> 'security.capabilities', if you are curious - try to run
> `getfattr -m.  /usr/bin/ping`.

I don't see why tar should care how those extended attributes are
interpreted. It should archive & extract all of them, since it's an
archiver.

> The correct extraction should be (in f19+) like this:
> tar -x --selinux --acls --xattrs-include='*' -f - -C dst

For the record, that command works.


(In reply to Wolfgang Denk from comment #24)
> 
> I don't know how many documents exist that claim that something like
> 
> 	tar -cf - | ( cd somewhere; tar -xpf - )
> 
> can be used for reliable copies.  I don;t remember how many sysadmins
> I've seen using this method to copy root file systems - and it has
> been working perfectly fine for decades.

By the way, the -p parameter isn't needed for sysadmins because:

    -p, --preserve-permissions, --same-permissions
                           extract information about file permissions
                           (default for superuser)

Something similar should be done for capabilities. They should be
restored by default at least by the superuser.


(In reply to Pavel Raiskup from comment #25)
> This bug is "RFE: tar should support file capabilities" and our default tar
> (GNU tar) supports it.  I see your concerns regarding the default behavior,
> Wolfgang.  Sadly, there is IMO not much we can do for your request via this
> resolved bugreport.

So another bug report is needed?

> > tar -cf - | ( cd somewhere; tar -xpf - )
> 
> 100% expectation that this will work the same way among all systems is bad.
> You always must take into account where the 'tar -p' command is run.
> 
> The problem is that what must be stored/restored by default is not defined
> (even the -p option is not defined), so you can't expect anything.  As tar is
> marked LEGACY by POSIX, I would say that there is high chance that this will
> never get changed.

By the way, what other archivers that support capabilities (properly)
could be used?

(In reply to Cristian Ciupitu from comment #26)
> (In reply to Pavel Raiskup from comment #21)
> > Hi Cristian, tar does not extract extended attributes other than user.*
> > by default.  This are usually not really "extended attributes", but other
> > features which use extended attributes as its storage place.  Usually,
> > non-root users are unable to set these.  Thus, capabilities are stored under
> > 'security.capabilities', if you are curious - try to run
> > `getfattr -m.  /usr/bin/ping`.
> 
> I don't see why tar should care how those extended attributes are
> interpreted. It should archive & extract all of them, since it's an
> archiver.

Full agreement here.  This is what the average user would expect.

> > can be used for reliable copies.  I don;t remember how many sysadmins
> > I've seen using this method to copy root file systems - and it has
> > been working perfectly fine for decades.
> 
> By the way, the -p parameter isn't needed for sysadmins because:
> 
>     -p, --preserve-permissions, --same-permissions
>                            extract information about file permissions
>                            (default for superuser)
> 
> Something similar should be done for capabilities. They should be
> restored by default at least by the superuser.

Agreed.

> (In reply to Pavel Raiskup from comment #25)
> > This bug is "RFE: tar should support file capabilities" and our default tar
> > (GNU tar) supports it.  I see your concerns regarding the default behavior,
> > Wolfgang.  Sadly, there is IMO not much we can do for your request via this
> > resolved bugreport.
> 
> So another bug report is needed?

This is my understanding, too.

> By the way, what other archivers that support capabilities (properly)
> could be used?

Properly?  None...

Sorry, but simply archiving the binary "security.capability" extended attribute is a very bad idea.

Please take a look in "include/uapi/linux/capability.h" from the linux kernel. The developers write there:

 20 /* User-level do most of the mapping between kernel and user
 21    capabilities based on the version tag given by the kernel. The
 22    kernel might be somewhat backwards compatible, but don't bet on
 23    it. */

and later we find that there already 3 (in words: three) dirrent versions of the binary data for the extended attribute. Soome kernel versions are only supporting some of these versions:

 31 #define _LINUX_CAPABILITY_VERSION_1  0x19980330
 34 #define _LINUX_CAPABILITY_VERSION_2  0x20071026  /* deprecated - use v3 */
 37 #define _LINUX_CAPABILITY_VERSION_3  0x20080522

Version 1 is supported for kernel 2.6.24 and above.
Version 2 is supported for kernel 2.6.25 and above.
Dunno from which kernel version the capability version 3 is supported.

You see, it is quite simple to get into problems if you only have an archive with the binary representation of the capabilities.


Currently I am porting my patches to GNU tar 1.26 to GNU tar 1.27. These patches include a proposal how to deal with file capabilities (AFAIK no currecnt tar implementation can properly process them) - with a backward compatibility layer for archives that have been created with binary file capabilities (letting libcap doing the conversion twice to get a file system capability of the right version).

Created attachment 849830 [details]
Patch to support file capabilities

Note that my above patch needs the patches from Bug #1052876 to be applied before.

What is the status of the above patch? Has this been incorporated into any Fedora distribution?

(In reply to Ani Sinha from comment #31)
> What is the status of the above patch?

Well I am not sure which patch you mean -- if the latest one, I am not sure
whether this was proposed upstream or not, Markus?   If not, Please, try to
propose your work as you are original author -- that will make
100% probability of getting this into (not only) Fedora the fastest way.

> Has this been incorporated into any Fedora distribution?

No, we really need to have this _upstream first_.  Until then, I am not going
to risk we will keep downstream incompatibilities forever.  Historically, that
way it was done for xattrs itself and it was quite a few work to get it
upstream (and it could be seen as quite offensive to GNU tar upstream).
No need to repeat the history - GNU tar folks are pretty communicative and
tar itself had several releases in latest years.  This is the way to go.

Pavel

(In reply to Pavel Raiskup from comment #32)
> (In reply to Ani Sinha from comment #31)
> > What is the status of the above patch?
> 
> Well I am not sure which patch you mean -- 

I am talking about the patch in comment#29 which supports file capabilities in GNU tar. I would really like to see this feature pushed upstream. Markus, please enlighten us.

thanks.

I see. Notice that we have to get the patches mentioned in Comment#30 upstream before.

I sent upstream a remainer, see http://lists.gnu.org/archive/html/bug-tar/2014-07/msg00024.html .

The patch from Comment#29 has been mentioned upstream ( can find http://lists.gnu.org/archive/html/bug-tar/2014-02/msg00019.html quite easy ), but without link - so they are aware there is one.

(In reply to M. Steinborn from comment #34)
> I see. Notice that we have to get the patches mentioned in Comment#30
> upstream before.

Any ETA for this?

Can you please escalate this? I don't see any response from anyone else from your email. We kind of need this ASAP.

Escalate where? This is Fedora, reminder was sent to upstream (as written in https://bugzilla.redhat.com/show_bug.cgi?id=771927#c34 ) . We need to wait for upstream inclusion at least to the git head.

(In reply to Ondrej Vasik from comment #37)
> Escalate where? 

I meant escalate this with the upstream guys.

As written in http://lists.gnu.org/archive/html/bug-tar/2014-01/msg00032.html , Marcus didn't proposed capability patch upstream so far, as it will likely need some adjustments for upstream inclusion. First, there are other patches to be reviewed and included.
We are not going to include this independently on upstream to Fedora, we did so in the case of extended attributes and it was very expensive to maintain (every rebase meant quite a lot of work and it took ~6 years to get xattrs included upstream). It is very likely capabilities will not take so long.
You can try to escalate it yourself on bug-tar , if you believe this will make it happen faster. Marcus sent reminder for the previous patchset ~1 week ago - without response - but as it is PTO time, it might be expectable.

This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Still there, as far as I understand. Could anyone bump the affected version? No such a possibility for me.

(In reply to Andrew Travneff from comment #41)
> Still there, as far as I understand. Could anyone bump the affected version?

To 20 or 21?

I see nothing clearly related in tar-1.27.1-7.fc21 changelog and docs.
So it's 21, I think.

And I think Rawhide might be a good choice. The problem is not solved upstream.

Created attachment 1164880 [details]
Patch rebased and updated to tar 1.29

Hi Marcus, is upstream aware of this patch?

Sorry, comment #39 is still up to date.

But I am quite sure that I mentioned the URL to this bugzilla bug report, so combined with the linked bug-tar mail (again comment #39) upstream should be aware (cf comment #34 here).

Anyway, I am using my patches on top of the redhat patches. For upstream, I need to test them on top of tar 1.29 and/or tar git master first.

Anyway, I'll perpare a new posting to bug-tar this week. Comment #28 here is a good reason why this is wanted.

> Anyway, I am using my patches on top of the redhat patches. For upstream, I
> need to test them on top of tar 1.29 and/or tar git master first.

Thanks for the update.
 
> Anyway, I'll perpare a new posting to bug-tar this week. Comment #28 here is
> a good reason why this is wanted.

Makes sense, I'm aware of those reasons.  Binary form of capabilities might
be subject of change of course.

According to the low priority of this request and as it did not bother any user for years, I am closing this tracker. If you think this issue should be handled and investigated, feel free to reopen it.

Does it mean that tar will not support saving and restoring file capabilities?
Or is it related just with specific implementation (libcap)?

There are another similar tickets, and it bothers users, I think
bug #771926, bug #771134, bug #832494

Thank for your reply, we will try to communicate this with upstream ans see what we can do. Fro tracking reasons leaving this tracker open.

> Does it mean that tar will not support saving and restoring file capabilities?

GNU tar supports storing and extracting capabilities, but it is not binary-safe.
`tar -x -f <file> --xattrs-include='security.*` should do that, you _have to_
check _yourself_ that the extracted binary format is compatible with the system
where you are extracting.