Bug 1308997 - Loosing selinux context on certain file like /etc/machine_id while doing simple operations on images
Loosing selinux context on certain file like /etc/machine_id while doing simp...
Status: CLOSED DUPLICATE of bug 1264253
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs (Show other bugs)
7.2
x86_64 Linux
unspecified Severity medium
: rc
: ---
Assigned To: Richard W.M. Jones
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-16 12:03 EST by Raoul Scarazzini
Modified: 2016-05-25 10:07 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-17 10:17:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Raoul Scarazzini 2016-02-16 12:03:43 EST
Description of problem:

The command virt-sysprep loose the selinux context of certain files while used to change things like the root password.

Version-Release number of selected component (if applicable):

libguestfs-tools-1.28.1-1.55.el7.noarch

How reproducible:

Always.

Steps to Reproduce:
1. Take an image named for example overcloud-full.qcow2
2. Covert it into a raw image to mount it on a rhel7 with qemu-img convert overcloud-full.qcow2 overcloud-full.img
3. Mount it with mount overcloud-full.img /mnt/
4. Check the context of machine_id:
# ls -lZ /mnt/etc/machine-id 
-r--r--r--. root root unconfined_u:object_r:machineid_t:s0 /mnt/etc/machine-id
5. Launch virt-sysprep --root-password password:redhat -a overcloud-full.qcow2
6. Convert this image into a new raw image with qemu-img convert overcloud-full.qcow2 overcloud-full-after-sysprep.img
7. Mount the new image into /mnt
8. Check the new selinux context:
# ls -lZ /mnt/etc/machine-id 
-rw-r--r--. root root system_u:object_r:unlabeled_t:s0 /mnt/etc/machine-id

Actual results:

The command output is this one:

[   0.0] Examining the guest ...
[  57.0] Performing "abrt-data" ...
[  57.0] Performing "bash-history" ...
[  57.0] Performing "blkid-tab" ...
[  57.0] Performing "crash-data" ...
[  57.0] Performing "cron-spool" ...
[  57.0] Performing "dhcp-client-state" ...
[  57.0] Performing "dhcp-server-state" ...
[  57.0] Performing "dovecot-data" ...
[  57.0] Performing "logfiles" ...
[  57.0] Performing "machine-id" ...
[  57.0] Performing "mail-spool" ...
[  57.0] Performing "net-hostname" ...
[  57.0] Performing "net-hwaddr" ...
[  57.0] Performing "pacct-log" ...
[  57.0] Performing "package-manager-cache" ...
[  57.0] Performing "pam-data" ...
[  57.0] Performing "puppet-data-log" ...
[  57.0] Performing "rh-subscription-manager" ...
[  57.0] Performing "rhn-systemid" ...
[  57.0] Performing "rpm-db" ...
[  57.0] Performing "samba-db-log" ...
[  57.0] Performing "script" ...
[  57.0] Performing "smolt-uuid" ...
[  57.0] Performing "ssh-hostkeys" ...
[  57.0] Performing "ssh-userdir" ...
[  57.0] Performing "sssd-db-log" ...
[  57.0] Performing "tmp-files" ...
[  57.0] Performing "udev-persistent-net" ...
[  57.0] Performing "utmp" ...
[  57.0] Performing "yum-uuid" ...
[  57.0] Performing "customize" ...
[  57.0] Setting a random seed
[  57.0] Setting passwords
[  61.0] Performing "lvm-uuids" ...

And after executing it, the /etc/machine_id selinux context is lost.

Expected results:

Context remains unconfined_u:object_r:machineid_t

Additional info:

Situation remains the same, even if you do:

# virt-sysprep --root-password password:redhat --selinux-relabel -a overcloud-full.qcow2

So, is this behavior expected? Why selinux context are messed up if you just want to change the password?
Comment 2 Pino Toscano 2016-02-16 12:24:11 EST
(In reply to Raoul Scarazzini from comment #0)
> Steps to Reproduce:
> 1. Take an image named for example overcloud-full.qcow2
> 2. Covert it into a raw image to mount it on a rhel7 with qemu-img convert
> overcloud-full.qcow2 overcloud-full.img

libguestfs tools can handle qcow2 images just fine.  In general, libguestfs can handle anything qemu can handle.

> 3. Mount it with mount overcloud-full.img /mnt/
> 4. Check the context of machine_id:
> # ls -lZ /mnt/etc/machine-id 
> -r--r--r--. root root unconfined_u:object_r:machineid_t:s0
> /mnt/etc/machine-id

This may be subject to the SELinux policies/contexts on the host system.  Please instead provide the output of the following command to check what are the actual SELinux attributes on the file:

  $ guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id

> 5. Launch virt-sysprep --root-password password:redhat -a
> overcloud-full.qcow2

Since this is a RHEL/Fedora guest, then you most probably need to tell virt-sysprep to take care of the SELinux attributes: pass also --selinux-relabel as argument.

> 6. Convert this image into a new raw image with qemu-img convert
> overcloud-full.qcow2 overcloud-full-after-sysprep.img
> 7. Mount the new image into /mnt
> 8. Check the new selinux context:
> # ls -lZ /mnt/etc/machine-id 
> -rw-r--r--. root root system_u:object_r:unlabeled_t:s0 /mnt/etc/machine-id

Same as before, check the actual value in the guest using the guestfish command above.
Comment 3 Raoul Scarazzini 2016-02-16 12:31:27 EST
(In reply to Pino Toscano from comment #2)
> (In reply to Raoul Scarazzini from comment #0)
> > Steps to Reproduce:
> > 1. Take an image named for example overcloud-full.qcow2
> > 2. Covert it into a raw image to mount it on a rhel7 with qemu-img convert
> > overcloud-full.qcow2 overcloud-full.img
> 
> libguestfs tools can handle qcow2 images just fine.  In general, libguestfs
> can handle anything qemu can handle.

That's fine. I didn't know the command you suggested below, so I've done what I was able to do.

> > 3. Mount it with mount overcloud-full.img /mnt/
> > 4. Check the context of machine_id:
> > # ls -lZ /mnt/etc/machine-id 
> > -r--r--r--. root root unconfined_u:object_r:machineid_t:s0
> > /mnt/etc/machine-id
> 
> This may be subject to the SELinux policies/contexts on the host system. 
> Please instead provide the output of the following command to check what are
> the actual SELinux attributes on the file:
> 
>   $ guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id

Before:

# guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id
[0] = {
  attrname: 
  attrval: 1\x00
}
[1] = {
  attrname: security.selinux
  attrval: unconfined_u:object_r:machineid_t:s0\x00
}

After:

# guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id
[0] = {
  attrname: 
  attrval: 0\x00
}

> > 5. Launch virt-sysprep --root-password password:redhat -a
> > overcloud-full.qcow2
> 
> Since this is a RHEL/Fedora guest, then you most probably need to tell
> virt-sysprep to take care of the SELinux attributes: pass also
> --selinux-relabel as argument.

As I wrote on the additional info, I already tried with --selinux-relabel, without success.

> > 6. Convert this image into a new raw image with qemu-img convert
> > overcloud-full.qcow2 overcloud-full-after-sysprep.img
> > 7. Mount the new image into /mnt
> > 8. Check the new selinux context:
> > # ls -lZ /mnt/etc/machine-id 
> > -rw-r--r--. root root system_u:object_r:unlabeled_t:s0 /mnt/etc/machine-id
> 
> Same as before, check the actual value in the guest using the guestfish
> command above.
Comment 4 Richard W.M. Jones 2016-02-16 12:59:09 EST
When you say "mount" do you mean guestmount?  If so, it's
probably a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=691389
which is a massive pita to fix because it requires the
mount-local API to become multithreaded.
Comment 5 Richard W.M. Jones 2016-02-16 13:01:02 EST
... although as Pino says, even if guestmount supported
SELinux contexts properly, it doesn't mean that things would
work smoothly, because SELinux versions on the host and guest
might different to such an extent that a label required in
the guest would not exist in the host, or the host might not
have SELinux at all.  It's a bit of a minefield, and simply
touching /.autorelabel in the guest is the safest thing,
although note also this systemd bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1049656#c31
Comment 6 Pino Toscano 2016-02-16 13:03:11 EST
(In reply to Raoul Scarazzini from comment #3)
> (In reply to Pino Toscano from comment #2)
> > (In reply to Raoul Scarazzini from comment #0)
> > > 3. Mount it with mount overcloud-full.img /mnt/
> > > 4. Check the context of machine_id:
> > > # ls -lZ /mnt/etc/machine-id 
> > > -r--r--r--. root root unconfined_u:object_r:machineid_t:s0
> > > /mnt/etc/machine-id
> > 
> > This may be subject to the SELinux policies/contexts on the host system. 
> > Please instead provide the output of the following command to check what are
> > the actual SELinux attributes on the file:
> > 
> >   $ guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id
> 
> Before:
> 
> # guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id
> [0] = {
>   attrname: 
>   attrval: 1\x00
> }
> [1] = {
>   attrname: security.selinux
>   attrval: unconfined_u:object_r:machineid_t:s0\x00
> }
> 
> After:
> 
> # guestfish --ro -a overcloud-full.qcow2 -i lxattrlist /etc machine-id
> [0] = {
>   attrname: 
>   attrval: 0\x00
> }

This happens indeed because the "machine-id" operation of virt-sysprep (enabled by default) removes /etc/machine-id, and then touch an empty file.

> > > 5. Launch virt-sysprep --root-password password:redhat -a
> > > overcloud-full.qcow2
> > 
> > Since this is a RHEL/Fedora guest, then you most probably need to tell
> > virt-sysprep to take care of the SELinux attributes: pass also
> > --selinux-relabel as argument.
> 
> As I wrote on the additional info, I already tried with --selinux-relabel,
> without success.

What we do currently for --selinux-relabel is:

      if load_policy && fixfiles restore; then
        rm -f /.autorelabel
      else
        touch /.autorelabel
        echo '%s: SELinux relabelling failed, will relabel at boot instead.'
      fi

(1) fixfiles can fail if the running kernel and the SELinux policies to be applied don't match; this can happen for example when the host and the guest don't match -- e.g. RHEL 7.2 and Fedora 23.

(2) if (1) fails (you should see the message above in the verbose output you get when running virt-sysprep (or any libguestfs tool) with -v), we still tell systemd to do the (costly, but still a way to fix the system) relabel operation at the next boot.  Does the guest have the file after the virt-sysprep operation?  (See the output of `guestfish -a guest.qcow2 -i ll /`.)

Our SELinux handling could be indeed improved (see bug #1089100), although I think we could reset /etc/machine-id (and the other versions of this on the system) just truncating it if present, instead rm+touch.  I'm going to send a patch about this.

Checking more with what I have here: in my RHEL VMs the /etc/machine-id files there have no SELinux attributes at all, yet they work fine.  IIRC systemd bind-mounts at runtime a new version of the file, overriding whatever exists (or not) in the system.
Is the lack of context an actual issue, i.e. something that prevents the boot of the guest?  Or it this more a cosmetic issue?
Comment 7 Pino Toscano 2016-02-16 13:21:29 EST
(In reply to Pino Toscano from comment #6)
> Our SELinux handling could be indeed improved (see bug #1089100), although I
> think we could reset /etc/machine-id (and the other versions of this on the
> system) just truncating it if present, instead rm+touch.  I'm going to send
> a patch about this.

Turns out I was looking at the 1.28 branch, and this is something it was implemented (by me, even) since libguestfs 1.31.1 (1.31.x was the development series for the stable series 1.32.x):
https://github.com/libguestfs/libguestfs/commit/bc598ff52af415d8acc923402bd6dcfd462f2e4d
So it will be automatically part of RHEL 7.3, since we will rebase libguestfs to 1.32.x (bug #1218766).

I'm still interested whether the lack of context is actually problematic.
Comment 8 Raoul Scarazzini 2016-02-17 02:08:53 EST
Adding some considerations to make things more clear:

1) My tests were originally done by using the qcow2 image as a disk for a virtual machine: this means that guestmount didn't have a role inside the analysis, I just use it to report problems in this bug.
I checked the context in the disk image before and after and the problem was there, this is the reason of the bug.

2) The lack of context is very problematic, since it prevents journald from starting after a reboot, and this means that some components (in my case some OpenStack components like httpd that runs horizon) are not starting.

3) I didn't try, but I suppose that using --selinux-relabel will fix things at the first boot, so once known, I will always use it. What I find strange is that what it seems an harmless operation like the one I made can affect some critical services like journald. So I think that this behavior should be reconsidered.
Comment 9 Pino Toscano 2016-02-17 07:07:49 EST
(In reply to Raoul Scarazzini from comment #8)
> 3) I didn't try, but I suppose that using --selinux-relabel will fix things
> at the first boot, so once known, I will always use it.

If you want proper SELinux handling for a guest, then you must use --selinux-relabel indeed, there is no way around that.  As I explained earlier, if the contexts of the files cannot be fixed, then the autorelabel-at-boot method is used (which should work, modulo systemd bugs like bug #1049656).

So yes, please do try booting an image that you customized with virt-sysprep --selinux-relabel: if that does not work, then we will need to investigate that.

To sum up once again: I don't find particularly worrying that the contexts are not set right away, provided the system is told via /.autorelabel to be relabeled at the next boot.
Comment 10 Raoul Scarazzini 2016-02-17 07:33:13 EST
Your explanation makes sense, as I wrote on the first message of this bug I was aware of the --selinux-relabel option, but the fact remains: in this specific case without that option specified in the command you will have problems (big problems).

So I'm trying again to make you reconsider this problem: why since critical files are hit while managing things that on the surface does not seem to involve them, we don't always set the selinux relabel? Or why we're touching that files not changing their context?

I know that this sounds like a philosophical issue, but the problems described are real.
Comment 11 Pino Toscano 2016-02-17 07:49:38 EST
(In reply to Raoul Scarazzini from comment #10)
> Your explanation makes sense, as I wrote on the first message of this bug I
> was aware of the --selinux-relabel option, but the fact remains: in this
> specific case without that option specified in the command you will have
> problems (big problems).

That's why you must, ought, need, whatever, use --selinux-relabel if you know the guest you are customizing with each of virt-builder, virt-customize, or virt-sysprep uses SELinux.

> So I'm trying again to make you reconsider this problem: why since critical
> files are hit while managing things that on the surface does not seem to
> involve them, we don't always set the selinux relabel?

Because it depends on various factors: SELinux installed and configured to be enforcing, user already requested global relabeling by touching /.autorelabel, and I don't know what else.  Also, you might want to relabel a guest only after a number of offline editing with libguestfs tools (or others).  So at the moment the better choice is having the user ask for relabeling for SELinux-enabled guests.

> Or why we're touching that files not changing their context?

In this specific case, the "machine-id" operation does a better job in more recent versions (see comment #7).  We are using also other tools and libraries (like Augeas) which don't handle xattrs properly (or at all), so there is little chance we can rely on them for this.

> I know that this sounds like a philosophical issue, but the problems
> described are real.

Yep, and I'd prefer to be back to reality and have my questions in previous comments to be answered:

| So yes, please do try booting an image that you customized with virt-sysprep
| --selinux-relabel: if that does not work, then we will need to investigate
| that.
Comment 12 Raoul Scarazzini 2016-02-17 09:38:06 EST
(In reply to Pino Toscano from comment #11)
> (In reply to Raoul Scarazzini from comment #10)
> > Your explanation makes sense, as I wrote on the first message of this bug I
> > was aware of the --selinux-relabel option, but the fact remains: in this
> > specific case without that option specified in the command you will have
> > problems (big problems).
> 
> That's why you must, ought, need, whatever, use --selinux-relabel if you
> know the guest you are customizing with each of virt-builder,
> virt-customize, or virt-sysprep uses SELinux.
> 
> > So I'm trying again to make you reconsider this problem: why since critical
> > files are hit while managing things that on the surface does not seem to
> > involve them, we don't always set the selinux relabel?
> 
> Because it depends on various factors: SELinux installed and configured to
> be enforcing, user already requested global relabeling by touching
> /.autorelabel, and I don't know what else.  Also, you might want to relabel
> a guest only after a number of offline editing with libguestfs tools (or
> others).  So at the moment the better choice is having the user ask for
> relabeling for SELinux-enabled guests.

Sorry, but it's still not clear to me. What does it mean to relabel a guest after a number of offline editing? It doesn't matter if for each edit you touch ./autorelabel, the relabeling will be executed just when you'll use that image. Or am I missing something?

> > Or why we're touching that files not changing their context?
> 
> In this specific case, the "machine-id" operation does a better job in more
> recent versions (see comment #7).  We are using also other tools and
> libraries (like Augeas) which don't handle xattrs properly (or at all), so
> there is little chance we can rely on them for this.
> 
> > I know that this sounds like a philosophical issue, but the problems
> > described are real.
> 
> Yep, and I'd prefer to be back to reality and have my questions in previous
> comments to be answered:
> 
> | So yes, please do try booting an image that you customized with
> virt-sysprep
> | --selinux-relabel: if that does not work, then we will need to investigate
> | that.

I verified the fact that .autorelabel is created, so it doesn't matter what happens next. Even if the relabeling is not working at the first boot in any case it's not a guestfs problem.

Again, I got your point, but I must add that those context from my point of view should remain untouched. Of course we can say that if you are using selinux then you must use the --selinux-relabel option, but I think that some other people will find themselves in problems like the one I've described.
Comment 13 Pino Toscano 2016-02-17 09:55:48 EST
(In reply to Raoul Scarazzini from comment #12)
> (In reply to Pino Toscano from comment #11)
> > (In reply to Raoul Scarazzini from comment #10)
> > > Your explanation makes sense, as I wrote on the first message of this bug I
> > > was aware of the --selinux-relabel option, but the fact remains: in this
> > > specific case without that option specified in the command you will have
> > > problems (big problems).
> > 
> > That's why you must, ought, need, whatever, use --selinux-relabel if you
> > know the guest you are customizing with each of virt-builder,
> > virt-customize, or virt-sysprep uses SELinux.
> > 
> > > So I'm trying again to make you reconsider this problem: why since critical
> > > files are hit while managing things that on the surface does not seem to
> > > involve them, we don't always set the selinux relabel?
> > 
> > Because it depends on various factors: SELinux installed and configured to
> > be enforcing, user already requested global relabeling by touching
> > /.autorelabel, and I don't know what else.  Also, you might want to relabel
> > a guest only after a number of offline editing with libguestfs tools (or
> > others).  So at the moment the better choice is having the user ask for
> > relabeling for SELinux-enabled guests.
> 
> Sorry, but it's still not clear to me. What does it mean to relabel a guest
> after a number of offline editing? It doesn't matter if for each edit you
> touch ./autorelabel, the relabeling will be executed just when you'll use
> that image. Or am I missing something?

Touching /.autorelabel as relabeling strategy is done only if the normal context fixing does not work -- again, please see comment #6.

What I mean with "after a number of offline editing with libguestfs tools" is:

  [get an image, say image.qcow2, from somewhere]
  $ virt-customize -a image.qcow2 --update --install some-package etc...
  $ virt-customize -a image.qcow2 --etc
  [yet more libguestfs and/or other tools]
  $ virt-sysprep --selinux-relabel --etc

Fixing SELinux contexts is needed only at the last point before booting a guest, so you don't need to do it at every step.
Comment 14 Raoul Scarazzini 2016-02-17 10:17:16 EST
Closing as not a bug, if you are using Selinux then you must use --selinux-relabel, end of the story.
Comment 15 Pino Toscano 2016-05-25 10:07:12 EDT
This was a bug in virt-sysprep, fixed upstream with:

  commit bc598ff52af415d8acc923402bd6dcfd462f2e4d
  Author: Pino Toscano <ptoscano@redhat.com>
  Date:   Wed Jul 29 13:36:23 2015 +0200

    sysprep: machine_id: simplify implementation
    
    Instead of rm + touch, just truncate the file if existing.
    In case there was no file previously, now nothing is created, as most
    probably the system was fine even without it.
    
    Also, turn the single path into a list, so more machine-id files can be
    handled at once.

This fix will be in RHEL 7.3 as part of the libguestfs rebase (bug #1218766), and it's available also in Z-Stream (libguestfs-1.28.1-1.55.el7_2.4).

*** This bug has been marked as a duplicate of bug 1264253 ***

Note You need to log in before you can comment on or make changes to this bug.