Bug 1356810

Summary: Fliesystem extended attributes in trusted namespace are not persisted after node reboot
Product: Red Hat Enterprise Linux 7 Reporter: Mohamed Ashiq <mliyazud>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED NOTABUG QA Contact: atomic-bugs <atomic-bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.2CC: hchiramm, jhonce, kdudka, lsm5, mliyazud
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-18 14:24:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mohamed Ashiq 2016-07-15 06:03:50 UTC
Description of problem:

We have a container running on RHEL 7.2 latest update where we set extended attribute on a file from the container and after node reboot these extended atributes are not seen on subject file.

Version-Release number of selected component (if applicable):

libattr-2.4.46-12.el7.x86_64
pyxattr-0.5.1-5.el7.x86_64
attr-2.4.46-12.el7.x86_64


[root@dhcp42-69 ~]# uname -a
Linux dhcp42-69.lab.eng.blr.redhat.com 3.10.0-327.18.2.el7.x86_64 #1 SMP Fri Apr 8 05:09:53 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@dhcp42-69 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

How reproducible:

#docker run -v /run -v /etc/glusterfs:/etc/glusterfs:z -v /var/lib/glusterd:/var/lib/glusterd:z -v /var/log/glusterfs:/var/log/glusterfs:z -v /sys/fs/cgroup:/sys/fs/cgroup:ro -d --privileged=true --net=host -v /dev:/dev brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-server-rhel7

# Create a PV?VG/ and LV

# Format it with the XFS FS

# mount LV inside the container

# Make this mount happening in the container whenever container is spawned. We have a custom fstab entry 

# set xattr on trusted namespace. 

# listxattr from the container
 
 -  make sure the xattr can be retrieved.

# Reboot the node

# Spawn the container again

# Check the xattr again
  - At this stage we are not able to see the xattr in subjected file.


Steps to Reproduce:

Same as above

Actual results:

The extended attributes in trusted namespace are vanished after node reboot.
 
Expected results:

The xattr has to be available even after the reboot of the node.

Additional info:

BUgzilla ID # https://bugzilla.redhat.com/show_bug.cgi?id=1355801

Comment 2 Kamil Dudka 2016-07-15 09:15:28 UTC
This is either problem with the file system implementation or the containerized environment.  There is no flag that the libattr users-pace library could use to tell kernel whether the attributes should persist a reboot or not.

Not a bug in the attr package for sure.

Comment 3 Humble Chirammal 2016-07-16 09:35:26 UTC
(In reply to Kamil Dudka from comment #2)
> This is either problem with the file system implementation or the
> containerized environment.  There is no flag that the libattr users-pace
> library could use to tell kernel whether the attributes should persist a
> reboot or not.
> 
> Not a bug in the attr package for sure.

Yes, We do suspect it may be something wrong with the underlying FS. In this context it is XFS. We also did some isolation here and we noticed that the attributes we set from the container was not visible at times when we inspect from the HOST system. Thats one of the reason for opening this bug. But, you were fast to respond before updating this second scenario :). Can you please confirm libattr dont have a role in this scenario as well ? We are doing more isolation wrt container environment and also with underlying FS/XFS. We would like to eliminate the possibilities of libattr in both the scenarios.

Comment 4 Kamil Dudka 2016-07-16 11:05:30 UTC
Sure I meant the attr package with all its subpackages, including libattr.  It just implements library interface on top of the corresponding syscalls.  There is nothing in libattr that could influence whether the extended attributes are propagated to the host file system or not.

Comment 5 Kamil Dudka 2016-07-18 11:24:10 UTC
I am switching the component to docker as it takes care of the bind mounts.

Comment 6 Daniel Walsh 2016-07-19 10:25:00 UTC
The SELinux xattrs are set at container creation time.  It is up to the file system to maintain the Xattrs.

As an experiment outside of docker, could you check the contexts on these directories before reboot?


After creating the container

ls -ldZ /etc/glusterfs /var/lib/glusterd /var/log/glusterfs

After reboot, run the same command.  If these labels are changing, then something in the boot is changing the labels. or the file system is broken.

Nothing in docker should change them after the container creation?

Comment 7 Daniel Walsh 2016-08-19 22:39:46 UTC
Kamil any update on this?

Comment 8 Kamil Dudka 2016-08-20 11:09:46 UTC
(In reply to Daniel Walsh from comment #7)
> Kamil any update on this?

I am just the maintainer of libattr.  This bug was assigned to me by mistake.  Setting needinfo on Mohamed, who reported it originally...

Comment 9 Mohamed Ashiq 2016-08-23 11:45:55 UTC
(In reply to Daniel Walsh from comment #6)
> The SELinux xattrs are set at container creation time.  It is up to the file
> system to maintain the Xattrs.
> 
> As an experiment outside of docker, could you check the contexts on these
> directories before reboot?
> 
> 
> After creating the container
> 
> ls -ldZ /etc/glusterfs /var/lib/glusterd /var/log/glusterfs
> 
> After reboot, run the same command.  If these labels are changing, then
> something in the boot is changing the labels. or the file system is broken.
> 
> Nothing in docker should change them after the container creation?

Dan, sorry for really late reply. I am able to see the labels on the host and container in these(/etc/glusterfs /var/lib/glusterd /var/log/glusterfs). The problem is on the LV's which was created inside the container and mounted in the container using a custom fstab. After reboot, the container is spawned instantly and few mountpoints are sometimes read-only and sometimes not mounted.

Comment 10 Mohamed Ashiq 2016-08-23 11:51:18 UTC
(In reply to Mohamed Ashiq from comment #9)
> (In reply to Daniel Walsh from comment #6)
> > The SELinux xattrs are set at container creation time.  It is up to the file
> > system to maintain the Xattrs.
> > 
> > As an experiment outside of docker, could you check the contexts on these
> > directories before reboot?
> > 
> > 
> > After creating the container
> > 
> > ls -ldZ /etc/glusterfs /var/lib/glusterd /var/log/glusterfs
> > 
> > After reboot, run the same command.  If these labels are changing, then
> > something in the boot is changing the labels. or the file system is broken.
> > 
> > Nothing in docker should change them after the container creation?
> 
> Dan, sorry for really late reply. I am able to see the labels on the host
> and container in these(/etc/glusterfs /var/lib/glusterd /var/log/glusterfs).
> The problem is on the LV's which was created inside the container and
> mounted in the container using a custom fstab. After reboot, the container
> is spawned instantly and few mountpoints are sometimes read-only and
> sometimes not mounted.

So these mountpoints were mounted using custom fstab. We were checking the mount points which was not mounted. As glusterfs was still setting few xattrs for checking(which was something unexpected), We thought the rest of the xattrs are lost.

Comment 11 Daniel Walsh 2016-08-24 12:00:39 UTC
Does this mean there is no bug?  Or at least on bug in docker?