Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1670210

Summary:	supermin fails inside docker: Invalid cross-device link
Product:	Red Hat Enterprise Linux 7	Reporter:	Richard W.M. Jones <rjones>
Component:	supermin	Assignee:	Richard W.M. Jones <rjones>
Status:	CLOSED UPSTREAM	QA Contact:	YongkuiGuo <yoguo>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	7.7	CC:	extras-qa, linl, ptoscano, rjones, shmuel.eiderman, yoguo
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Mac OS
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1670191	Environment:
Last Closed:	2019-06-05 21:55:13 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1670191
Bug Blocks:

Description Richard W.M. Jones 2019-01-28 22:46:36 UTC

+++ This bug was initially created as a clone of Bug #1670191 +++

Description of problem:

Libguestfs "make" fails with when building inside a docker with

"supermin: error: rename: Directory not empty: supermin.d.ibactpan"

Version-Release number of selected component (if applicable):
libguestfs-1.38.6
supermin-5.1.20

How reproducible:

Happens 100% on my MacOS setup
Happens 10% on my Ubuntu setup


Steps to Reproduce:
1. MacOS High Sierra
2. Docker version 18.09.0, build 4d60db4

Actual results:
"supermin: error: rename: Directory not empty: supermin.d.ibactpan"

Expected results:
Build succeeds

Additional info:

After debugging supermin (building it from source and changing some lines) the actual problem is not here:

  if debug >= 1 then
    printf "supermin: renaming %s to %s\n%!" new_outputdir outputdir;
  rename new_outputdir outputdir;

But actually in the previous rename:

  let old_outputdir =
    try
      let old_outputdir = outputdir ^ "." ^ string_random8 () in
      rename outputdir old_outputdir;
      Some old_outputdir
    with
      Unix_error _ -> None in

It fails with Unix_error: supermin: error: rename: Invalid cross-device link: supermin.d

But it is not propagated.

According to docker with OverlayFS: https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Renaming directories: Calling rename(2) for a directory is allowed only when both the source and the destination path are on the top layer. Otherwise, it returns EXDEV error (“cross-device link not permitted”). Your application needs to be designed to handle EXDEV and fall back to a “copy and unlink” strategy.

Should we handle this to support building libguestfs inside a docker?

--- Additional comment from Richard W.M. Jones on 2019-01-28 22:23 UTC ---

Yes we should handle EXDEV as it suggests.  Could you see if
the attached patch fixes it?  I only compile tested it, and
I'm just guessing that 'mv' works on Docker.

--- Additional comment from Sam on 2019-01-28 22:35:38 UTC ---

That was fast.

Yes, it works perfectly!

--- Additional comment from Richard W.M. Jones on 2019-01-28 22:44:13 UTC ---

Thanks for testing, it's upstream in git now and will appear in supermin >= 5.1.21.

Comment 2 Richard W.M. Jones 2019-01-28 22:47:26 UTC

Upstream fix is:
https://github.com/libguestfs/supermin/commit/6579cf5f72d5de345ae1cc97d0344dfa1771460a

Comment 3 YongkuiGuo 2019-02-02 05:56:58 UTC

Hi rjones,
I have no MacOS env. I installed docker（18.09.1）and updated the OverlayFS storage driver on rhel7.6 host. Then I built the supermin from the tarball（supermin-5.1.19.tar.gz）in a container. There was no error during the building. So I cannot reproduce this bug on rhel7 env. Is there any other method to reproduce it?

Comment 4 Sam 2019-02-02 10:55:34 UTC

It only happened to me once on my Ubuntu 14.04 Server - and only when I logged into the docker manually and tried to make libguestfs from sources (docker build script worked fine)

On my MacOS it happens every time.

Extended information

1. MacOS High Sierra 10.13.6 (17G65)
2. Docker version 18.09.0, build 4d60db4
3. Docker image of OL 7.6 (Oracle Linux)

Dockerfile:

FROM oraclelinux:7.6

RUN yum-config-manager --add-repo http://yum.oracle.com/repo/OracleLinux/OL7/kvm/utils/x86_64
RUN yum install -y qemu-kvm
RUN yum install -y qemu-img
RUN ln /usr/bin/qemu-system-x86_64 /usr/libexec/qemu-kvm

# Configuration for libguestfs
RUN yum-config-manager --add-repo http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64
RUN yum-config-manager --add-repo http://yum.oracle.com/repo/OracleLinux/OL7/optional/developer/x86_64
RUN yum-config-manager --add-repo https://yum.oracle.com/repo/OracleLinux/OL7/developer_EPEL/x86_64
RUN yum install -y python36-devel
ADD get-pip.py get-pip.py
RUN python36 get-pip.py
RUN python36 -m pip install --upgrade pip
RUN yum install -y libguestfs-winsupport
RUN yum install -y *ntfs*
# python-libguestfs has errors with python3
# Waiting for fix: https://bugzilla.redhat.com/show_bug.cgi?id=1627964
# In the meanwhile, we have to compile libguestfs and its python3 bindings from source.
ENV LIBGUESTFS_PARENT_DIR=1.38-stable
ENV LIBGUESTFS_DIR=libguestfs-1.38.6
RUN yum install -y wget
RUN yum install -y kernel
RUN yum install -y gcc
RUN yum install -y make
RUN yum install -y gperf
RUN yum install -y genisoimage
RUN yum install -y flex
RUN yum install -y bison
RUN yum install -y ncurses-devel
RUN yum install -y pcre-devel
RUN yum install -y augeas-devel
RUN yum install -y file-devel
RUN yum install -y yajl-devel
RUN yum install -y hivex-devel
RUN yum install -y supermin5
RUN ln /usr/bin/supermin5 /usr/bin/supermin
RUN yum install -y ocaml
RUN yum install -y ocaml-findlib
RUN yum install -y ocaml-hivex-devel
RUN yum install -y fuse-devel
RUN wget http://download.libguestfs.org/$LIBGUESTFS_PARENT_DIR/$LIBGUESTFS_DIR.tar.gz >/dev/null 2>&1
RUN tar xf $LIBGUESTFS_DIR.tar.gz
ENV PYTHON=python36
# Another bug in the python3 bindings in libguestfs.
# https://bugzilla.redhat.com/show_bug.cgi?id=1627964#c18
# Changing to support g.read_file(path) on binary files - hope this doesn't destroy other calls.
RUN sed -i 's/PyUnicode_FromStringAndSize/PyBytes_FromStringAndSize/g' $LIBGUESTFS_DIR/python/handle.c
# --with-distro=REDHAT is very important - do not omit.
RUN cd $LIBGUESTFS_DIR && ./autogen.sh --with-distro=REDHAT
RUN yum install -y automake
RUN yum install -y git
RUN git clone https://github.com/libguestfs/supermin
RUN yum-builddep -y supermin
RUN cd supermin && ./autogen.sh
RUN cd supermin && ./configure --prefix /usr
RUN cd supermin && make
RUN cd supermin && make install
RUN cd $LIBGUESTFS_DIR && make -j8
RUN cd $LIBGUESTFS_DIR && make install REALLY_INSTALL=yes
ENV LIBGUESTFS_BACKEND=direct
ENV LIBGUESTFS_PATH=/usr/local/lib/guestfs
RUN mkdir -p /usr/local/share/oracle/appliance
RUN libguestfs-make-fixed-appliance /usr/local/share/oracle/appliance
ENV LIBGUESTFS_PATH=/usr/local/share/oracle/appliance
RUN python36 -m pip install pexpect

Comment 5 Richard W.M. Jones 2019-02-06 10:54:05 UTC

(In reply to YongkuiGuo from comment #3)
> Hi rjones,
> I have no MacOS env. I installed docker（18.09.1）and updated the OverlayFS
> storage driver on rhel7.6 host. Then I built the supermin from the
> tarball（supermin-5.1.19.tar.gz）in a container. There was no error during the
> building. So I cannot reproduce this bug on rhel7 env. Is there any other
> method to reproduce it?

You would not see an error during building.  However it would fail if you
did 'make check' assuming that the filesystem containing /tmp was using
overlayfs (and isn't a regular filesystem, tmpfs, etc.)  If /tmp is not
overlayfs then try setting TMPDIR to point to an overlayfs filesystem
before running the tests.

Comment 6 YongkuiGuo 2019-02-11 03:02:28 UTC

(In reply to Richard W.M. Jones from comment #5)
> (In reply to YongkuiGuo from comment #3)
> > Hi rjones,
> > I have no MacOS env. I installed docker（18.09.1）and updated the OverlayFS
> > storage driver on rhel7.6 host. Then I built the supermin from the
> > tarball（supermin-5.1.19.tar.gz）in a container. There was no error during the
> > building. So I cannot reproduce this bug on rhel7 env. Is there any other
> > method to reproduce it?
> 
> You would not see an error during building.  However it would fail if you
> did 'make check' assuming that the filesystem containing /tmp was using
> overlayfs (and isn't a regular filesystem, tmpfs, etc.)  If /tmp is not
> overlayfs then try setting TMPDIR to point to an overlayfs filesystem
> before running the tests.

The 'make check' command run successfully in the container(docker image of rhel7.6). The filesystem of /tmp is overlay.

# make check
...
PASS: test-basic.sh
PASS: test-execstack.sh
PASS: test-build-bash.sh
PASS: test-binaries-exist.sh
PASS: test-harder.sh
PASS: test-build-bash-network.sh
PASS: test-binaries-exist-network.sh
PASS: test-harder-network.sh
============================================================================
Testsuite summary for supermin 5.1.19
============================================================================
# TOTAL: 8
# PASS:  8
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

# df -T -h
Filesystem                            Type     Size  Used Avail Use% Mounted on
overlay                               overlay   50G   45G  5.0G  90% /
tmpfs                                 tmpfs     64M     0   64M   0% /dev
tmpfs                                 tmpfs    3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/mapper/rhel_rhel7libguestfs-root xfs       50G   45G  5.0G  90% /etc/hosts
shm                                   tmpfs     64M     0   64M   0% /dev/shm
tmpfs                                 tmpfs    3.8G     0  3.8G   0% /proc/asound
tmpfs                                 tmpfs    3.8G     0  3.8G   0% /proc/acpi
tmpfs                                 tmpfs    3.8G     0  3.8G   0% /proc/scsi
tmpfs                                 tmpfs    3.8G     0  3.8G   0% /sys/firmware

Comment 7 YongkuiGuo 2019-02-18 10:51:40 UTC

Thanks Sam for providing the Dockerfile. I tried but I still could not reproduce this bug without the MacOS env.

Comment 8 Sam 2019-02-18 10:57:14 UTC

No problem,

As I mentioned, I could not reproduce it on my Ubuntu environment either.
Only happens on my MacOS.

Do you need any more information on my MacOS environment? I think I can still reproduce this (before the fix) if needed.

Sam

Comment 9 YongkuiGuo 2019-02-19 10:04:33 UTC

(In reply to Sam from comment #8)
> No problem,
> 
> As I mentioned, I could not reproduce it on my Ubuntu environment either.
> Only happens on my MacOS.
> 
> Do you need any more information on my MacOS environment? I think I can
> still reproduce this (before the fix) if needed.
Thanks a lot. In fact, I don't know the difference between MacOS and RHEL on the docker side.

Comment 10 YongkuiGuo 2019-02-21 10:13:35 UTC

Hi Sam,
Could you help verify this bug when the fixed supermin version is built by dev?

Comment 11 Sam 2019-02-26 12:40:23 UTC

(In reply to YongkuiGuo from comment #10)
> Hi Sam,
> Could you help verify this bug when the fixed supermin version is built by
> dev?

What do you mean? I already verified this when building supermin from sources.
After the fix by RwJ in commit https://github.com/libguestfs/supermin/commit/6579cf5f72d5de345ae1cc97d0344dfa1771460a it works great.

Sam

Comment 12 YongkuiGuo 2019-02-28 07:29:53 UTC

(In reply to Sam from comment #11)
> (In reply to YongkuiGuo from comment #10)
> > Hi Sam,
> > Could you help verify this bug when the fixed supermin version is built by
> > dev?
> 
> What do you mean? I already verified this when building supermin from
> sources.
> After the fix by RwJ in commit
> https://github.com/libguestfs/supermin/commit/
> 6579cf5f72d5de345ae1cc97d0344dfa1771460a it works great.
> 
Sorry, you are right. Thanks.

Comment 14 Richard W.M. Jones 2019-06-05 21:55:13 UTC

I'm closing this as the bug does not affect any Red Hat supported products and
RHEL 7 is going into phase 3.  It is fixed upstream already.