Bug 1805015

This bug has been verified in fuse-overlayfs-0.7.2-3.module+el8.2.0+5808+3aafbc8f.

[root@kvm-08-guest15 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)

[root@kvm-08-guest15 ~]# yum module list | grep container-tools
container-tools      rhel8 [d]   common [d]                               Common tools and dependencies for container runtimes                        
container-tools      1.0         common [d]                               Common tools and dependencies for container runtimes                        
container-tools      2.0 [e]     common [d] [i]                           Common tools and dependencies for container runtimes 

[root@kvm-08-guest15 ~]# rpm -q podman fuse-overlayfs
podman-1.6.4-4.module+el8.2.0+5770+b478fbe7.x86_64
fuse-overlayfs-0.7.2-3.module+el8.2.0+5808+3aafbc8f.x86_64

[ajia@kvm-08-guest15 ~]$ id
uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[ajia@kvm-08-guest15 ~]$ podman unshare cat /proc/self/uid_map 
         0       1001          1
         1     165536      65536

[ajia@kvm-08-guest15 ~]$ podman run ohpc-gnu9-easybuild:1.9.9 'eb --robot lz4-1.9.2-GCCcore-8.3.0.eb'
ERROR: Build of /opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/easybuild/easyconfigs/b/binutils/binutils-2.32.eb failed (err: 'build failed (first 300 chars): cmd " env LIBS=\'-Wl,-rpath=/usr/lib -Wl,-rpath=/usr/lib64\'  make -j 2  CFLAGS="-g -O2 -fPIC" " exited with exit code 2 and output:\nmake[1]: Entering directory \'/home/openhpc/.local/easybuild/build/binutils/2.32/system-system/binutils-2.32\'\nConfiguring in ./intl\nConfiguring in ./libiberty\nconfigure: ')
== temporary log file in case of crash /tmp/eb-0grwbf9f/easybuild-v4s4s9re.log
== resolving dependencies ...
== processing EasyBuild easyconfig /opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/easybuild/easyconfigs/m/M4/M4-1.4.18.eb
== building and installing M4/1.4.18...
== fetching files...
== creating build dir, resetting environment...
== unpacking...

...ignore...

== configuring...
== building...
== FAILED: Installation ended unsuccessfully (build directory: /home/openhpc/.local/easybuild/build/binutils/2.32/system-system): build failed (first 300 chars): cmd " env LIBS='-Wl,-rpath=/usr/lib -Wl,-rpath=/usr/lib64'  make -j 2  CFLAGS="-g -O2 -fPIC" " exited with exit code 2 and output:
make[1]: Entering directory '/home/openhpc/.local/easybuild/build/binutils/2.32/system-system/binutils-2.32'
Configuring in ./intl
Configuring in ./libiberty
configure:  (took 1 min 18 sec)
== Results of the build can be found in the log file(s) /tmp/eb-0grwbf9f/easybuild-binutils-2.32-20200221.100914.YxaTJ.log

Still got exception occurred in fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64
w/ podman-1.6.4-9.module+el8.2.0+5951+eb56bde6.x86_64.


[ajia@kvm-07-guest27 test]$ podman run ohpc-gnu9-easybuild:1.9.9 'eb --robot lz4-1.9.2-GCCcore-8.3.0.eb'
ERROR: Traceback (most recent call last):
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/base/fancylogger.py", line 585, in logToFile
    os.makedirs(directory)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
FileExistsError: [Errno 17] File exists: '/tmp/eb-uhipgt4u'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/main.py", line 113, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/framework/easyblock.py", line 3206, in build_and_install_one
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/framework/easyblock.py", line 296, in close_log
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/base/fancylogger.py", line 588, in logToFile
    raise_with_traceback(exc, "Cannot create logdirectory %s: %s \n detail: %s" % (directory, ex, detail), tb)
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/tools/py2vs3/py3.py", line 75, in raise_with_traceback
  File "/opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/lib/python3.6/site-packages/easybuild/base/fancylogger.py", line 585, in logToFile
    os.makedirs(directory)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
FileExistsError: Cannot create logdirectory /tmp/eb-uhipgt4u: [Errno 17] File exists: '/tmp/eb-uhipgt4u' 
 detail: [Errno 17] File exists: '/tmp/eb-uhipgt4u'

== temporary log file in case of crash /tmp/eb-uhipgt4u/easybuild-nimjp1n9.log
== resolving dependencies ...
== processing EasyBuild easyconfig /opt/ohpc/pub/libs/easybuild/4.1.1/software/EasyBuild/4.1.1/easybuild/easyconfigs/m/M4/M4-1.4.18.eb
== building and installing M4/1.4.18...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...

Hi Alex,

the python backtraces you present doesn't mean that fuse-overlayfs actually segfaults. I have no idea what is Easybuild and how it is relevant to fuse-overlayfs. Could you please test the fuse-overlayfs directly and either set this to VERIFIED when you see no Segmentation fault in the log? Please set this back to ASSIGNED/FailedQA only in case you see an actual segmentation fault from withing fuse-overlayfs providing a valid core/gdb backtrace.

Thanks!
Jindrich

(In reply to Jindrich Novy from comment #8)
> Hi Alex,
> 
> the python backtraces you present doesn't mean that fuse-overlayfs actually
> segfaults. I have no idea what is Easybuild and how it is relevant to
> fuse-overlayfs. Could you please test the fuse-overlayfs directly and either
> set this to VERIFIED when you see no Segmentation fault in the log? Please
> set this back to ASSIGNED/FailedQA only in case you see an actual
> segmentation fault from withing fuse-overlayfs providing a valid core/gdb
> backtrace.
> 

[ajia@hp-dl360g9-03 ~]$ rpm -q fuse-overlayfs buildah podman
fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64
buildah-1.11.6-6.module+el8.2.0+5855+8192c413.x86_64
podman-1.6.4-9.module+el8.2.0+5951+eb56bde6.x86_64

---------------segfault---------------
           PID: 61118 (fuse-overlayfs)
           UID: 1001 (ajia)
           GID: 1001 (ajia)
        Signal: 11 (SEGV)
     Timestamp: Tue 2020-03-17 23:52:44 EDT (8min ago)
  Command Line: /usr/bin/fuse-overlayfs -o lowerdir=/home/ajia/.local/share/containers/storage/overlay/l/DU2LDMRE4BOHIXDKAUP2MVA7VO:/home/ajia/.local/share/containers/storage/overlay/l/O24SMG4LN7Z7YG3DFP4BKBNZNI:/home/ajia/.local/share/c>
    Executable: /usr/bin/fuse-overlayfs
 Control Group: /user.slice/user-1001.slice/user/user.slice/podman-61094.scope
          Unit: user
     User Unit: podman-61094.scope
         Slice: user-1001.slice
     Owner UID: 1001 (ajia)
       Boot ID: 60ea5fff8159487fb47063f02c70d5f2
    Machine ID: b0cf519462cc4cd1987349a0bf368fdc
      Hostname: hp-dl360g9-03.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.fuse-overlayfs.1001.60ea5fff8159487fb47063f02c70d5f2.61118.1584503564000000.lz4
       Message: Process 61118 (fuse-overlayfs) of user 1001 dumped core.
                
                Stack trace of thread 61118:
                #0  0x000000000040d5c0 hash_get_first (fuse-overlayfs)
                #1  0x0000000000405074 count_dir_entries.isra.15 (fuse-overlayfs)
                #2  0x000000000040c1cc do_rm (fuse-overlayfs)
                #3  0x00007f9750e37afa fuse_session_process_buf_int (libfuse3.so.3)
                #4  0x00007f9750e32f5b fuse_session_loop (libfuse3.so.3)
                #5  0x0000000000403835 main (fuse-overlayfs)
                #6  0x00007f975065d6a3 __libc_start_main (libc.so.6)
                #7  0x00000000004038ee _start (fuse-overlayfs)

Verified in fuse-overlayfs-0.7.2-5.module+el8.2.0+6060+9dbc027d.x86_64.
Please also see https://bugzilla.redhat.com/show_bug.cgi?id=1815139

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1675