I have what seems to be a fix for this now. It's a bit more complex than I would like (it extends the _check_path() changes we introduced to deal with the first Archive/threadpool race).

The fundamental problem was our reliance on the Python os.makedirs() function to create leading path components. This just creates directories: it does not attempt to accurately reflect an existing namespace structure.

The solution is to walk the paths ourselves, and to inspect the host file system to decide whether to create a directory, or a symbolic link. It's complicated by the fact that we need to handle both paths that exist in the host file system, and those that are specific to the archive (sos_commands etc.), although it also means that we can hide the path setup inside _check_path(), avoiding the need for all add_* methods to call it directly (and so it's always called under the path lock, avoiding races while we are actually setting things up).

https://github.com/sosreport/sos/tree/bmr-fix-archive-leading-paths

I'll give it some more review and testing time with a view to merging it at the beginning of next week.

(In reply to Bryn M. Reeves from comment #1)
> 
> https://github.com/sosreport/sos/tree/bmr-fix-archive-leading-paths
> 
> I'll give it some more review and testing time with a view to merging it at
> the beginning of next week.

I tested the branch via various scenarios of creating (cross-)symlinks to a content and collecting them in various order, and found no issue. Good work!

How to verify:

1) in general: check sosreport does not contain broken symlinks (until they appear on the system as well):

dir=$(sosreport --batch --build | grep "sosreport build tree is located at" | cut -d: -f2)
file $(find -L $dir -lname "*")

Prior the patch, there were broken symlinks like:

/var/tmp/sosreport-gss5-2018-09-13-riwrrdl/lib/systemd/system/dracut-shutdown.service:                            broken symbolic link to `../lib/dracut/modules.d/98systemd/dracut-shutdown.service'

while the real symlink was working:

# file /lib/systemd/system/dracut-shutdown.service
/lib/systemd/system/dracut-shutdown.service: symbolic link to `../../dracut/modules.d/98systemd/dracut-shutdown.service'
#

Now, the "file $(find ..)" command shall not find any broken symlink - or more precisely whatever it detects inside sosreport directory, it must happen on the system directory structure as well.


2) in particular: typical example is systemd plugin collecting /lib/systemd where /lib is symlink to /usr/lib (where further symlinks appear with relative AND also absolute path) - either was possible to be broken but now it should work

3) in particular: s390 has complex /sys/bus/ccwgroup directory structure - verify 1) on s390 machine in particular

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3144