We can extend add_copy_path by not following symlinks, but I am not sure if it will help you / what in particular you want to achieve.

> Actual results:
>
> We grab the scheduler multiple times.

Each scheduler is grabbed just once, though it can be optionally accessible via multiple symlinks in the sosreport tree, like:

# ls -lah /sys/block/
total 0
drwxr-xr-x.  2 root root 0 Jan 14 07:57 .
dr-xr-xr-x. 13 root root 0 Dec 30 21:49 ..
lrwxrwxrwx.  1 root root 0 Dec 30 21:49 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx.  1 root root 0 Dec 30 21:49 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx.  1 root root 0 Dec 30 21:49 sda -> ../devices/pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda
lrwxrwxrwx.  1 root root 0 Dec 30 21:49 sr0 -> ../devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:0/1:0:0:0/block/sr0
# cd /var/tmp/sosreport-pmoravec-caps64-rhev-2019-01-14-bdmaenp
# cat sys/block/sda/queue/scheduler
noop [deadline] cfq 
#  cat sys/devices/pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda/queue/scheduler
noop [deadline] cfq 


But each the file is collected just once:

# find . -name scheduler
./sys/devices/pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda/queue/scheduler
./sys/devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:0/1:0:0:0/block/sr0/queue/scheduler
./sys/devices/virtual/block/dm-1/queue/scheduler
./sys/devices/virtual/block/dm-0/queue/scheduler
#


B'cos:

# file sys/block/sda
sys/block/sda: symbolic link to `../devices/pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda'
#


We can stop following symlinks when _collecting_ files (i.e. if scheduler itself would be a file or directory, we wont follow symlinks from it) - example patch will follow. BUT this won't help here since here we _resolve_ symlink /sys/block/sda . If we would not do so, we won't collect /sys/devices/pci* and we even dont know where /sys/block/sda/queue/scheduler targets to (since we prohibited resolving the symlinks on its path).

Is that intended?

Or could you show me desired sosreport "tree" (unpacked directory content of "sosreport -o block --batch --build"), and what in particular you dont want to there?

Example patch not following symlinks when _collecting_ files/dirs (might not work 100%, on RHEL7 but portable to RHEL8 directly):


--- /usr/lib/python2.7/site-packages/sos/plugins/__init__.py.orig	2019-01-14 08:03:18.041092265 +0100
+++ /usr/lib/python2.7/site-packages/sos/plugins/__init__.py	2019-01-14 08:34:05.249811741 +0100
@@ -327,7 +327,7 @@ class Plugin(object):
     def do_regex_find_all(self, regex, fname):
         return regex_findall(regex, fname)
 
-    def _copy_symlink(self, srcpath):
+    def _copy_symlink(self, srcpath, follow_symlinks=True):
         # the target stored in the original symlink
         linkdest = os.readlink(srcpath)
         dest = os.path.join(os.path.dirname(srcpath), linkdest)
@@ -359,6 +359,10 @@ class Plugin(object):
             self._log_debug("link '%s' is a directory, skipping..." % linkdest)
             return
 
+        if not follow_symlinks:
+            self._log_debug("link '%s' set not to follow, skipping..." % linkdest)
+            return
+
         self.copied_files.append({'srcpath': srcpath,
                                   'dstpath': dstpath,
                                   'symlink': "yes",
@@ -381,17 +385,17 @@ class Plugin(object):
 
         # skip recursive copying of symlink pointing to itself.
         if (absdest != srcpath):
-            self._do_copy_path(self.strip_sysroot(absdest))
+            self._do_copy_path(self.strip_sysroot(absdest), follow_symlinks=follow_symlinks)
         else:
             self._log_debug("link '%s' points to itself, skipping target..."
                             % linkdest)
 
-    def _copy_dir(self, srcpath):
+    def _copy_dir(self, srcpath, follow_symlinks=True):
         try:
             for afile in os.listdir(srcpath):
                 self._log_debug("recursively adding '%s' from '%s'"
                                 % (afile, srcpath))
-                self._do_copy_path(os.path.join(srcpath, afile), dest=None)
+                self._do_copy_path(os.path.join(srcpath, afile), dest=None, follow_symlinks=follow_symlinks)
         except OSError as e:
             if e.errno == errno.ELOOP:
                 msg = "Too many levels of symbolic links copying"
@@ -419,7 +423,7 @@ class Plugin(object):
         self.archive.add_node(path, mode, os.makedev(dev_maj, dev_min))
 
     # Methods for copying files and shelling out
-    def _do_copy_path(self, srcpath, dest=None):
+    def _do_copy_path(self, srcpath, dest=None, follow_symlinks=True):
         '''Copy file or directory to the destination tree. If a directory, then
         everything below it is recursively copied. A list of copied files are
         saved for use later in preparing a report.
@@ -441,11 +445,11 @@ class Plugin(object):
             return
 
         if stat.S_ISLNK(st.st_mode):
-            self._copy_symlink(srcpath)
+            self._copy_symlink(srcpath, follow_symlinks)
             return
         else:
             if stat.S_ISDIR(st.st_mode) and os.access(srcpath, os.R_OK):
-                self._copy_dir(srcpath)
+                self._copy_dir(srcpath, follow_symlinks)
                 return
 
         # handle special nodes (block, char, fifo, socket)
@@ -547,7 +551,7 @@ class Plugin(object):
     def _add_copy_paths(self, copy_paths):
         self.copy_paths.update(copy_paths)
 
-    def add_copy_spec(self, copyspecs, sizelimit=None, tailit=True):
+    def add_copy_spec(self, copyspecs, sizelimit=None, tailit=True, follow_symlinks=True):
         """Add a file or glob but limit it to sizelimit megabytes. If fname is
         a single file the file will be tailed to meet sizelimit. If the first
         file in a glob is too large it will be tailed to meet the sizelimit.
@@ -599,7 +603,7 @@ class Plugin(object):
                 if sizelimit and current_size > sizelimit:
                     limit_reached = True
                     break
-                self._add_copy_paths([_file])
+                self._add_copy_paths([(_file, follow_symlinks)])
 
             if limit_reached and tailit and not _file_is_compressed(_file):
                 file_name = _file
@@ -913,9 +917,9 @@ class Plugin(object):
         return glob.glob(copyspec)
 
     def _collect_copy_specs(self):
-        for path in self.copy_paths:
+        for path, follow_symlinks in self.copy_paths:
             self._log_info("collecting path '%s'" % path)
-            self._do_copy_path(path)
+            self._do_copy_path(path, follow_symlinks=follow_symlinks)
 
     def _collect_cmd_output(self):
         for progs in zip(self.collect_cmds):

This kind of internal API change is best discussed upstream before filing distro bugs - it would affect all users of the project, and so we don't want to split the discussion between there and Red Hat's bugzilla.

That said, I don't really see the purpose of this RFE - what you are seeing is simply the structure of sysfs. We capture that so that it faithfully reflects the structure of the file system on the collection host (modulo any absolute symlinks that must be rewritten, but those do not exist in sysfs).

Although it is possible to find this file by traversing multiple paths through the sysfs symlink forrest, you should find that they all have the same inode number - i.e. they are the same file:

# find sys/ -name scheduler
sys/devices/virtual/block/dm-0/queue/scheduler
sys/devices/virtual/block/dm-4/queue/scheduler
sys/devices/virtual/block/dm-3/queue/scheduler
sys/devices/virtual/block/dm-1/queue/scheduler
sys/devices/virtual/block/dm-2/queue/scheduler
sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/scheduler
sys/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.2/host4/target4:0:0/4:0:0:0/block/sdb/queue/scheduler

Here I have two physical devices and five virtual device-mapper devices. The find command reports one 'scheduler' file per device. This is the canonical path location (without traversing any symlinks).

Taking 'sda' as an example, we can reach that via a number of paths:

- First fia the canonical path, inode# 1608260

  # ls -li sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/scheduler
  1608260 -rw-r--r--. 1 root root 21 Jan 14 11:33 sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/scheduler

- Now via the /sys/block/ symbolic link:

  # ls -l sys/block/sda
  lrwxrwxrwx. 1 root root 75 Jan 14 11:33 sys/block/sda -> ../devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda
  # ls -li sys/block/sda/queue/scheduler
  1608260 -rw-r--r--. 1 root root 21 Jan 14 11:33 sys/block/sda/queue/scheduler

  Note the matching inode number.

I cannot find the third case that you mention (did not see an example for that).

I realise that this structure seems complex, but it is the complexity of the underlying file system - sos juts reflects that in the data that it captures.

The data from the devices/ subtree _is_ required - since it is the actual, canonical source of the information for the device.

Similarly, we must capture the block/ hierarchy since the symlinks found there are the standard way to discover the set of devices (and their real location) on a system.

> I _think_ this 'do not follow' symlinks idea is the solution, but I'm not sure.  We're planning on grabbing more data from these directories with https://github.com/sosreport/sos/pull/1490 so as we grab more files there, it will grab 3 times that amount if following the symlink.

It really doesn't. In the past sos used to butcher the sysfs data in various ways, e.g. flattening out symlinks when we are just interested in some deeper-nested file path. It's _that_ behaviour that actually leads to errors when adding more paths in this location - if the first flattens the symlink then that file type will conflict with the structure of the later path collection, raising an exception in the archive and leading to a loss of data.

We spent months getting this behaviour correct last year and have seen no cases since then of incorrect structure or failure to capture files from sysfs.

If you see any problems then let us know, but for now I think we can close this NOTABUG.