RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1940037 - toolbox does not provide /:/host mount required for sosreport
Summary: toolbox does not provide /:/host mount required for sosreport
Keywords:
Status: CLOSED ERRATA
Alias: None
Deadline: 2021-07-12
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: toolbox
Version: 8.4
Hardware: All
OS: Linux
unspecified
high
Target Milestone: rc
: 8.4
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-17 13:48 UTC by Derrick Ornelas
Modified: 2021-12-21 02:52 UTC (History)
13 users (show)

Fixed In Version: toolbox-0.0.99.3-0.2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-09 17:37:14 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers toolbox issues 689 0 None open RFE: Migration path for users of coreos/toolbox 2021-03-18 08:21:46 UTC
Red Hat Product Errata RHSA-2021:4154 0 None None None 2021-11-09 17:37:33 UTC

Description Derrick Ornelas 2021-03-17 13:48:02 UTC 
Description of problem:

github.com/containers/toolbox does not provide the required /:/host bind mount needed for sosreport to properly access the host root filesystem.  



Version-Release number of selected component (if applicable):

toolbox-0.0.99-1.module+el8.4.0+10198+36d1d0e3


How reproducible:  100%


Steps to Reproduce:
1.  Install toolbox >= v0.0.99

2.  Create and run a toolbox container using the support-tools image

# toolbox create --image registry.redhat.io/rhel8/support-tools

# toolbox enter support-tools


3.  Run sosreport

# sosreport


Actual results:

Sosreport will fail to collect any data from the host root filesystem, and most plugins will fail to run at all


Expected results:

Toolbox provides the required /:/host bind mount to allow sosreport to run as designed



Additional info:

rhcos-toolbox is designed to use the settings baked into the support-tools image to run it properly

---
run() {
    if ! image_exists; then
        image_pull
    fi

    local runlabel=$(image_runlabel)
    if ! container_exists; then
        echo "Spawning a container '$TOOLBOX_NAME' with image '$TOOLBOX_IMAGE'"
        if [[ -z "$runlabel" ]]; then
            container_run
            return
        else
            echo "Detected RUN label in the container image. Using that as the default..."
            container_runlabel
[...]
container_runlabel() {
    if ! sudo podman container runlabel --name "$TOOLBOX_NAME" RUN "$TOOLBOX_IMAGE" 2>&1; then
        echo "$0: failed to runlabel on image '$TOOLBOX_IMAGE'"
        exit 1
    fi
}
---

# podman container runlabel run --display registry.redhat.io/rhel8/support-tools
command: podman run -it --name support-tools --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=support-tools -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest



Without the correct mount, sosreport is rendered unusable


⬢[root@toolbox ~]# sosreport 

sosreport (version 3.9)

WARNING: unable to set option for disabled or non-existing plugin (boot)
WARNING: unable to set option for disabled or non-existing plugin (boot)


This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.8yhv7tux and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
caught exception in plugin method "block.setup()"
caught exception in plugin method "block.setup()"
writing traceback to sos_logs/block-plugin-errors.txt
writing traceback to sos_logs/block-plugin-errors.txt
caught exception in plugin method "cgroups.setup()"
caught exception in plugin method "cgroups.setup()"
writing traceback to sos_logs/cgroups-plugin-errors.txt
writing traceback to sos_logs/cgroups-plugin-errors.txt
caught exception in plugin method "crypto.setup()"
caught exception in plugin method "crypto.setup()"
writing traceback to sos_logs/crypto-plugin-errors.txt
writing traceback to sos_logs/crypto-plugin-errors.txt
caught exception in plugin method "dbus.setup()"
caught exception in plugin method "dbus.setup()"
writing traceback to sos_logs/dbus-plugin-errors.txt
writing traceback to sos_logs/dbus-plugin-errors.txt
caught exception in plugin method "dnf.setup()"
caught exception in plugin method "dnf.setup()"
writing traceback to sos_logs/dnf-plugin-errors.txt
writing traceback to sos_logs/dnf-plugin-errors.txt
caught exception in plugin method "filesys.setup()"
caught exception in plugin method "filesys.setup()"
writing traceback to sos_logs/filesys-plugin-errors.txt
writing traceback to sos_logs/filesys-plugin-errors.txt
caught exception in plugin method "hardware.setup()"
caught exception in plugin method "hardware.setup()"
writing traceback to sos_logs/hardware-plugin-errors.txt
writing traceback to sos_logs/hardware-plugin-errors.txt
caught exception in plugin method "host.setup()"
caught exception in plugin method "host.setup()"
writing traceback to sos_logs/host-plugin-errors.txt
writing traceback to sos_logs/host-plugin-errors.txt
caught exception in plugin method "i18n.setup()"
caught exception in plugin method "i18n.setup()"
writing traceback to sos_logs/i18n-plugin-errors.txt
writing traceback to sos_logs/i18n-plugin-errors.txt
caught exception in plugin method "kernel.setup()"
caught exception in plugin method "kernel.setup()"
writing traceback to sos_logs/kernel-plugin-errors.txt
writing traceback to sos_logs/kernel-plugin-errors.txt
caught exception in plugin method "krb5.setup()"
caught exception in plugin method "krb5.setup()"
writing traceback to sos_logs/krb5-plugin-errors.txt
writing traceback to sos_logs/krb5-plugin-errors.txt
caught exception in plugin method "ldap.setup()"
caught exception in plugin method "ldap.setup()"
writing traceback to sos_logs/ldap-plugin-errors.txt
writing traceback to sos_logs/ldap-plugin-errors.txt
caught exception in plugin method "libraries.setup()"
caught exception in plugin method "libraries.setup()"
writing traceback to sos_logs/libraries-plugin-errors.txt
writing traceback to sos_logs/libraries-plugin-errors.txt
caught exception in plugin method "libvirt.setup()"
caught exception in plugin method "libvirt.setup()"
writing traceback to sos_logs/libvirt-plugin-errors.txt
writing traceback to sos_logs/libvirt-plugin-errors.txt
caught exception in plugin method "login.setup()"
caught exception in plugin method "login.setup()"
writing traceback to sos_logs/login-plugin-errors.txt
writing traceback to sos_logs/login-plugin-errors.txt
caught exception in plugin method "logrotate.setup()"
caught exception in plugin method "logrotate.setup()"
writing traceback to sos_logs/logrotate-plugin-errors.txt
writing traceback to sos_logs/logrotate-plugin-errors.txt
caught exception in plugin method "logs.setup()"
caught exception in plugin method "logs.setup()"
writing traceback to sos_logs/logs-plugin-errors.txt
writing traceback to sos_logs/logs-plugin-errors.txt
caught exception in plugin method "lvm2.setup()"
caught exception in plugin method "lvm2.setup()"
writing traceback to sos_logs/lvm2-plugin-errors.txt
writing traceback to sos_logs/lvm2-plugin-errors.txt
caught exception in plugin method "md.setup()"
caught exception in plugin method "md.setup()"
writing traceback to sos_logs/md-plugin-errors.txt
writing traceback to sos_logs/md-plugin-errors.txt
caught exception in plugin method "memory.setup()"
caught exception in plugin method "memory.setup()"
writing traceback to sos_logs/memory-plugin-errors.txt
writing traceback to sos_logs/memory-plugin-errors.txt
caught exception in plugin method "multipath.setup()"
caught exception in plugin method "multipath.setup()"
writing traceback to sos_logs/multipath-plugin-errors.txt
writing traceback to sos_logs/multipath-plugin-errors.txt
caught exception in plugin method "networking.setup()"
caught exception in plugin method "networking.setup()"
writing traceback to sos_logs/networking-plugin-errors.txt
writing traceback to sos_logs/networking-plugin-errors.txt
caught exception in plugin method "openhpi.setup()"
caught exception in plugin method "openhpi.setup()"
writing traceback to sos_logs/openhpi-plugin-errors.txt
writing traceback to sos_logs/openhpi-plugin-errors.txt
caught exception in plugin method "pam.setup()"
caught exception in plugin method "pam.setup()"
writing traceback to sos_logs/pam-plugin-errors.txt
writing traceback to sos_logs/pam-plugin-errors.txt
caught exception in plugin method "pci.setup()"
caught exception in plugin method "pci.setup()"
writing traceback to sos_logs/pci-plugin-errors.txt
writing traceback to sos_logs/pci-plugin-errors.txt
caught exception in plugin method "process.setup()"
caught exception in plugin method "process.setup()"
writing traceback to sos_logs/process-plugin-errors.txt
writing traceback to sos_logs/process-plugin-errors.txt
caught exception in plugin method "processor.setup()"
caught exception in plugin method "processor.setup()"
writing traceback to sos_logs/processor-plugin-errors.txt
writing traceback to sos_logs/processor-plugin-errors.txt
caught exception in plugin method "release.setup()"
caught exception in plugin method "release.setup()"
writing traceback to sos_logs/release-plugin-errors.txt
writing traceback to sos_logs/release-plugin-errors.txt
caught exception in plugin method "rpm.setup()"
caught exception in plugin method "rpm.setup()"
writing traceback to sos_logs/rpm-plugin-errors.txt
writing traceback to sos_logs/rpm-plugin-errors.txt
caught exception in plugin method "scsi.setup()"
caught exception in plugin method "scsi.setup()"
writing traceback to sos_logs/scsi-plugin-errors.txt
writing traceback to sos_logs/scsi-plugin-errors.txt
caught exception in plugin method "selinux.setup()"
caught exception in plugin method "selinux.setup()"
writing traceback to sos_logs/selinux-plugin-errors.txt
writing traceback to sos_logs/selinux-plugin-errors.txt
caught exception in plugin method "services.setup()"
caught exception in plugin method "services.setup()"
writing traceback to sos_logs/services-plugin-errors.txt
writing traceback to sos_logs/services-plugin-errors.txt
caught exception in plugin method "ssh.setup()"
caught exception in plugin method "ssh.setup()"
writing traceback to sos_logs/ssh-plugin-errors.txt
writing traceback to sos_logs/ssh-plugin-errors.txt
caught exception in plugin method "subscription_manager.setup()"
caught exception in plugin method "subscription_manager.setup()"
writing traceback to sos_logs/subscription_manager-plugin-errors.txt
writing traceback to sos_logs/subscription_manager-plugin-errors.txt
caught exception in plugin method "system.setup()"
caught exception in plugin method "system.setup()"
writing traceback to sos_logs/system-plugin-errors.txt
writing traceback to sos_logs/system-plugin-errors.txt
caught exception in plugin method "systemd.setup()"
caught exception in plugin method "systemd.setup()"
writing traceback to sos_logs/systemd-plugin-errors.txt
writing traceback to sos_logs/systemd-plugin-errors.txt
caught exception in plugin method "sysvipc.setup()"
caught exception in plugin method "sysvipc.setup()"
writing traceback to sos_logs/sysvipc-plugin-errors.txt
writing traceback to sos_logs/sysvipc-plugin-errors.txt
caught exception in plugin method "udev.setup()"
caught exception in plugin method "udev.setup()"
writing traceback to sos_logs/udev-plugin-errors.txt
writing traceback to sos_logs/udev-plugin-errors.txt
caught exception in plugin method "xfs.setup()"
caught exception in plugin method "xfs.setup()"
writing traceback to sos_logs/xfs-plugin-errors.txt
writing traceback to sos_logs/xfs-plugin-errors.txt
caught exception in plugin method "yum.setup()"
caught exception in plugin method "yum.setup()"
writing traceback to sos_logs/yum-plugin-errors.txt
writing traceback to sos_logs/yum-plugin-errors.txt
 Running plugins. Please wait ...

  Finishing plugins              [Running: systemd]                                       
  Finished running plugins                                                               
caught exception in plugin method "filesys.postproc()"
caught exception in plugin method "filesys.postproc()"
writing traceback to sos_logs/filesys-plugin-errors.txt
writing traceback to sos_logs/filesys-plugin-errors.txt
caught exception in plugin method "ldap.postproc()"
caught exception in plugin method "ldap.postproc()"
writing traceback to sos_logs/ldap-plugin-errors.txt
writing traceback to sos_logs/ldap-plugin-errors.txt
caught exception in plugin method "openhpi.postproc()"
caught exception in plugin method "openhpi.postproc()"
writing traceback to sos_logs/openhpi-plugin-errors.txt
writing traceback to sos_logs/openhpi-plugin-errors.txt
Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-toolbox-2021-03-17-jabxvmf.tar.xz

 Size	38.00KiB
 Owner	root
 md5	5524dbd2ffbfd21b85025ca3a6d3ac68

Please send this file to your support representative.


Sosreport would also place the resulting file into /host/var/tmp so that it lives on the host and not in the container
Comment 6 Debarshi Ray 2021-03-17 23:42:19 UTC 
github.com/containers/toolbox puts the host bind mounts in /run/host. If sosreport looked at /run/host, instead of /host, would it then work? Or are there yet more things missing?
Comment 7 Jake Hunsaker 2021-03-18 00:37:15 UTC 
sos would (should) use the /run/host mount point if toolbox also set the `HOST` env var to that path.
Comment 9 Debarshi Ray 2021-03-18 08:31:22 UTC 
Note that so far we've only rebased 'toolbox' to github.com/containers/toolbox in RHEL 8.4, not RHOCP (which I think CoreOS is a part of).

We are aware of this issue, it is tracked upstream at https://github.com/containers/toolbox/issues/689 It should be addressed in one of the following toolbox releases that will be included in one of the next RHEL 8 versions (might be even 8.5).

We understand that this impacts RHCOS. Hence the plan is not to rebase 'toolbox' in RHOCP until issues like this can be sorted out - as agreed with the CoreOS team (CC'ing Micah). Therefore, I don't think this qualifies for the 8.4 exception as it would need weeks to be properly addressed and tested upstream.
Comment 10 Derrick Ornelas 2021-03-22 20:55:43 UTC 
I tested this the other day, and I don't believe /run/host is the complete rootfs from the host.  Here's a test where sosreport fails immediately:

# toolbox create --image registry.redhat.io/rhel8/support-tools
Created container: support-tools
Enter with: toolbox enter support-tools

# toolbox enter support-tools

⬢[root@toolbox ~]# HOST=/run/host/ sosreport -v

sosreport (version 3.9)

Could not obtain installed package list


⬢[root@toolbox ~]# ls -l /run/host/
total 20
dr-xr-xr-x.   5 root root 4096 Mar 16 19:17 boot
drwxr-xr-x. 107 root root 8192 Mar 19 16:32 etc
drwxr-xr-x.  38 root root 1060 Mar 17 08:39 run
drwxrwxrwt.   8 root root  240 Mar 22 16:29 tmp
drwxr-xr-x.  12 root root  144 Mar 16 19:10 usr
drwxr-xr-x.  21 root root 4096 Mar 16 19:21 var


⬢[root@toolbox ~]# grep "/run/host" /proc/mounts | grep -v "/var/lib/containers"
/dev/mapper/rhel_rhel84-root /run/host/tmp xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/mapper/rhel_rhel84-root /run/host/usr xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
tmpfs /run/host/run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /run/host/run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=402044k,mode=700 0 0
tmpfs /run/host/run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
tmpfs /run/host/run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=402044k,mode=700,uid=1000,gid=1000 0 0
/dev/sda1 /run/host/boot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/mapper/rhel_rhel84-root /run/host/var xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/mapper/rhel_rhel84-root /run/host/etc xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0


I believe it expects the entire host rootfs to be made available
Comment 11 Jake Hunsaker 2021-04-16 20:56:56 UTC 
(In reply to Derrick Ornelas from comment #10)
> 
> I believe it expects the entire host rootfs to be made available


This is correct. The entire / filesystem from the host should be exposed via the mount point referenced by the HOST env var.
Comment 16 Alex Jia 2021-08-03 08:31:39 UTC 
This bug has been verified on toolbox-0.0.99.2^1.git660b6970e998-1.module+el8.5.0+12014+438a5746.

[root@kvm-07-guest24 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[root@kvm-07-guest24 ~]# rpm -q toolbox podman runc kernel
toolbox-0.0.99.2^1.git660b6970e998-1.module+el8.5.0+12014+438a5746.x86_64
podman-3.3.0-0.17.module+el8.5.0+12014+438a5746.x86_64
runc-1.0.1-3.module+el8.5.0+12014+438a5746.x86_64
kernel-4.18.0-325.el8.x86_64

[root@kvm-07-guest24 ~]# toolbox create --image registry.redhat.io/rhel8/support-tools
Created container: support-tools
Enter with: toolbox enter support-tools
[root@kvm-07-guest24 ~]# toolbox enter support-tools
[root@toolbox ~]# sosreport 
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report '

sosreport (version 4.0)

WARNING: unable to set option for disabled or non-existing plugin (boot)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.23pzu2zj and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
caught exception in plugin method "block.setup()"
writing traceback to sos_logs/block-plugin-errors.txt
caught exception in plugin method "cgroups.setup()"
writing traceback to sos_logs/cgroups-plugin-errors.txt
caught exception in plugin method "crypto.setup()"
writing traceback to sos_logs/crypto-plugin-errors.txt
caught exception in plugin method "dbus.setup()"
writing traceback to sos_logs/dbus-plugin-errors.txt
caught exception in plugin method "dnf.setup()"
writing traceback to sos_logs/dnf-plugin-errors.txt
caught exception in plugin method "filesys.setup()"
writing traceback to sos_logs/filesys-plugin-errors.txt
caught exception in plugin method "hardware.setup()"
writing traceback to sos_logs/hardware-plugin-errors.txt
caught exception in plugin method "host.setup()"
writing traceback to sos_logs/host-plugin-errors.txt
caught exception in plugin method "i18n.setup()"
writing traceback to sos_logs/i18n-plugin-errors.txt
caught exception in plugin method "kernel.setup()"
writing traceback to sos_logs/kernel-plugin-errors.txt
caught exception in plugin method "krb5.setup()"
writing traceback to sos_logs/krb5-plugin-errors.txt
caught exception in plugin method "ldap.setup()"
writing traceback to sos_logs/ldap-plugin-errors.txt
caught exception in plugin method "libraries.setup()"
writing traceback to sos_logs/libraries-plugin-errors.txt
caught exception in plugin method "libvirt.setup()"
writing traceback to sos_logs/libvirt-plugin-errors.txt
caught exception in plugin method "login.setup()"
writing traceback to sos_logs/login-plugin-errors.txt
caught exception in plugin method "logrotate.setup()"
writing traceback to sos_logs/logrotate-plugin-errors.txt
caught exception in plugin method "logs.setup()"
writing traceback to sos_logs/logs-plugin-errors.txt
caught exception in plugin method "lvm2.setup()"
writing traceback to sos_logs/lvm2-plugin-errors.txt
caught exception in plugin method "md.setup()"
writing traceback to sos_logs/md-plugin-errors.txt
caught exception in plugin method "memory.setup()"
writing traceback to sos_logs/memory-plugin-errors.txt
caught exception in plugin method "multipath.setup()"
writing traceback to sos_logs/multipath-plugin-errors.txt
caught exception in plugin method "networking.setup()"
writing traceback to sos_logs/networking-plugin-errors.txt
caught exception in plugin method "openhpi.setup()"
writing traceback to sos_logs/openhpi-plugin-errors.txt
caught exception in plugin method "pam.setup()"
writing traceback to sos_logs/pam-plugin-errors.txt
caught exception in plugin method "pci.setup()"
writing traceback to sos_logs/pci-plugin-errors.txt
caught exception in plugin method "process.setup()"
writing traceback to sos_logs/process-plugin-errors.txt
caught exception in plugin method "processor.setup()"
writing traceback to sos_logs/processor-plugin-errors.txt
caught exception in plugin method "release.setup()"
writing traceback to sos_logs/release-plugin-errors.txt
caught exception in plugin method "rpm.setup()"
writing traceback to sos_logs/rpm-plugin-errors.txt
caught exception in plugin method "scsi.setup()"
writing traceback to sos_logs/scsi-plugin-errors.txt
caught exception in plugin method "selinux.setup()"
writing traceback to sos_logs/selinux-plugin-errors.txt
caught exception in plugin method "services.setup()"
writing traceback to sos_logs/services-plugin-errors.txt
caught exception in plugin method "ssh.setup()"
writing traceback to sos_logs/ssh-plugin-errors.txt
caught exception in plugin method "subscription_manager.setup()"
writing traceback to sos_logs/subscription_manager-plugin-errors.txt
caught exception in plugin method "system.setup()"
writing traceback to sos_logs/system-plugin-errors.txt
caught exception in plugin method "systemd.setup()"
writing traceback to sos_logs/systemd-plugin-errors.txt
caught exception in plugin method "sysvipc.setup()"
writing traceback to sos_logs/sysvipc-plugin-errors.txt
caught exception in plugin method "udev.setup()"
writing traceback to sos_logs/udev-plugin-errors.txt
caught exception in plugin method "xfs.setup()"
writing traceback to sos_logs/xfs-plugin-errors.txt
caught exception in plugin method "yum.setup()"
writing traceback to sos_logs/yum-plugin-errors.txt
 Running plugins. Please wait ...

  Finishing plugins              [Running: systemd]                                       
  Finished running plugins                                                               
caught exception in plugin method "filesys.postproc()"
writing traceback to sos_logs/filesys-plugin-errors.txt
caught exception in plugin method "ldap.postproc()"
writing traceback to sos_logs/ldap-plugin-errors.txt
caught exception in plugin method "openhpi.postproc()"
writing traceback to sos_logs/openhpi-plugin-errors.txt
Creating compressed archive...

Your sosreport has been generated and saved in:
	/var/tmp/sosreport-toolbox-2021-08-03-noabqck.tar.xz

 Size	27.96KiB
 Owner	root
 md5	5eaa680c9a1b415d4af739f973a924e1

Please send this file to your support representative.

[root@toolbox ~]# ls -lh /var/tmp/sosreport-toolbox-2021-08-03-noabqck.tar.xz
-rw-------. 1 root root 28K Aug  3 04:28 /var/tmp/sosreport-toolbox-2021-08-03-noabqck.tar.xz
Comment 17 Derrick Ornelas 2021-08-17 00:56:41 UTC 
This doesn't seem to be working at all

# rpm -q toolbox
toolbox-0.0.99.2^1.git660b6970e998-1.module+el8.5.0+12018+8bfb9f5a.x86_64

# toolbox create --image registry.redhat.io/rhel8/support-tools
Created container: support-tools
Enter with: toolbox enter support-tools

# toolbox enter support-tools
⬢[root@toolbox ~]# ls -l /host
ls: cannot access '/host': No such file or directory


Without the /host mount sosreport will not run properly at all.
Comment 18 Derrick Ornelas 2021-08-17 01:13:26 UTC 
Here is what a successful run of sosreport in a container looks like:

# rpm -q toolbox
toolbox-0.0.8-1.module+el8.4.0+11311+9da8acfb.noarch

# toolbox 
Error: stat /var/lib/kubelet/config.json: no such file or directory
Would you like to manually authenticate to registry: 'registry.redhat.io' and try again? [y/N] y
Username: myuser
Password: 
Login Succeeded!
Trying to pull registry.redhat.io/rhel8/support-tools:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 296e14ee2414 skipped: already exists  
Copying blob 356f18f3a935 skipped: already exists  
Copying blob 3b4376bfc048 done  
Copying config 6ad911cd1a done  
Writing manifest to image destination
Storing signatures
6ad911cd1a8b431676ea12e242b02c3210a28660fdbfda3b6b43b53909fb48a6
Spawning a container 'toolbox-root' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...


[root@podman2 /]# findmnt /host
TARGET SOURCE                FSTYPE OPTIONS
/host  /dev/mapper/rhel-root xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota


[root@podman2 /]# sosreport 
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report '

sosreport (version 4.0)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/host/var/tmp/sos.db_ki48g and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:firewalld] skipped command 'nft list ruleset': required kmods missing: nfnetlink, nf_tables.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat. 
[plugin:networking] skipped command 'ip6tables -t mangle -nvL': required kmods missing: nf_tables, ip6table_mangle. 
[plugin:networking] skipped command 'ip6tables -t filter -nvL': required kmods missing: nf_tables, ip6table_filter. 
[plugin:networking] skipped command 'nft list ruleset': required kmods missing: nf_tables. 
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: tcp_diag, netlink_diag, inet_diag, udp_diag, unix_diag, af_packet_diag.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'iptables -vnxL': required kmods missing: nf_tables, iptable_filter. 
[plugin:networking] skipped command 'ip6tables -vnxL': required kmods missing: nf_tables, ip6table_filter. 
[plugin:sssd] skipped command 'sssctl config-check': required services missing: sssd. 
caught exception in plugin method "system.setup()"
writing traceback to sos_logs/system-plugin-errors.txt
[plugin:systemd] skipped command 'systemd-resolve --status': required services missing: systemd-resolved. 
[plugin:systemd] skipped command 'systemd-resolve --statistics': required services missing: systemd-resolved. 
 Running plugins. Please wait ...

  Finishing plugins              [Running: yum]                                           
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
	/host/var/tmp/sosreport-podman2-2021-08-16-ydholoi.tar.xz

 Size	16.88MiB
 Owner	root
 md5	5516a4ca445483e610cbbdc2065ca93b

Please send this file to your support representative.



Now, outside of the container the sosreport can be found at /var/tmp/sosreport-podman2-2021-08-16-ydholoi.tar.xz, and it contains data collected from the host and not from the container.
Comment 19 Derrick Ornelas 2021-08-17 14:37:32 UTC 
Looking closer I see that host / is now available at /run/host

⬢[root@toolbox ~]# findmnt /run/host
TARGET    SOURCE                       FSTYPE OPTIONS
/run/host /dev/mapper/rhel_rhel84-root xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota


⬢[root@toolbox ~]# ls -l /run/host/
total 28
lrwxrwxrwx.   1 root root    7 Apr 23  2020 bin -> usr/bin
dr-xr-xr-x.   5 root root 4096 Aug 16 16:35 boot
drwxr-xr-x.  21 root root 3160 Aug 17 10:26 dev
drwxr-xr-x. 159 root root 8192 Aug 16 20:20 etc
drwxr-xr-x.   3 root root   19 Jun 14 09:17 home
lrwxrwxrwx.   1 root root    7 Apr 23  2020 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Apr 23  2020 lib64 -> usr/lib64
drwxr-xr-x.   2 root root    6 Apr 23  2020 media
drwxr-xr-x.   3 root root   17 Aug  5 12:44 mnt
drwxr-xr-x.   2 root root    6 Apr 23  2020 opt
dr-xr-xr-x. 164 root root    0 Aug 16 16:40 proc
dr-xr-x---.   8 root root 4096 Aug 17 10:27 root
drwxr-xr-x.  53 root root 1480 Aug 17 10:29 run
lrwxrwxrwx.   1 root root    8 Apr 23  2020 sbin -> usr/sbin
drwxr-xr-x.   2 root root    6 Apr 23  2020 srv
dr-xr-xr-x.  13 root root    0 Aug 16 16:40 sys
drwxrwxrwt.  13 root root 4096 Aug 17 10:30 tmp
drwxr-xr-x.  12 root root  144 May 27 11:39 usr
drwxr-xr-x.  23 root root 4096 May 27 12:21 var


⬢[root@toolbox ~]# HOST=/run/host/ sosreport
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report '

sosreport (version 4.0)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/run/host/var/tmp/sos.kd_c6igh and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: ip6table_nat, nf_tables. 
[plugin:networking] skipped command 'ip6tables -t mangle -nvL': required kmods missing: nf_tables, ip6table_mangle. 
[plugin:networking] skipped command 'ip6tables -t filter -nvL': required kmods missing: ip6table_filter, nf_tables. 
[plugin:networking] skipped command 'nft list ruleset': required kmods missing: nf_tables. 
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: unix_diag, udp_diag, netlink_diag, inet_diag, af_packet_diag, tcp_diag.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'iptables -vnxL': required kmods missing: iptable_filter, nf_tables. 
[plugin:networking] skipped command 'ip6tables -vnxL': required kmods missing: ip6table_filter, nf_tables. 
[plugin:pcp] could not parse /etc/pcp.conf
[plugin:sar] sar: could not list /var/log/sa
 Running plugins. Please wait ...

  Starting 35/77 libvirt         [Running: chrony dnf host libvirt]                       caught exception in plugin method "libvirt.collect()"
writing traceback to sos_logs/libvirt-plugin-errors.txt
  Starting 68/77 systemd         [Running: chrony libvirt rpm subscription_manager systemd]caught exception in plugin method "systemd.collect()"
writing traceback to sos_logs/systemd-plugin-errors.txt
  Finishing plugins              [Running: libvirt rpm systemd]                            yum]h]d]i]
 Plugin rpm timed out


Creating compressed archive...

Your sosreport has been generated and saved in:
	/run/host/var/tmp/sosreport-toolbox-2021-08-17-hratsgv.tar.xz

 Size	70.45MiB
 Owner	root
 md5	a15025516b07401415e28f0c583aae8a

Please send this file to your support representative.



So we're just missing the HOST env variable set on the container.
Comment 20 Debarshi Ray 2021-08-20 08:31:19 UTC 
Moving back to MODIFIED since we were missing the HOST environment variable.

The changes for HOST are now in dist-git:
http://pkgs.devel.redhat.com/cgit/rpms/toolbox/commit/?h=stream-container-tools-rhel8&id=c7ca7ed72ee04a745d5002d56992812efbebced1

We are now waiting for a new build.
Comment 21 Tom Sweeney 2021-08-20 15:14:19 UTC 
@ajia @ypu Can we retest this one once the new build is completed please?
Comment 23 Alex Jia 2021-08-23 08:32:58 UTC 
This bug has been verified on toolbox-0.0.99.3-0.1.module+el8.5.0+12234+e4609207.

[root@hpe-dl380pgen8-02-vm-5 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[root@hpe-dl380pgen8-02-vm-5 ~]# rpm -q toolbox podman kernel
toolbox-0.0.99.3-0.1.module+el8.5.0+12234+e4609207.x86_64
podman-3.3.0-8.module+el8.5.0+12332+af0a4708.x86_64
kernel-4.18.0-330.el8.x86_64

[root@hpe-dl380pgen8-02-vm-5 ~]# toolbox enter support-tools
⬢[root@toolbox ~]# findmnt /run/host
TARGET    SOURCE                                           FSTYPE OPTIONS
/run/host /dev/mapper/rhel_hpe--dl380pgen8--02--vm--5-root xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
⬢[root@toolbox ~]# ls -l /run/host/
total 20
lrwxrwxrwx.   1 root root    7 Jun 21 05:52 bin -> usr/bin
dr-xr-xr-x.   5 root root 4096 Aug 20 05:38 boot
drwxr-xr-x.  19 root root 2900 Aug 20 09:37 dev
drwxr-xr-x. 104 root root 8192 Aug 23 01:17 etc
drwxr-xr-x.   3 root root   18 Aug 20 05:36 home
-rw-r--r--.   1 root root    0 Aug 20 05:35 initrd.root.sqsh
lrwxrwxrwx.   1 root root    7 Jun 21 05:52 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Jun 21 05:52 lib64 -> usr/lib64
drwxr-xr-x.   2 root root    6 Jun 21 05:52 media
drwxr-xr-x.   5 root root   55 Aug 20 05:38 mnt
drwxr-xr-x.   2 root root    6 Jun 21 05:52 opt
dr-xr-xr-x. 127 root root    0 Aug 20 04:53 proc
dr-xr-x---.   6 root root  270 Aug 23 01:17 root
drwxr-xr-x.  32 root root  860 Aug 23 04:08 run
lrwxrwxrwx.   1 root root    8 Jun 21 05:52 sbin -> usr/sbin
drwxr-xr-x.   2 root root    6 Jun 21 05:52 srv
dr-xr-xr-x.  13 root root    0 Aug 20 09:37 sys
drwxrwxrwt.  10 root root 4096 Aug 23 04:20 tmp
drwxr-xr-x.  12 root root  144 Aug 20 05:32 usr
drwxr-xr-x.  20 root root  278 Aug 20 09:37 var
⬢[root@toolbox ~]# HOST=/run/host/ sosreport
Please note the 'sosreport' command has been deprecated in favor of the new 'sos' command, E.G. 'sos report'.
Redirecting to 'sos report '

sosreport (version 4.0)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/run/host/var/tmp/sos.ksi3uk82 and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:networking] skipped command 'iptables -t nat -nvL': required kmods missing: nf_tables, iptable_nat. 
[plugin:networking] skipped command 'iptables -t mangle -nvL': required kmods missing: nf_tables, iptable_mangle. 
[plugin:networking] skipped command 'iptables -t filter -nvL': required kmods missing: nf_tables, iptable_filter. 
[plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat. 
[plugin:networking] skipped command 'ip6tables -t mangle -nvL': required kmods missing: nf_tables, ip6table_mangle. 
[plugin:networking] skipped command 'ip6tables -t filter -nvL': required kmods missing: nf_tables, ip6table_filter. 
[plugin:networking] skipped command 'nft list ruleset': required kmods missing: nf_tables. 
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, udp_diag, af_packet_diag, tcp_diag, netlink_diag, unix_diag.  Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'iptables -vnxL': required kmods missing: nf_tables, iptable_filter. 
[plugin:networking] skipped command 'ip6tables -vnxL': required kmods missing: nf_tables, ip6table_filter. 
 Running plugins. Please wait ...

  Starting 51/57 systemd         [Running: chrony host rpm systemd]                       caught exception in plugin method "systemd.collect()"
writing traceback to sos_logs/systemd-plugin-errors.txt
  Finishing plugins              [Running: systemd]                                       
Creating compressed archive...

Your sosreport has been generated and saved in:
	/run/host/var/tmp/sosreport-toolbox-2021-08-23-rhzcfgv.tar.xz

 Size	33.14MiB
 Owner	root
 md5	2e42a01179c8a491f98c1d26550793eb

Please send this file to your support representative.
Comment 24 Timothée Ravier 2021-08-24 18:34:57 UTC 
```
⬢[root@toolbox ~]# HOST=/run/host/ sosreport
```

This should work **without** the need to explicitly set the environment variable. Could you test that?

Thanks
Comment 27 Alex Jia 2021-08-26 02:43:29 UTC 
(In reply to Timothée Ravier from comment #24)
> ```
> ⬢[root@toolbox ~]# HOST=/run/host/ sosreport
> ```
> 
> This should work **without** the need to explicitly set the environment
> variable. Could you test that?
> 
> Thanks

It will be failed to generate report w/o HOST=/run/host environment
for toolbox-0.0.99.3-0.1.module+el8.5.0+12234+e4609207.x86_64
and toolbox-0.0.99.3-0.1.module+el8.5.0+12381+e822eb26.x86_64.
Comment 32 Debarshi Ray 2021-08-26 10:38:31 UTC 
(In reply to Alex Jia from comment #27)
> It will be failed to generate report w/o HOST=/run/host environment
> for toolbox-0.0.99.3-0.1.module+el8.5.0+12234+e4609207.x86_64
> and toolbox-0.0.99.3-0.1.module+el8.5.0+12381+e822eb26.x86_64.

That seems like an older build.

Setting the HOST environment variable was added in toolbox-0.0.99.3-0.3. The release is 0.3, not 0.1. See:
http://pkgs.devel.redhat.com/cgit/rpms/toolbox/commit/?h=stream-container-tools-rhel8&id=c7ca7ed72ee04a745d5002d56992812efbebced1

Maybe the toolbox module needs a new build?
Comment 33 Alex Jia 2021-08-26 13:09:24 UTC 
(In reply to Debarshi Ray from comment #32)
> That seems like an older build.
> 
> Setting the HOST environment variable was added in toolbox-0.0.99.3-0.3. The
> release is 0.3, not 0.1. See:
> http://pkgs.devel.redhat.com/cgit/rpms/toolbox/commit/?h=stream-container-
> tools-rhel8&id=c7ca7ed72ee04a745d5002d56992812efbebced1
> 
> Maybe the toolbox module needs a new build?

The current toolbox 0.0.99.3-0.1 is attached into https://errata.devel.redhat.com/advisory/70723, 
we need a new toolbox 0.0.99.3-0.3 is attached into errata.
Comment 35 Alex Jia 2021-08-26 14:53:46 UTC 
This bug has been verified on toolbox-0.0.99.3-0.3.module+el8.5.0+12372+12f82d56,
for details, please see [1]. 

In fact, I still met some issues about backwards compatibility with existing containers
created by the older implementation, it should be irrelevant w/ this bug, so I filed
a separated bug for it - bug 1998191.


[1] 

[root@kvm-07-guest10 ~]# rm -rf /run/.containerenv

NOTE: make sure there is no .containerenv file under the /run

[root@kvm-07-guest10 ~]# toolbox enter support-tools
⬢[root@toolbox ~]# echo $HOST
/run/host
⬢[root@toolbox ~]# sosreport 

XXX

 Running plugins. Please wait ...

  Starting 51/57 systemd         [Running: chrony host rpm systemd]                       caught exception in plugin method "systemd.collect()"
writing traceback to sos_logs/systemd-plugin-errors.txt
  Finishing plugins              [Running: systemd]                                       
Creating compressed archive...

Your sosreport has been generated and saved in:
	/run/host/var/tmp/sosreport-toolbox-2021-08-26-hcksucs.tar.xz

 Size	33.32MiB
 Owner	root
 md5	3961d8db276bf7ed72b22bf596dfbead

Please send this file to your support representative.
Comment 37 Jindrich Novy 2021-08-27 08:20:56 UTC 
This bug is now attached to https://errata.devel.redhat.com/advisory/70723
Comment 39 errata-xmlrpc 2021-11-09 17:37:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4154
Note You need to log in before you can comment on or make changes to this bug.