RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1992938 - SAP NW Plugin exception "IndexError: list index out of range"
Summary: SAP NW Plugin exception "IndexError: list index out of range"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sos
Version: 8.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: Miroslav Hradílek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-12 04:38 UTC by Rajesh Dulhani
Modified: 2023-09-18 00:29 UTC (History)
12 users (show)

Fixed In Version: sos-4.3-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-08 10:50:23 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)
sosreport from the client system (14.97 MB, application/x-xz)
2021-08-12 04:38 UTC, Rajesh Dulhani 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github sosreport sos pull 2643 0 None Merged [sapnw] Fix IndexError exception 2022-03-01 08:09:56 UTC
Github sosreport sos pull 2647 0 None Merged [sapnw] Add condition for length of fields var 2022-03-01 08:09:56 UTC
Red Hat Issue Tracker RHELPLAN-93328 0 None None None 2021-08-12 04:40:09 UTC
Red Hat Product Errata RHBA-2022:7732 0 None None None 2022-11-08 10:51:06 UTC

Description Rajesh Dulhani 2021-08-12 04:38:23 UTC 
Created attachment 1813314 [details]
sosreport from the client system

Description of problem:


Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
[plugin:firewalld] skipped command 'nft list ruleset': required kmods missing: nfnetlink, nf                                                                                 _tables.  Use '--allow-system-changes' to enable collection.
[plugin:ipa] skipped command 'getcert list': required services missing: certmonger.
[plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: ip6tabl                                                                                 e_nat.
[plugin:networking] skipped command 'ip6tables -t mangle -nvL': required kmods missing: ip6t                                                                                 able_mangle.
[plugin:networking] skipped command 'ip6tables -t filter -nvL': required kmods missing: ip6t                                                                                 able_filter.
[plugin:networking] skipped command 'nft list ruleset': required kmods missing: nf_tables.
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.  Us                                                                                 e '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, udp_di                                                                                 ag, tcp_diag, netlink_diag, af_packet_diag, unix_diag.  Use '--allow-system-changes' to enab                                                                                 le collection.
[plugin:networking] skipped command 'ip6tables -vnxL': required kmods missing: ip6table_filt                                                                                 er.
caught exception in plugin method "sapnw.setup()"
caught exception in plugin method "sapnw.setup()"
writing traceback to sos_logs/sapnw-plugin-errors.txt
writing traceback to sos_logs/sapnw-plugin-errors.txt
 Running plugins. Please wait ...

  Finishing plugins              [Running: yum]                                           ]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-li-lc-1437-2021-08-04-rtskgnq.tar.xz

 Size   14.97MiB
 Owner  root
 md5    1557e9faf61fa030cc9304c982b7b5c8

Please send this file to your support representative.
~~~

sos_logs/sapnw-plugin-errors.txt contains:
~~~
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 981, in setup
    plug.setup()
  File "/usr/lib/python2.7/site-packages/sos/plugins/sapnw.py", line 135, in setup
    self.collect_list_instances()
  File "/usr/lib/python2.7/site-packages/sos/plugins/sapnw.py", line 44, in collect_list_instances
    sid = fields[3]
IndexError: list index out of range
~~~


Version-Release number of selected component (if applicable):

  SAPHostAgent.x86_64 0:721.51-0
  Package - sos-3.9-5.el7_9.6.noarch.rpm



How reproducible:


Steps to Reproduce:
 
Reproducer is simple:
- Install SAPHostAgent RPM
- Run  'sudo sosreport -o sapnw'

See below the proof on a system where the plugin first works and then after intsalling SAPHostAgent rpm it gives an exception:

~~~
[Azure] vrempet-admin@li-lc-2622 ~
$ sudo sosreport -o sapnw

sosreport (version 3.9)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.lQPoFe and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   sapnw           [Running: sapnw]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-li-lc-2622-2021-08-05-cezxrru.tar.xz

 Size   1.08KiB
 Owner  root
 md5    d402734ea2b06a22bef72ded95bf1b4a

Please send this file to your support representative.


[Azure] vrempet-admin@li-lc-2622 ~
$ sudo yum install SAPHostAgent
Loaded plugins: changelog, enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
Hilti_HOICI_HOICI-2_1_99-ci                                                                                            | 2.1 kB  00:00:00
Hilti_HOIOS_HOIOS-7_9_99-ci                                                                                            | 2.1 kB  00:00:00
Hilti_HOIPRODUCTS_HOIPRODUCTS-2_0_99-ci                                                                                | 2.1 kB  00:00:00
Hilti_HOIRHEL_HOIRHEL-7_9-d20210801                                                                                    | 2.1 kB  00:00:00
rhel-7-server-optional-rpms                                                                                            | 1.8 kB  00:00:00
rhel-7-server-rpms                                                                                                     | 2.0 kB  00:00:00
rhel-7-server-satellite-tools-6.9-rpms                                                                                 | 2.1 kB  00:00:00
rhel-sap-for-rhel-7-server-rpms                                                                                        | 2.0 kB  00:00:00
rhel-server-rhscl-7-rpms                                                                                               | 2.0 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package SAPHostAgent.x86_64 0:721.51-0 will be installed
y--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================
 Package                      Arch                   Version                    Repository                                               Size
==============================================================================================================================================
Installing:
 SAPHostAgent                 x86_64                 721.51-0                   Hilti_HOIPRODUCTS_HOIPRODUCTS-2_0_99-ci                  73 M

Transaction Summary
==============================================================================================================================================
Install  1 Package

Total download size: 73 M
Installed size: 225 M
Is this ok [y/d/N]: y
Downloading packages:
SAPHostAgent-721.51-0.x86_64.rpm                                                                                       |  73 MB  00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
You should set a password for the new user "sapadm" after the package has been installed.
  Installing : SAPHostAgent-721.51-0.x86_64                                                                                               1/1
-> Start /usr/sap/hostctrl/exe/saphostexec pf=/usr/sap/hostctrl/exe/host_profile <-
start hostcontrol using profile /usr/sap/hostctrl/exe/host_profile
Uploading Package Profile
Loaded plugins: product-id, subscription-manager
  Verifying  : SAPHostAgent-721.51-0.x86_64                                                                                               1/1

Installed:
  SAPHostAgent.x86_64 0:721.51-0

Complete!
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager

[Azure] vrempet-admin@li-lc-2622 ~
$ sudo sosreport -o sapnw

sosreport (version 3.9)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.0IiEVm and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
caught exception in plugin method "sapnw.setup()"
caught exception in plugin method "sapnw.setup()"
writing traceback to sos_logs/sapnw-plugin-errors.txt
writing traceback to sos_logs/sapnw-plugin-errors.txt
 Running plugins. Please wait ...

  Starting 1/1   sapnw           [Running: sapnw]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-li-lc-2622-2021-08-05-vyhyjcd.tar.xz

 Size   1.49KiB
 Owner  root
 md5    b90b69edd7734468352fd69fc8cb8553

Please send this file to your support representative.

[crash/Azure] root@li-lc-2622:~#
~~~

Actual results:

sos_logs/sapnw-plugin-errors.txt contains:
~~~
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 981, in setup
    plug.setup()
  File "/usr/lib/python2.7/site-packages/sos/plugins/sapnw.py", line 135, in setup
    self.collect_list_instances()
  File "/usr/lib/python2.7/site-packages/sos/plugins/sapnw.py", line 44, in collect_list_instances
    sid = fields[3]
IndexError: list index out of range
~~~


Expected results:

Should not throw the exception.


Additional info:
Comment 3 Jose Castillo 2021-08-12 06:41:06 UTC 
The fail occurs here:


 38         sidsunique = set()
 39         # Cycle through all the instances, get 'sid', 'instance_number'
 40         # and 'vhost' to determine the proper profile
 41         for inst_line in inst_out['output'].splitlines():
 42             if "DAA" not in inst_line:
 43                 fields = inst_line.strip().split()
 44                 sid = fields[3]
 45                 inst = fields[5]
 46                 vhost = fields[7]
 47                 sidsunique.add(sid)

And we populate inst_line from inst_out['output'], which is gathered via the following command:

 31         inst_out = self.collect_cmd_output(
 32             "/usr/sap/hostctrl/exe/saphostctrl -function ListInstances",
 33             suggest_filename="SAPInstances"
 34         )


So what's the result of running:

/usr/sap/hostctrl/exe/saphostctrl -function ListInstances

Directly in this server after installing SAPHostAgent?
Comment 4 Peter Vreman 2021-08-12 06:58:21 UTC 
~~~
[Azure] vrempet-admin@li-lc-2622 ~
$ sudo /usr/sap/hostctrl/exe/saphostctrl -function ListInstances
No instances found

[Azure] vrempet-admin@li-lc-2622 ~
$
~~~
Comment 5 Jose Castillo 2021-08-12 07:05:48 UTC 
Thank you for running the command Peter. I don't think we deal with this result in the plugin, be it the one reported here or the latest code upstream. I'm writing a quick patch to deal with this case - would you be able to test it?
Comment 6 Pavel Moravec 2021-08-12 07:06:54 UTC 
.. and the return code of that command is 0. We assumed that nonzero would be returned in that case, hence 

https://github.com/sosreport/sos/blob/legacy-3.9/sos/plugins/sapnw.py#L35-L36

will prevent continuation (and the traceback).

I guess replacing line https://github.com/sosreport/sos/blob/legacy-3.9/sos/plugins/sapnw.py#L42 :

            if "DAA" not in inst_line:

by:

            if "DAA" not in inst_line and not inst_line.startswith("No instances found"):

will prevent the uncaught exception?
Comment 7 Jose Castillo 2021-08-12 07:12:38 UTC 
(In reply to Pavel Moravec from comment #6)

> I guess replacing line
> https://github.com/sosreport/sos/blob/legacy-3.9/sos/plugins/sapnw.py#L42 :
> 
>             if "DAA" not in inst_line:
> 
> by:
> 
>             if "DAA" not in inst_line and not inst_line.startswith("No
> instances found"):
> 
> will prevent the uncaught exception?

Yes, I just tested it and it seems to work perfectly. The result captured is:

# cat sosreport-localhost-2021-08-12-slaxlbr/sos_commands/sapnw/SAPInstances 
No instances found
Comment 8 Peter Vreman 2021-08-12 07:25:28 UTC 
Confirming proposed patch works:

~~~
[Azure] vrempet-admin@li-lc-2622 ~
$ sudo /usr/sap/hostctrl/exe/saphostctrl -function ListInstances
No instances found

[Azure] vrempet-admin@li-lc-2622 ~
$ sudo sosreport -o sapnw --batch

sosreport (version 3.9)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.ivjXGk and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
caught exception in plugin method "sapnw.setup()"
caught exception in plugin method "sapnw.setup()"
writing traceback to sos_logs/sapnw-plugin-errors.txt
writing traceback to sos_logs/sapnw-plugin-errors.txt
 Running plugins. Please wait ...

  Starting 1/1   sapnw           [Running: sapnw]

  Finished running plugins

Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-li-lc-2622-2021-08-12-hoefbcd.tar.xz

 Size   1.82KiB
 Owner  root
 md5    5434ff417c9e4d2e9f383c8d2943ec4b

Please send this file to your support representative.


[Azure] vrempet-admin@li-lc-2622 ~
$ sudo vi /usr/lib/python2.7/site-packages/sos/plugins/sapnw.py

[Azure] vrempet-admin@li-lc-2622 ~
$ diff -u /usr/lib/python2.7/site-packages/sos/plugins/sapnw.py.210812-1 /usr/lib/python2.7/site-packages/sos/plugins/sapnw.py
--- /usr/lib/python2.7/site-packages/sos/plugins/sapnw.py.210812-1      2021-06-16 07:51:00.000000000 +0000
+++ /usr/lib/python2.7/site-packages/sos/plugins/sapnw.py       2021-08-12 07:23:44.781100030 +0000
@@ -39,7 +39,7 @@
         # Cycle through all the instances, get 'sid', 'instance_number'
         # and 'vhost' to determine the proper profile
         for inst_line in inst_out['output'].splitlines():
-            if "DAA" not in inst_line:
+           if "DAA" not in inst_line and not inst_line.startswith("No instances found"):
                 fields = inst_line.strip().split()
                 sid = fields[3]
                 inst = fields[5]

[Azure] vrempet-admin@li-lc-2622 ~
$

[Azure] vrempet-admin@li-lc-2622 ~
$ sudo sosreport -o sapnw --batch

sosreport (version 3.9)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.yLNZet and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   sapnw           [Running: sapnw]

  Finished running plugins

Creating compressed archive...

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-li-lc-2622-2021-08-12-faprlin.tar.xz

 Size   1.67KiB
 Owner  root
 md5    2d5ec31cd238e4897105261fab1c232b

Please send this file to your support representative.
~~~
Comment 9 Peter Vreman 2021-08-12 07:30:03 UTC 
Safe code requires also that the outcome of 'fields = inst_line.strip().split()' is validated that the number of returned elements >=7 before you access the elements like 'fields[7]'
Comment 10 Jose Castillo 2021-08-12 07:51:41 UTC 
(In reply to Peter Vreman from comment #9)
> Safe code requires also that the outcome of 'fields =
> inst_line.strip().split()' is validated that the number of returned elements
> >=7 before you access the elements like 'fields[7]'

Do you have an example of the output with no vhost? 
And what value of vhost is assumed if it doesn't appear in the output - localhost, or something similar?
Comment 11 Peter Vreman 2021-08-12 08:25:05 UTC 
Jose,

It was just normal code-review remark that the array indexing is done without a protection that enough elements are available. 
I have no real-world output available, but i am also not looking into the 


I have a real-world example where the output is an error, but that is gracefully handled by sosreport (older 3.7 version because it is RHEL7.7-E4S)
~~~
vrempet-admin@li-lc-1875 ~
$ sudo /usr/sap/hostctrl/exe/saphostctrl -function ListInstances
Error: NIECONN_BROKEN (Connection reset by peer), NiRawRead failed in plugin_sapfrecv()

vrempet-admin@li-lc-1875 ~
$ echo $?
1

vrempet-admin@li-lc-1875 ~
$ sudo sosreport -o sapnw --batch

sosreport (version 3.7)

This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.

An archive containing the collected information will be generated in
/var/tmp/sos.ATUw_D and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

  https://access.redhat.com/support/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   sapnw           [Running: sapnw]

  Finished running plugins
~~~

Peter
Comment 12 Pavel Moravec 2021-08-16 08:01:41 UTC 
I am in favour of improving the plugin robustness, if we are already changing it. Something like:

         # Cycle through all the instances, get 'sid', 'instance_number'
         # and 'vhost' to determine the proper profile
         for inst_line in inst_out['output'].splitlines():
            fields = inst_line.strip().split()
            if "DAA" not in inst_line and not inst_line.startswith("No instances found") and len(fields)>6:
                 sid = fields[3]
                 inst = fields[5]

?
Comment 13 Jose Castillo 2021-08-17 12:58:10 UTC 
Pavel, I'm trying to get a machine in the lab to test this command, and how the output varies. Once I have it we can adjust more.
Comment 17 Pavel Moravec 2022-03-01 08:07:04 UTC 
As agreed internally, no RHEL7 backport is further needed and a fix in RHEL8 is sufficient.

The main fix (https://github.com/sosreport/sos/pull/2643) will be released in RHEL8.6 and its robustness enhancement (https://github.com/sosreport/sos/pull/2647) will land in RHEL8.7. Reassigning the BZ to RHEL8.
Comment 19 Pavel Moravec 2022-03-18 10:56:11 UTC 
Hello,
as our QE might not have available systems for testing the fix, would you be able to verify the fix once available (as a candidate package for 8.7)?
Comment 22 Pavel Moravec 2022-03-24 11:12:35 UTC 
How to test this:

mock /usr/sap/hostctrl/exe/saphostctrl to *either* (test both options)

- print "No instances found" and return nonzero (#2643)
- print a line with less than 7 words (i.e. the same) and return zero (#2647)
Comment 23 Pavel Moravec 2022-04-22 13:19:21 UTC 
(In reply to Pavel Moravec from comment #22)
> How to test this:
> 
> mock /usr/sap/hostctrl/exe/saphostctrl to *either* (test both options)
> 
> - print "No instances found" and return nonzero (#2643)
> - print a line with less than 7 words (i.e. the same) and return zero (#2647)

Oh wrong description as nonzero return code means nothing to collect. Proper tests:

mock /usr/sap/hostctrl/exe/saphostctrl to return 0 and *either* (test both options)

- print "No instances found" (#2643)
- print a line with other text having less than 7 words (#2647)
Comment 30 errata-xmlrpc 2022-11-08 10:50:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7732
Comment 31 Red Hat Bugzilla 2023-09-18 00:29:44 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.