Bug 1395874

We have tested (or attempted to test) bug like this when Spacewalk 2.6 was to be released. This is a clone of that bug to get it re-tested on Satellite 5.8.0. In some cases bug can be completely irrelevant (in such cases I suggest to close the bug as NOTABUG), in some cases it might be wise to take the bug as a note that something changed in a given area and we might want to sanity-test that area (and mark the bug as VERIFIED with "SanityOnly" keyword) and in some cases bug might be totally relevant to Satellite.



+++ This bug was initially created as a clone of Bug #1365530 +++

+++ This bug was initially created as a clone of Bug #1091883 +++

Description of problem:
Inverted Advanced Search Results when downloaded in csv are missing 'Matching' field for BIOS related information.

Version-Release number of selected component (if applicable):


How reproducible:
The steps were reproduced by me on an internal test system

Steps to Reproduce:
1. Go to advance search, select search filed as BIOS and search for "QEMU QEMU 01/01/2007".
2. This will list all the system with BIOS " QEMU QEMU 01/01/2007 ".
3. No click check these options "Invert search results" and "Fine Grained Search results" and click search once again.
4. Now download the search results in .csv format.

Actual results:
the file has missing files for the BIOS information.

Expected results:
BIOS information must be available in the downloaded CSV file.

Additional info:
-- please check the downloaded .CSV file form my test

--- Additional comment from ashwini ladke on 2014-04-28 05:38:59 EDT ---

Not sure whether it has to do something with resolved bz#980482

--- Additional comment from Eric Herget on 2016-07-27 11:53:29 EDT ---

This issue with missing data in csv appears to only happen when "invert
search results" is checked.  It appears to impact ALL "Field to Search" options.  In fact, there are just a few options where the results page itself displays values.  The table below shows the results of my testing.

field                  value in results  value in CSV
Address                      no                no       (tested and same results for address1 and address2 fields)
Building                     no                no
Room                         no                no
Rack                         no                no
Description                  no                no       (this is hardware description - hw.description)
Driver                       no                no       (there can be multiple drivers - usb, ata_piix, etc)
Device ID                    no                no       (there can be multiple device ids - 7113, 293A, etc)
Vendor ID                    no                no       (there can be multiple vendor ids - 1B36, 1AF4, etc)
Hostname                     yes               no
IP Address                   no                no
IPv6 Address                 no                no
Installed Packages           no                no       
Needed Packages                                         (not instantly testable in my current setup)
System                       no                no       
BIOS                         no                no       
Asset Tag                    no                no
CPU Model                    no                no
CPU MHz Less Than            no                no
CPU MHz Greater Than         no                no
Number of CPUs Less Than     no                no
Number of CPUs Greater Than  no                no
RAM Less Than                no                no
RAM Greater Than             no                no
Days Since Last Check-in     yes               no       (for non-inverted search, display is # days, while csv contains date/timestamp)
Days Since First Registered  yes               no       (for non-inverted search, display is # days, while csv contains date/timestamp)
Name/Description             yes               no       (this searches name and description fields, on non-inverted search, the field that matches is included in csv)
ID                           no                no
Custom Info                  no                no
Snapshot Tag                 no                no
Running Kernel               no                no
UUID                         no                no


NOTES:
Many, but not all of these "Field to Search" options result in multiple fields or fields with multiple values being search.  As such, this likely prohibits them from being displayed in "inverted" search results - the UI and csv would have to display or contain *all* the values that did not match the search query.  The options that do search a single field that can only contain a single value *could* display results and include results in csv.  Currently only 4 options display result values for inverted search and I suspect a few more *could*.  However, no options include these values in downloaded csv.

There are a couple of search options that have a reciprocal option, which makes the use of "inverted" search largely unnecessary - CPU MHz, # of CPUs and RAM size all have Less Than and Greater Than options.  Sure, a reciprocal option may not be inclusive of the range boundary, while the inverted search is.  So technically there is a difference between reciprocal and inverted.

Section 7.5.1.3 of the Satellite 5.7 docs discuss "Advanced Search".  It's very minimal info.  No explanation of "inverted results", "fine grained", etc.

I think there are a couple options for addressing this BZ:

1) Remove the "Download CSV" link from inverted search results since no option currently puts the name and values of matched fields in the csv for inverted search results.
2) For inverted search results, don't include the Matching Field Name and Matching Field Value columns in the downloaded csv.
3) Do either option above, except for the few search options where the field values are displayed for inverted search results (hostname, days since last check-in, days since first registered, and name/description).  In those cases, allow the csv download and include the values displayed on the results page
4) Do the same as option 3, after first identifying other search options that search a single field that can contain only 1 value and updating the results display and csv download to include these values.

In all options above, possibly the search results page should be updated so it doesn't display matched value columns that contain nothing for inverted search.

The documentation should be updated with much more detail about how to use the Advanced Search facility and its options.

--- Additional comment from Eric Herget on 2016-08-10 07:32:27 EDT ---

After collecting the information above that describes which advanced search options result in what content displayed in the UI and what data in the downloaded csv, I began in depth code examination.

I found that the actual search is done by querying the search engine via xml rpc call, and then those results get augmented with a decorator that retrieves more data about each system in the results.

On inverted searches, there is never a matching field value returned.  This makes sense.  In addition, the data displayed on the screen is not always just a single field that matched (or didn't match) the query.  For example, when searching against "BIOS", 3 separate fields are displayed by concatenating dmiBiosVendor, dmiBiosVersion and dmiBiosRelease.

While considering options, we realized there is a use case for returning the list of systems that result from an inverted query as that list could be input to a process, script, etc for performing some operation on each system in the list.  So removing the option to download csv for inverted searches was not an option.

In addition, we also see value in having system information in the downloaded csv.

Ultimately, our chosen solution is to add several columns to the downloaded csv for all of the data available for each system that makes sense in the csv format.  The following fields are now included in every downloaded csv (inverted search or otherwise):
room, rack, building, address1, address2, city, state, country, dmiBiosVendor, dmiBiosVersion, dmiBiosRelease, dmiAsset, cpuModel, cpuMHz, cpuNumberOfCpus, ram, name, hostname, serverArchName, os, runningKernel and release.

The description field was not included because it contained content that included cr/lf that might not be handled cleanly by csv parsers.  Other fields that contain multiple values, such as drivers, were also not included.

--- Additional comment from Eric Herget on 2016-08-10 07:33:01 EDT ---

spacewalk.github:
a741e990845de6467140b95f30956bace4a5fe86

--- Additional comment from Lukáš Hellebrandt on 2016-11-14 05:06:06 EST ---

Verified on SW nightly from 2016-11-11.

Used reproducer from OP. When reproducing on sat570, the "Matching Field Value" was present but empty.

On SW nightly, the value is still empty. However, there are additional columns in the CSV, as mentioned in comment 1. Although the Matching Field Value is still empty, this behavior is explicitly mentioned as intended by development and I thus consider it correct.