Bug 1080458

+++ This bug was initially created as a clone of Bug #1077223 +++

Description of problem:

A new spec and cert entry can be created via the "hwcert" command of the hwcert-client package, but subsequent runs fail to find the certification when searching:

Is this system being tested for certification? (y|n) y
response: y
Red Hat Catalog User Name: gnichols
response: gnichols
Password: 
Searching...
done.
Could not find any matching certification
What certification is this system being tested for? (new|existing|none) new
response: new
Opening a new specification...created specification ID 990404
http://hwcert-web2-test.devel.redhat.com/show.cgi?id=990404
Invalid vendor name has been set.


done.
Opening a new certification...created certification ID 990405
http://hwcert-web2-test.devel.redhat.com/show.cgi?id=990405

done.

[Later, restarting hwcert on the same certification:]


Is this system being tested for certification? (y|n) y
response: y
Red Hat Catalog User Name: gnichols
response: gnichols
Password: 
Searching...Error: searching for certification:
    Fault code: -32000
    Fault string: Application error
Continue? (y|n) y
response: y


Version-Release number of selected component (if applicable):

hwcert-client 1.6.5.2-20140316 and hwcert-client 1.7.0-20140317


How reproducible:

Every time.


Steps to Reproduce:
1. run hwcert select "new" from (new, existing, none), then quit.
2. run hwcert again

--- Additional comment from Huan Zhang on 2014-03-18 01:35:34 EDT ---

Hi Greg,

Which fields does hwcert-client use for the search?


Thanks for reporting.

Huan

--- Additional comment from Huan Zhang on 2014-03-18 02:37:40 EDT ---

After I re-built catalog on web2, the 500 error was gone.

Greg,

Could you try again to see if the issue is still there?

--- Additional comment from Greg Nichols on 2014-03-18 10:35:24 EDT ---

I retried it just now, same failure.

For parameters, I'm using:

Searching for certification:

        parameters = {
                      "product":rhelProduct,
                      "vendor":resultsDocument.getHardware(Tags.vendor),
                      "make":resultsDocument.getHardware(Tags.make),
                      "model":resultsDocument.getHardware(Tags.model),
                      "release":rhelVersion,
                      "arch":arch
                    }
 
Searching for specification:


        parameters = {
                      "vendor":resultsDocument.getHardware(Tags.vendor),
                      "make":resultsDocument.getHardware(Tags.make),
                      "product":"Hardware Layer",
                      "model":resultsDocument.getHardware(Tags.model)
                    }

Both calls add:

            parameters["login"] = self.login
            parameters["password"] = self.password


Note that vendor, make and model may not match up to any known by the catalog.

--- Additional comment from Greg Nichols on 2014-03-18 10:51:42 EDT ---

An example run (some debug printing added):

Please verify the hardware product information:
    Vendor:Intel Corporation
    Make:S2600CP
    Model:S2600CP
    Product URL:
    Specification URL:
    Support URL:
    category (Desktop/Workstation|Laptop|Component/Peripheral|Server) Server

Is this system being tested for certification? (y|n) y
response: y
Red Hat Catalog User Name: gnichols
response: gnichols
Password: 

Search Parameters:
    product:Red Hat Enterprise Linux
    vendor:Intel Corporation
    make:S2600CP
    release:7
    model:S2600CP
    password:****
    arch:x86_64
    login:gnichols

Searching...Error: searching for certification:
    Fault code: -32000
    Fault string: Application error
Continue? (y|n) 

My guess is that this error happens when the certification is found, since I don't see it the first time I run a new system.

--- Additional comment from Huan Zhang on 2014-03-18 22:31:26 EDT ---

(In reply to Greg Nichols from comment #4)
> An example run (some debug printing added):
> 
> Please verify the hardware product information:
>     Vendor:Intel Corporation
>     Make:S2600CP
>     Model:S2600CP
>     Product URL:
>     Specification URL:
>     Support URL:
>     category (Desktop/Workstation|Laptop|Component/Peripheral|Server) Server
> 
> Is this system being tested for certification? (y|n) y
> response: y
> Red Hat Catalog User Name: gnichols
> response: gnichols
> Password: 
> 
> Search Parameters:
>     product:Red Hat Enterprise Linux
>     vendor:Intel Corporation
>     make:S2600CP
>     release:7
>     model:S2600CP
>     password:****
>     arch:x86_64
>     login:gnichols
> 
> Searching...Error: searching for certification:
>     Fault code: -32000
>     Fault string: Application error
> Continue? (y|n) 
> 
> My guess is that this error happens when the certification is found, since I
> don't see it the first time I run a new system.

Greg,

Thank you for the detailed reproducing information. The example is very helpful, I reproduced it directly using xmlrpc call.

I will troubleshot this today. Will let this bug updated, when I get any progress.

Thanks!

--- Additional comment from Huan Zhang on 2014-03-18 23:08:45 EDT ---


curl -d "<?xml version='1.0'?><methodCall><methodName>Search.simple</methodName><params><param><value><struct><member><name>product</name><value><string>Red Hat Enterprise Linux</string></value></member><member><name>vendor</name><value><string>Intel Corporation</string></value></member><member><name>make</name><value><string>S2600CP</string></value></member><member><name>model</name><value><string>S2600CP</string></value></member><member><name>release</name><value><string>7</string></value></member><member><name>arch</name><value><string>x86_64</string></value></member><member><name>login</name><value><string>hzhang</string></value></member><member><name>password</name><value><string>******</string></value></member></struct></value></param></params></methodCall>" -H "Content-Type: text/xml" http://hwcert-web2-test.devel.redhat.com/xmlrpc.cgi

Run with the about xmlrpc call the second time, I got the following error:

<?xml version="1.0" encoding="UTF-8"?><methodResponse><fault><value><struct><member><name>faultString</name><value><string>Application error</string></value></member><member><name>faultCode</name><value><int>-32000</int></value></member></struct></value></fault></methodResponse>

--- Additional comment from Huan Zhang on 2014-03-19 01:31:30 EDT ---

Hi Greg,

Debugging on the web2 server, I have gotten the root cause of this error. This error is caused by the timeout of bugzilla search.

To let you understand this, I will introduce the workflow of Simple.search briefly:

When we call the catalog xmlrpc Simple.search, Catalog calls bugzilla xmlrpc Bug.search to search bugs with the given params. Bucause the bugzilla server is not very stable and sometimes its xmlrpc call gets timeout or some unexpected error, catalog retries at most 4 times to get the search results from bugzilla.

The problem is that, if the bugzilla search takes too long time(more than 60 secs), catalog xmlrpc call will be killed upexpected. The upexpected termination of xmlrpc call will cause an application error return.


To avoid this issue, I suggest you also using a retry mechanism in v7 to do Simple.search.

Thanks,
Huan

--- Additional comment from Huan Zhang on 2014-03-19 01:33:20 EDT ---

The timeout happens more frequently especially when doing bugs searching.

--- Additional comment from Greg Nichols on 2014-03-19 09:13:18 EDT ---

Are you saying that searching for a spec or cert will typically take several minutes?

If so, I don't think we can implement this feature in the hwcert-client.

--- Additional comment from Greg Nichols on 2014-03-19 10:02:25 EDT ---

Searches either take 2 seconds, or 60, and there seems to me no pattern I can discern.  This is true for both unsuccessful and successful searches.

I wrote a test driver and here's an example output:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:S2600CP
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:01:02.244012
Error: searching for specification:
    Fault code: -32000
    Fault string: Application error
Continue? (y|n) y
response: y
Try: 1, id=0
Searching:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:S2600CP
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:00:02.913702

done.
Could not find any matching specification
Try: 2, id=0
Searching:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:S2600CP
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:01:01.883766
Error: searching for specification:
    Fault code: -32000
    Fault string: Application error
Continue? (y|n) y
response: y
Try: 3, id=0
Searching:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:S2600CP
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:01:01.917852
Error: searching for specification:
    Fault code: -32000
    Fault string: Application error
Continue? (y|n) 




Searches that succeede show the same problem:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:Unknown
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:01:01.924612
Error: searching for specification:
    Fault code: -32000
    Fault string: Application error
Continue? (y|n) y
response: y
Try: 1, id=0
Searching:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:Unknown
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:00:02.787530

done.
Found specification ID 990406
See http://hwcert-web2-test.devel.redhat.com/show.cgi?id=990406
Is this correct? (y|n) y
response: y
Error: searching for specification:
global name 'certificationIDs' is not defined
Try: 2, id=0
Searching:

Search Parameters:
    product:Hardware Layer
    vendor:Intel Corporation
    make:Unknown
    model:S2600CP
    password:****
    login:gnichols

Searching...Search took 0:01:01.857345
Error: searching for specification:
    Fault code: -32000
    Fault string: Application error



At a minimum, the catalog's fault string should indicate that the search timed-out.  In addition, I don't understand why the search should take more than a minute, except in rare cases, and it seems to happen most of the time.

--- Additional comment from Huan Zhang on 2014-03-20 04:21:40 EDT ---

@Rob,

Search without make, I still got timeout and -32000 error returned.