Bug 1313907 - No valid host was found. Reason: No conductor service registered which supports driver pxe_ssh. (HTTP 400)
No valid host was found. Reason: No conductor service registered which suppor...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
unspecified Severity medium
: ---
: 8.0 (Liberty)
Assigned To: Imre Farkas
Raviv Bar-Tal
: Reopened, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-02 10:20 EST by Harry Rybacki
Modified: 2016-11-03 11:44 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-03 07:35:28 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Full console output from gate demonstrating failure (429.31 KB, text/plain)
2016-03-02 10:20 EST, Harry Rybacki
no flags Details
ironic-conductor log (14.38 KB, application/x-gzip)
2016-03-29 06:48 EDT, Harry Rybacki
no flags Details

  None (edit)
Description Harry Rybacki 2016-03-02 10:20:02 EST
Created attachment 1132368 [details]
Full console output from gate demonstrating failure

Description of problem:

Multiple builds failing on the OSPd-8 Ironic integration gate: https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RDO/view/rdo-manager/job/rdo-manager-integration-ironic-8_director/

Ironic is failing to register nodes.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. using rdo-manager install an undercloud 
2. register nodes: `source /home/stack/stackrc; openstack baremetal import --json instackenv.json`

Actual results:

Request returned failure status. No valid host was found. Reason: No conductor service registered which supports driver pxe_ssh. (HTTP 400)

Expected results:


Additional info:

Full console logs from failing gate are attached or visible onlone[1]

[1] - https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Owned%20Jobs/job/rdo-manager-integration-ironic-8_director/60/console
Comment 2 Harry Rybacki 2016-03-02 10:32:46 EST
Failure output in console:

[undercloud] => {"attempts": 10, "changed": true, "cmd": "source /home/stack/stackrc; openstack baremetal import --json instackenv.json;", "delta": "0:00:01.171291", "end": "2016-03-02 10:01:43.849593", "failed": true, "rc": 1, "start": "2016-03-02 10:01:42.678302", "warnings": []}

stderr: Request returned failure status.
No valid host was found. Reason: No conductor service registered which supports driver pxe_ssh. (HTTP 400)
Task failed as maximum retries was encountered
Comment 4 Harry Rybacki 2016-03-03 11:02:30 EST
Verified that the last passing build did have the `pxe_ssh` driver enabled in /etc/ironic/ironic.conf

    enabled_drivers=pxe_ipmitool,pxe_ssh,pxe_drac,pxe_ilo,pxe_wol

fwiw it is also enabled in the passing upstream gate[1]

[1] - http://logs.openstack.org/34/287134/1/check/gate-tempest-dsvm-ironic-pxe_ipa/9a8af1c/logs/etc/ironic/ironic.conf.txt.gz
Comment 6 Lucas Alvares Gomes 2016-03-24 12:22:48 EDT
Hi Harry,

Can you please upload the logs from the openstack-ironic-conductor service please?
Comment 8 Harry Rybacki 2016-03-24 13:09:43 EDT
(In reply to Lucas Alvares Gomes from comment #6)
> Hi Harry,
> 
> Can you please upload the logs from the openstack-ironic-conductor service
> please?

I need to reprovision a host and run through some stuff but I'll get those up.
Comment 10 Imre Farkas 2016-03-25 05:39:45 EDT
Putting back the needinfo flag as the ironic-conductor log is still missing.
Comment 11 Harry Rybacki 2016-03-29 06:48 EDT
Created attachment 1141190 [details]
ironic-conductor log
Comment 12 Imre Farkas 2016-03-29 06:55:39 EDT
ironic-conductor looks OK:
2016-03-28 16:42:33.594 8515 DEBUG ironic.cmd.conductor [-] enabled_drivers                = ['pxe_ipmitool', 'pxe_ssh', 'pxe_drac', 'pxe_ilo', 'pxe_wol'] log_opt_values /usr/lib/python2.7/s
ite-packages/oslo_config/cfg.py:2229

No error in the log, all driver seems to be loaded successfully.
Comment 18 Imre Farkas 2016-09-06 10:47:46 EDT
I am still not able to reproduce the issue. Can you please prepare an environment for me to debug?
Comment 19 Dmitry Tantsur 2016-09-06 11:24:00 EDT
Hello!

We've fixed the dependencies, so now the required package get pulled. Please feel free to reopen if you still have problems with it after updating to the latest packages.
Comment 20 Matt Young 2016-11-02 18:01:34 EDT
We're hitting this in CI as of today with:

https://trunk.rdoproject.org/centos7-newton/ef/c1/efc1fbe18783e9d24e46b99edb3282c87eb85244_96121e15

Could you please provide details on which specific packages are needed?  We have observed that restarting the openstack-ironic-conductor service allows this to succeed.

This is currently blocking CI / automation.
Comment 21 Dmitry Tantsur 2016-11-03 05:42:56 EDT
Hello,

I can't detect your problem by looking at repositories only :/ could you please provide at least ironic-conductor logs?
Comment 28 Justin Kilpatrick 2016-11-03 08:33:24 EDT
(In reply to Dmitry Tantsur from comment #21)
> Hello,
> 
> I can't detect your problem by looking at repositories only :/ could you
> please provide at least ironic-conductor logs?

I'm seeing the same issue on Newton images, here are my logs. 



https://paste.fedoraproject.org/469489/47817627/ < - Ironic Conductor 
https://paste.fedoraproject.org/469490/76344147  < - Ironic api
Comment 29 Dmitry Tantsur 2016-11-03 09:33:35 EDT
The same issue means "conductor crashed due to missing dracclient". This is not your case, please cooperate with other folks to file a separate bug. This one is fixed and won't be reopened even though you have vaguely similar symptoms.
Comment 30 Matt Young 2016-11-03 09:51:35 EDT
Dmitry: thanks for the feedback.  I'll open a new bug for what we're observing presently.

For those of us "new to the party" (and by "us" I mean "me" :) )could you please provide a link to the change referred to in comment #19 that resolved the similar in appearance (yet different root cause) issue? 

@Chris Jones: I'll update this bug with a link to the new one.
Comment 32 Matt Young 2016-11-03 11:44:37 EDT
Opened https://bugzilla.redhat.com/show_bug.cgi?id=1391602 to track t his per guidance.

Note You need to log in before you can comment on or make changes to this bug.