Bug 1313907

Summary: No valid host was found. Reason: No conductor service registered which supports driver pxe_ssh. (HTTP 400)
Product: Red Hat OpenStack Reporter: Harry Rybacki <hrybacki>
Component: openstack-ironicAssignee: Imre Farkas <ifarkas>
Status: CLOSED CURRENTRELEASE QA Contact: Raviv Bar-Tal <rbartal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: abregman, acruz, athomas, cchen, chjones, dtantsur, dtrainor, hrybacki, jkilpatr, jschluet, matyoung, mburns, rbartal, rhel-osp-director-maint, rlandy, srevivo
Target Milestone: ---Keywords: Reopened, ZStream
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 11:35:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Full console output from gate demonstrating failure
none
ironic-conductor log none

Description Harry Rybacki 2016-03-02 15:20:02 UTC
Created attachment 1132368 [details]
Full console output from gate demonstrating failure

Description of problem:

Multiple builds failing on the OSPd-8 Ironic integration gate: https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RDO/view/rdo-manager/job/rdo-manager-integration-ironic-8_director/

Ironic is failing to register nodes.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. using rdo-manager install an undercloud 
2. register nodes: `source /home/stack/stackrc; openstack baremetal import --json instackenv.json`

Actual results:

Request returned failure status. No valid host was found. Reason: No conductor service registered which supports driver pxe_ssh. (HTTP 400)

Expected results:


Additional info:

Full console logs from failing gate are attached or visible onlone[1]

[1] - https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Owned%20Jobs/job/rdo-manager-integration-ironic-8_director/60/console

Comment 2 Harry Rybacki 2016-03-02 15:32:46 UTC
Failure output in console:

[undercloud] => {"attempts": 10, "changed": true, "cmd": "source /home/stack/stackrc; openstack baremetal import --json instackenv.json;", "delta": "0:00:01.171291", "end": "2016-03-02 10:01:43.849593", "failed": true, "rc": 1, "start": "2016-03-02 10:01:42.678302", "warnings": []}

stderr: Request returned failure status.
No valid host was found. Reason: No conductor service registered which supports driver pxe_ssh. (HTTP 400)
Task failed as maximum retries was encountered

Comment 4 Harry Rybacki 2016-03-03 16:02:30 UTC
Verified that the last passing build did have the `pxe_ssh` driver enabled in /etc/ironic/ironic.conf

    enabled_drivers=pxe_ipmitool,pxe_ssh,pxe_drac,pxe_ilo,pxe_wol

fwiw it is also enabled in the passing upstream gate[1]

[1] - http://logs.openstack.org/34/287134/1/check/gate-tempest-dsvm-ironic-pxe_ipa/9a8af1c/logs/etc/ironic/ironic.conf.txt.gz

Comment 6 Lucas Alvares Gomes 2016-03-24 16:22:48 UTC
Hi Harry,

Can you please upload the logs from the openstack-ironic-conductor service please?

Comment 8 Harry Rybacki 2016-03-24 17:09:43 UTC
(In reply to Lucas Alvares Gomes from comment #6)
> Hi Harry,
> 
> Can you please upload the logs from the openstack-ironic-conductor service
> please?

I need to reprovision a host and run through some stuff but I'll get those up.

Comment 10 Imre Farkas 2016-03-25 09:39:45 UTC
Putting back the needinfo flag as the ironic-conductor log is still missing.

Comment 11 Harry Rybacki 2016-03-29 10:48:26 UTC
Created attachment 1141190 [details]
ironic-conductor log

Comment 12 Imre Farkas 2016-03-29 10:55:39 UTC
ironic-conductor looks OK:
2016-03-28 16:42:33.594 8515 DEBUG ironic.cmd.conductor [-] enabled_drivers                = ['pxe_ipmitool', 'pxe_ssh', 'pxe_drac', 'pxe_ilo', 'pxe_wol'] log_opt_values /usr/lib/python2.7/s
ite-packages/oslo_config/cfg.py:2229

No error in the log, all driver seems to be loaded successfully.

Comment 18 Imre Farkas 2016-09-06 14:47:46 UTC
I am still not able to reproduce the issue. Can you please prepare an environment for me to debug?

Comment 19 Dmitry Tantsur 2016-09-06 15:24:00 UTC
Hello!

We've fixed the dependencies, so now the required package get pulled. Please feel free to reopen if you still have problems with it after updating to the latest packages.

Comment 20 Matt Young 2016-11-02 22:01:34 UTC
We're hitting this in CI as of today with:

https://trunk.rdoproject.org/centos7-newton/ef/c1/efc1fbe18783e9d24e46b99edb3282c87eb85244_96121e15

Could you please provide details on which specific packages are needed?  We have observed that restarting the openstack-ironic-conductor service allows this to succeed.

This is currently blocking CI / automation.

Comment 21 Dmitry Tantsur 2016-11-03 09:42:56 UTC
Hello,

I can't detect your problem by looking at repositories only :/ could you please provide at least ironic-conductor logs?

Comment 28 Justin Kilpatrick 2016-11-03 12:33:24 UTC
(In reply to Dmitry Tantsur from comment #21)
> Hello,
> 
> I can't detect your problem by looking at repositories only :/ could you
> please provide at least ironic-conductor logs?

I'm seeing the same issue on Newton images, here are my logs. 



https://paste.fedoraproject.org/469489/47817627/ < - Ironic Conductor 
https://paste.fedoraproject.org/469490/76344147  < - Ironic api

Comment 29 Dmitry Tantsur 2016-11-03 13:33:35 UTC
The same issue means "conductor crashed due to missing dracclient". This is not your case, please cooperate with other folks to file a separate bug. This one is fixed and won't be reopened even though you have vaguely similar symptoms.

Comment 30 Matt Young 2016-11-03 13:51:35 UTC
Dmitry: thanks for the feedback.  I'll open a new bug for what we're observing presently.

For those of us "new to the party" (and by "us" I mean "me" :) )could you please provide a link to the change referred to in comment #19 that resolved the similar in appearance (yet different root cause) issue? 

@Chris Jones: I'll update this bug with a link to the new one.

Comment 32 Matt Young 2016-11-03 15:44:37 UTC
Opened https://bugzilla.redhat.com/show_bug.cgi?id=1391602 to track t his per guidance.