Bug 1401326

Summary: Can't succesfully introspect 7 IPMI nodes with RHOSP 10
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: python-virtualbmcAssignee: Lucas Alvares Gomes <lmartins>
Status: CLOSED CURRENTRELEASE QA Contact: mlammon
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: bfournie, dhill, dtantsur, hbrock, jjoyce, jslagle, lmartins, mburns, mcornea, rhel-osp-director-maint, srevivo
Target Milestone: z4Keywords: TestOnly, Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-virtualbmc-1.0.0-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-12 15:09:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
ironic logs none

Description David Hill 2016-12-04 20:03:25 UTC 
Description of problem:
Can't successfully introspect 7 nodes with RHOSP 10 but don't have those issues with previous RHOSP versions.


2016-12-04 13:13:33.597 18848 ERROR ironic.drivers.modules.ipmitool [-] IPMI power on timed out after 4 retries on node 94793c31-1d69-4a08-b8a7-43eed25aa95e.
2016-12-04 13:13:33.956 18848 ERROR ironic.drivers.modules.ipmitool [-] IPMI power on timed out after 4 retries on node 387b477e-8f3d-486c-a730-0802872c8e4c.
2016-12-04 13:13:34.265 18848 ERROR ironic.drivers.modules.ipmitool [-] IPMI power on timed out after 4 retries on node 026c0cd5-e401-4415-a60b-8634d1b13c48.
2016-12-04 13:13:34.562 18848 ERROR ironic.drivers.modules.ipmitool [-] IPMI power on timed out after 4 retries on node 303055f9-48fb-4c04-bf65-bdfa9581ccb6.


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Deploy undercloud
2. Intropsect 7 nodes

Actual results:
Fails

Expected results:
Succeeds

Additional info:
Always fails on 4 nodes after 4 retries each.  Perhaps the retries are done too fast and perhaps 4 retry is not enough
Comment 1 David Hill 2016-12-07 19:13:42 UTC 
Created attachment 1229208 [details]
ironic logs
Comment 4 Lucas Alvares Gomes 2016-12-12 12:32:08 UTC 
Following the upstream bug ticket: https://bugs.launchpad.net/virtualbmc/+bug/1647229

I've proposed a fix for the VirtualBMC project: https://review.openstack.org/#/c/409755/
Comment 5 Dmitry Tantsur 2017-10-20 11:20:17 UTC 
Folks, I'm moving this to Ocata, as virtualbmc was not shipped with OSP 10/Newton.
Comment 6 Jon Schlueter 2017-11-15 01:59:13 UTC 
According to our records, this should be resolved by python-virtualbmc-1.0.0-3.el7ost.  This build is available now.
Comment 8 mlammon 2017-12-11 20:03:33 UTC 
This can be marked verified as I no longer see the issue with osp versions 11 or 12

2017-12-06.2[stack@undercloud-0 ~]$ rpm -qa | grep python-virt
python-virtualbmc-1.0.0-3.el7ost.noarch
[stack@undercloud-0 ~]$ vbmc list
+--------------+---------+-------------------+------+
| Domain name  |  Status |      Address      | Port |
+--------------+---------+-------------------+------+
|    ceph-0    | running | ::ffff:172.16.0.2 | 6239 |
|    ceph-1    | running | ::ffff:172.16.0.2 | 6238 |
|    ceph-2    | running | ::ffff:172.16.0.2 | 6237 |
|  compute-0   | running | ::ffff:172.16.0.2 | 6232 |
|  compute-1   | running | ::ffff:172.16.0.2 | 6236 |
| controller-0 | running | ::ffff:172.16.0.2 | 6233 |
| controller-1 | running | ::ffff:172.16.0.2 | 6231 |
| controller-2 | running | ::ffff:172.16.0.2 | 6230 |
|   ironic-0   | running | ::ffff:172.16.0.2 | 6235 |
|   ironic-1   | running | ::ffff:172.16.0.2 | 6234 |
+--------------+---------+-------------------+------+