Bug 1268992

Summary: [RDO-Manager][Liberty] : openstack baremetal introspection bulk start causes "Internal server error" ( introspection fails) .
Product: [Community] RDO Reporter: Omri Hochman <ohochman>
Component: rdo-managerAssignee: John Trowbridge <jtrowbri>
Status: CLOSED CURRENTRELEASE QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: medium    
Version: trunkCC: goneri, jtrowbri, mburns, sasha, srevivo
Target Milestone: ---   
Target Release: Liberty   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: instack-undercloud-2.1.3-dev224 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Omri Hochman 2015-10-05 20:46:51 UTC 
[RDO-Manager][Liberty] :  openstack baremetal introspection bulk start causes "Internal server error"  ( introspection fails) .

Environment:
-------------
instack-0.0.8-dev4.el7.centos.noarch
instack-undercloud-2.1.3-dev219.el7.centos.noarch
openstack-tripleo-puppet-elements-0.0.2-dev13.el7.centos.noarch
openstack-tripleo-0.0.1-dev1769.el7.centos.noarch
openstack-tripleo-heat-templates-0.8.7-dev249.el7.centos.noarch
openstack-tripleo-image-elements-0.9.7-dev30.el7.centos.noarch


Steps: 
------
(1) Follow the guide to install liberty rdo-manager http://docs.openstack.org/developer/tripleo-docs/environments/virtual.html 

(2) Attempt openstack baremetal introspection bulk start 

Results: 
---------
openstack baremetal introspection bulk start
Setting available nodes to manageable...
Starting introspection of node: 52552cc6-4adb-4e57-a52d-469f0ac23cae
Internal server error


Workaround: 
------------
Execute this command for each node ( in "ironic node-list") :  
ironic node-set-provision-state fc3c15be-0744-4b52-aad8-d4cc717b23cd provide
ironic node-set-provision-state e8d20300-4257-4e45-9163-43679b264ac6 provide
ironic node-set-provision-state 4e61ec9a-f5dd-4b45-a043-99dafaf3cf8d provide
Comment 1 Alexander Chuzhoy 2015-10-05 21:41:51 UTC 
The workaround in comment #0 allows the overcloud deployment to use the nodes without introspection.

The allow introspection:
1. edit the /etc/ironic-inspector/inspector.conf file
and make sure to add another '/' character after sqlite under [database], like below:
[database]
connection = sqlite:////var/lib/ironic-inspector/inspector.sqlite

2. sudo systemctl restart openstack-ironic-inspector.service

3. sudo ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade
Comment 2 John Trowbridge 2015-10-06 10:56:29 UTC 
Fixed in latest liberty repo:

http://trunk.rdoproject.org/liberty/centos7/current/
Comment 3 John Trowbridge 2015-10-07 16:11:05 UTC 
This was close but not quite right. Since we switched to using a service user and rootwrap for ironic-inspector, we need to create the database as that user.

https://review.openstack.org/#/c/231656/ is the fix

The workaround is to run:

sudo chown ironic-inspector /var/lib/ironic-inspector/inspector.sqlite
Comment 4 Omri Hochman 2015-10-27 14:32:06 UTC 
Verified with the RDO GA bits :

sudo yum install -y http://rdoproject.org/repos/openstack-liberty/rdo-release-liberty.rpm