Bug 1303321 - ospd-8 poodle 2016-01-29.4 deployment fails w/ Node failed to get image for deploy
ospd-8 poodle 2016-01-29.4 deployment fails w/ Node failed to get image for ...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Lucas Alvares Gomes
Toure Dunnon
: Automation, AutomationBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-30 13:50 EST by wes hayutin
Modified: 2016-02-02 08:51 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-02 08:51:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description wes hayutin 2016-01-30 13:50:02 EST
Description of problem:

ospd-8 poodle build candidate fails to deploy properly.
Error appears to be the following.

an 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception for node 77ad67b6-e7e1-47d7-ab45-78aa2bbcce86: Node failed to get image for deploy. exception: Failed to deploy instance: Failed to start the iSCSI target to deploy the node 77ad67b6-e7e1-47d7-ab45-78aa2bbcce86. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 345, in heartbeat
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task, **kwargs)
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 134, in wrapper
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 873, in continue_deploy
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor     uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 443, in do_agent_iscsi_deploy
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor     raise exception.InstanceDeployFailure(reason=msg)
Jan 30 13:09:51 instack.localdomain ironic-conductor[7259]: 2016-01-30 13:09:51.251 7259 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Failed to deploy instance: Failed to start the iSCSI target to deploy the node 77ad67b6-e7e1-47d7-ab45-78aa2bbcce86. Error: {u'message': u'[Errno 2] No such file or directory', u'code': 500, u'type': u'OSError', u'details': u''}
Comment 2 wes hayutin 2016-01-30 13:56:11 EST
Also see the following error in the ironic-api log

2015-12-08 22:20:10.034 27416 DEBUG keystonemiddleware.auth_token [-] Storing token in cache store /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:170
2015-12-08 22:20:10.035 27416 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 274ec72cac704533a101b1d4d01d8b4f, project_id 66f8d9954f024550a6c6dd192f51e77c, roles admin,_member_ process_request /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:759
2015-12-08 22:20:10.055 27416 CRITICAL ironic [-] IOError: [Errno 32] Broken pipe
2015-12-08 22:20:10.055 27416 ERROR ironic Traceback (most recent call last):
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/bin/ironic-api", line 10, in <module>
2015-12-08 22:20:10.055 27416 ERROR ironic     sys.exit(main())
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib/python2.7/site-packages/ironic/cmd/api.py", line 64, in main
2015-12-08 22:20:10.055 27416 ERROR ironic     wsgi.serve_forever()
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib64/python2.7/SocketServer.py", line 236, in serve_forever
2015-12-08 22:20:10.055 27416 ERROR ironic     poll_interval)
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib64/python2.7/SocketServer.py", line 155, in _eintr_retry
2015-12-08 22:20:10.055 27416 ERROR ironic     return func(*args)
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 83, in select
2015-12-08 22:20:10.055 27416 ERROR ironic     return hub.switch()
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2015-12-08 22:20:10.055 27416 ERROR ironic     return self.greenlet.switch()
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in run
2015-12-08 22:20:10.055 27416 ERROR ironic     self.fire_timers(self.clock())
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
2015-12-08 22:20:10.055 27416 ERROR ironic     self.squelch_timer_exception(timer, sys.exc_info())
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 385, in squelch_timer_exception
2015-12-08 22:20:10.055 27416 ERROR ironic     traceback.print_exception(*exc_info)
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib64/python2.7/traceback.py", line 124, in print_exception
2015-12-08 22:20:10.055 27416 ERROR ironic     _print(file, 'Traceback (most recent call last):')
2015-12-08 22:20:10.055 27416 ERROR ironic   File "/usr/lib64/python2.7/traceback.py", line 13, in _print
2015-12-08 22:20:10.055 27416 ERROR ironic     file.write(str+terminator)
2015-12-08 22:20:10.055 27416 ERROR ironic IOError: [Errno 32] Broken pipe
2015-12-08 22:20:10.055 27416 ERROR ironic
Comment 4 Dmitry Tantsur 2016-02-02 08:20:25 EST
Was not able to reproduce with my images, will try yours.

P.S.
The ironic-api message is worrying, but I don't think it's related.
Comment 5 wes hayutin 2016-02-02 08:51:29 EST
We've found the root cause of the issue to be a bug in the rhos-release rpm.

The invocation of `rhos-release -r 7.2 -t /home/stack/DIB -d -P 8-director`
is not setting the repos in the DIB dir to be poodles, the repos installed in /etc/yum.repos.d/  *are* set to poodles.  This caused the confusion.

Note You need to log in before you can comment on or make changes to this bug.