Bug 805171 - various deltacloud exceptions while working w/ vsphere
various deltacloud exceptions while working w/ vsphere
Status: CLOSED ERRATA
Product: CloudForms Cloud Engine
Classification: Red Hat
Component: deltacloud-core (Show other bugs)
1.0.0
Unspecified Unspecified
unspecified Severity medium
: rc
: ---
Assigned To: Michal Fojtik
Ronelle Landy
: ZStream
Depends On:
Blocks: 827454
  Show dependency treegraph
 
Reported: 2012-03-20 11:55 EDT by wes hayutin
Modified: 2012-12-04 10:00 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
An overloaded VSphere provider caused Conductor to constantly switch between pending and vanished states. This update adds error handling for VSphere internal errors.
Story Points: ---
Clone Of:
: 827454 (view as bug list)
Environment:
Last Closed: 2012-12-04 10:00:09 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description wes hayutin 2012-03-20 11:55:44 EDT
recreate:

user1 launch instances
user2 reboot other instances..

notice instances jumping from pending to vanished to pending to vanished to running etc..etc..


usr/bin/deltacloudd:5
thin server (localhost:3002) [deltacloud-mock][11354]: 127.0.0.1 - - [20/Mar/2012 11:50:01] "GET /api/instances/wesScenario1-rhel-x86-64-server-6-cf-configserver HTTP/1.1" 500 7010 0.1828
thin server (localhost:3002) [deltacloud-mock][11354]: [NO HANDLED] Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //soapenv:Body/*
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `evaluate'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `xpath'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `map'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `xpath'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/trivial_soap.rb:91:in `request'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/connection.rb:67:in `call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:182:in `_call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:70:in `RetrieveServiceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:53:in `serviceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:35:in `connect'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `tap'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `connect'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:389:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:388:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:142:in `instances'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/base_driver.rb:172:in `instance'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `send'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `show'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:92:in `show'
/usr/share/deltacloud-core/lib/deltacloud/server.rb:470
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `instance_eval'
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `GET /api/instances/:id'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `compile!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_syslog.rb:48:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_date.rb:31:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_accept.rb:149:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_driver_select.rb:45:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_matrix_params.rb:106:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_runtime.rb:36:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_etag.rb:41:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:235
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
thin server (localhost:3002) [deltacloud-mock][11354]: 127.0.0.1 - - [20/Mar/2012 11:50:01] "GET /api/instances/rhel-x86-64-server-6-cf-configserver-rhel-x86-64-server-6-cf-configserver HTTP/1.1" 500 7034 0.1833
thin server (localhost:3002) [deltacloud-mock][11354]: [NO HANDLED] Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //soapenv:Body/*
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `evaluate'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `xpath'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `map'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `xpath'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/trivial_soap.rb:91:in `request'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/connection.rb:67:in `call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:182:in `_call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:70:in `RetrieveServiceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:53:in `serviceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:35:in `connect'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `tap'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `connect'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:389:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:388:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:142:in `instances'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/base_driver.rb:172:in `instance'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `send'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `show'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:92:in `show'
/usr/share/deltacloud-core/lib/deltacloud/server.rb:470
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `instance_eval'
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `GET /api/instances/:id'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `compile!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_syslog.rb:48:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_date.rb:31:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_accept.rb:149:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_driver_select.rb:45:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_matrix_params.rb:106:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_runtime.rb:36:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_etag.rb:41:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:235
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
thin server (localhost:3002) [deltacloud-mock][11354]: 127.0.0.1 - - [20/Mar/2012 11:50:02] "GET /api/instances/wesRegister02-rhel-x86-64-server-6-cf-configserver HTTP/1.1" 500 7011 0.2742
thin server (localhost:3002) [deltacloud-mock][11354]: [NO HANDLED] Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //soapenv:Body/*
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `evaluate'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `xpath'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `map'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `xpath'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/trivial_soap.rb:91:in `request'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/connection.rb:67:in `call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:182:in `_call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:70:in `RetrieveServiceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:53:in `serviceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:35:in `connect'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `tap'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `connect'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:389:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:388:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_client.rb:30:in `find_vm'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:147:in `instances'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:144:in `instances'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/base_driver.rb:172:in `instance'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `send'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `show'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:92:in `show'
/usr/share/deltacloud-core/lib/deltacloud/server.rb:470
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `instance_eval'
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `GET /api/instances/:id'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `compile!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_syslog.rb:48:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_date.rb:31:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_accept.rb:149:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_driver_select.rb:45:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_matrix_params.rb:106:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_runtime.rb:36:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_etag.rb:41:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:235
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
thin server (localhost:3002) [deltacloud-mock][11354]: [NO HANDLED] Deltacloud::ExceptionHandler::BackendError: Unhandled exception or status code (Undefined namespace prefix: //soapenv:Body/*)
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `evaluate'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:159:in `xpath'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `map'
/usr/lib/ruby/gems/1.8/gems/nokogiri-1.5.0.beta.4/lib/nokogiri/xml/node.rb:150:in `xpath'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/trivial_soap.rb:91:in `request'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/connection.rb:67:in `call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:182:in `_call'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/basic_types.rb:70:in `RetrieveServiceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:53:in `serviceContent'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:35:in `connect'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `tap'
/usr/lib/ruby/gems/1.8/gems/rbvmomi-1.2.3/lib/rbvmomi/vim.rb:34:in `connect'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:389:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:388:in `new_client'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_client.rb:30:in `find_vm'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:147:in `instances'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/vsphere/vsphere_driver.rb:144:in `instances'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/base_driver.rb:172:in `instance'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `send'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `show'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:92:in `show'
/usr/share/deltacloud-core/lib/deltacloud/server.rb:470
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `instance_eval'
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `GET /api/instances/:id'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `compile!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_syslog.rb:48:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_date.rb:31:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_accept.rb:149:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_driver_select.rb:45:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_matrix_params.rb:106:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_runtime.rb:36:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_etag.rb:41:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:235
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
thin server (localhost:3002) [deltacloud-mock][11354]: 127.0.0.1 - - [20/Mar/2012 11:50:02] "GET /api/instances/wesKatelloRegister-rhel-x86-64-server-6-cf-configserver HTTP/1.1" 500 7422 0.6201
I, [2012-03-20T11:51:10.794129 #11354]  INFO -- : New Aws::Ec2 using per_thread-connection mode
I, [2012-03-20T11:51:10.795408 #11354]  INFO -- : New Aws::Ec2 using per_thread-connection mode
I, [2012-03-20T11:51:10.796587 #11354]  INFO -- : Closing HTTPS connection to ec2.us-east-1.amazonaws.com:443
W, [2012-03-20T11:51:10.798001 #11354]  WARN -- : Rightscale::HttpConnection : request failure count: 1, exception: #<Errno::EPIPE: Broken pipe>
I, [2012-03-20T11:51:10.798728 #11354]  INFO -- : Opening new HTTPS connection to ec2.us-west-1.amazonaws.com:443
I, [2012-03-20T11:51:10.801812 #11354]  INFO -- : Opening new HTTPS connection to ec2.us-east-1.amazonaws.com:443
[root@qeblade31 ~]#
Comment 1 wes hayutin 2012-03-20 11:56:26 EDT
[root@qeblade31 ~]# rpm -qa | grep deltacloud
deltacloud-core-ec2-0.5.0-5.el6.noarch
deltacloud-core-0.5.0-5.el6.noarch
rubygem-deltacloud-client-0.5.0-2.el6.noarch
deltacloud-core-vsphere-0.5.0-5.el6.noarch
deltacloud-core-rhevm-0.5.0-5.el6.noarch
Comment 2 Michal Fojtik 2012-03-21 06:00:00 EDT
Hi Wes,

Those exceptions are normally returned when the VSphere is overloaded or misconfigured. I would recommend to reboot the VSphere machine. We can't do anything better than 'translating' this exception to something more human-readable, since this appears like backend error.
Comment 3 jrd 2012-03-28 10:29:52 EDT
If this isn't fixable by dev, should it be on-qa?
Comment 4 Ronelle Landy 2012-03-28 11:15:19 EDT
Spoke with Wes regarding what is needed to resolve this issue. The problem is two-fold:

Firstly - as Michal points out in comment 2 above - we see this error sporadically when vsphere is overloaded. See JIRA https://issues.apache.org/jira/browse/DTACLOUD-150 for a report of a similar error when the vsphere provider is invalid or otherwise unavailable. Deltacloud needs better error handling and reporting. Moving this BZ back to ASSIGNED (dev) to address the error handling and reporting issue in Deltacloud. As both this BZ and the JIRA show, the error is 'Unhandled'. DTACLOUD-150 is in 'reopened' status.  

Secondly, there is an issue of conductor retrying the request if the vsphere provider is busy/unavailable. Copying chat comment:

<weshay> rlandy, actually I was hoping for a conductor fix.. to limit the number of times the request is submitted

Wes will clone this BZ to request a conductor fix for this second part.
Comment 5 Hugh Brock 2012-03-28 13:19:39 EDT
Not clear how this got blocker+, it isn't. MOving to 1.0.z/1.1.0
Comment 7 Michal Fojtik 2012-06-01 05:42:43 EDT
Fix pushed to master repo and also to brew:

commit b922f47472fdb35db2cd36be58bbfd35412515fb
Author: Michal Fojtik <mfojtik@redhat.com>
Date:   Fri Jun 1 08:48:29 2012 +0200

VSphere: Added error handling for the Undefined namespace prefix error (BZ: #805171)
Comment 12 Jan Provaznik 2012-06-13 11:09:00 EDT
From above, it seems that "Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //soapenv:Body/*" exception is raised both when vsphere server is overloaded or is unreachable.

This not descriptive message is the displayed in UI if an instance launch fails -> we might catch the error and display something more descriptive.

Switching between vanished/pending states is not a bug, but if this overload is quite common we might add extend dbomatic to swtich an instance into VANISHED state only after X fails.
Comment 14 Ronelle Landy 2012-09-27 16:19:33 EDT
Tested rpms:

>> rpm -qa |grep aeolus
aeolus-configure-2.8.7-1.el6cf.noarch
rubygem-aeolus-image-0.3.0-12.el6.noarch
rubygem-aeolus-cli-0.7.2-1.el6cf.noarch
aeolus-conductor-0.13.14-1.el6cf.noarch
aeolus-conductor-daemons-0.13.14-1.el6cf.noarch
aeolus-conductor-doc-0.13.14-1.el6cf.noarch
aeolus-all-0.13.14-1.el6cf.noarch

>> rpm -qa |grep deltacloud
rubygem-deltacloud-client-0.5.0-2.el6.noarch
deltacloud-core-vsphere-0.5.0-10.el6_2.noarch
deltacloud-core-rhevm-0.5.0-10.el6_2.noarch
deltacloud-core-ec2-0.5.0-10.el6_2.noarch
deltacloud-core-0.5.0-10.el6_2.noarch

Launching an instance to a Vsphere provider that is currently unavailable does result in the instance status being shown as 'Vanished'. 

***********

 Alerts 1

    ce-2h6kn/testVsphere
        Instance Failure
        vanished


***********

See the attached screenshot.

Current versions of Deltacloud (Deltacloud 1.0.3) return a 504 error if a user tries to access a Vsphere provider that is unavailable. Copying from the logs:

10.11.9.168 - - [27/Sep/2012 16:02:17] "GET /api/images?format=xml HTTP/1.1" 504 46057 60.0481


What is interesting is that looking at the Application History, the output shows:

**************

    27-Sep-2012 19:51:26: Instance ce-gje7a/AutoImageImport-DoNotDelete created
    27-Sep-2012 19:51:26: State changed to pending
    27-Sep-2012 19:51:27: Attempting to launch this deployment on provider account vsphere-default_Administrator
    27-Sep-2012 19:51:32: Instance ce-gje7a/AutoImageImport-DoNotDelete ce-gje7a/AutoImageImport-DoNotDelete: 500 : Unhandled exception or status code (No route to host - connect(2))

**************

which picks up an unhandled exception - although with a correct error message. It is true that there is no route to the host.

Considering that the error message is now correct, that the unhandled exception no longer shows up in an newer version of Deltacloud and that the application itself is shown as 'Vanished', going to mark this as verified and touch base with development wrt the exception in the History.
Comment 16 errata-xmlrpc 2012-12-04 10:00:09 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-1516.html

Note You need to log in before you can comment on or make changes to this bug.