Hide Forgot
Description of problem: Cloudforms's fog-openstack ruby gem randomly selects a tenant when allocating a floating IP instead of using the tenant specified during connection setup. Version-Release number of selected component (if applicable): CFME 5.6.1.2.20160810181333_8ba817b fog-core (1.42.0) fog-google (0.3.2) fog-json (1.0.2) fog-openstack (0.1.11) fog-xml (0.1.2) How reproducible: 100% Steps to Reproduce: 1. Create two OpenStack tenants 2. In each tenant, create an instance WITHOUT assigning a floating ip 3. On the cloudforms appliance, run the script attached below to create a floating IP and allocate it to the instance, once for each tenant&instance Actual results: While one invocation will succeed, the other invocation will fail with a log output such as: [root@cloudforms ~]# ./test2.rb http://192.168.101.81:5000/v3/auth/tokens 59fef26e2edb49a381560f53651a494e admin xxxxx demo-infra e66f50e2-4fc4-4b8e-8058-25f7eb264d60 I, [2016-11-15T22:02:07.139591 #44641] INFO -- : Allocated <Fog::Compute::OpenStack::Address id="33ef8841-404b-4070-ab70-e8a713da72a2", ip="10.32.105.136", pool="guests", fixed_ip=nil, instance_id=nil > /opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/middlewares/expects.rb:6:in `response_call': Expected([200, 202]) <=> Actual(400 Bad Request) (Excon::Error::BadRequest) excon.error.response :body => "{\"badRequest\": {\"message\": \"Unable to associate floating IP 10.32.105.136 to fixed IP 172.20.0.7 for instance e66f50e2-4fc4-4b8e-8058-25f7eb264d60. Error: Bad floatingip request: Port 81c66147-9015-4872-a561-3ebbd818e73e is associated with a different tenant than Floating IP 33ef8841-404b-4070-ab70-e8a713da72a2 and therefore cannot be bound..\\nNeutron server returns request_ids: ['req-d8741196-38f5-4826-80e0-eace85e43ce5']\", \"code\": 400}}" :cookies => [ ] :headers => { "Content-Length" => "442" "Content-Type" => "application/json; charset=UTF-8" "Date" => "Tue, 15 Nov 2016 21:02:09 GMT" "Vary" => "X-OpenStack-Nova-API-Version" "X-Compute-Request-Id" => "req-af949c79-84fc-470b-a80d-23e6b4b8f5e9" "X-Openstack-Nova-Api-Version" => "2.1" } :host => "192.168.101.81" :local_address => "192.168.101.17" :local_port => 47740 :path => "/v2.1/a9374b0dde9c4b14b0b4661cb3402e27/servers/e66f50e2-4fc4-4b8e-8058-25f7eb264d60/action.json" :port => 8774 :reason_phrase => "Bad Request" :remote_ip => "192.168.101.81" :status => 400 :status_line => "HTTP/1.1 400 Bad Request\r\n" from /opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/middlewares/response_parser.rb:8:in `response_call' from /opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/connection.rb:389:in `response' from /opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/connection.rb:253:in `request' from /opt/rh/cfme-gemset/gems/fog-core-1.42.0/lib/fog/core/connection.rb:81:in `request' from /opt/rh/cfme-gemset/gems/fog-openstack-0.1.11/lib/fog/openstack/core.rb:81:in `request' from /opt/rh/cfme-gemset/gems/fog-openstack-0.1.11/lib/fog/compute/openstack/requests/server_action.rb:6:in `server_action' from /opt/rh/cfme-gemset/gems/fog-openstack-0.1.11/lib/fog/compute/openstack/requests/associate_address.rb:7:in `associate_address' from ./test2.rb:31:in `<main>' [root@cloudforms ~]# Expected results: Both invocations should succeed with a log output such as: [root@cloudforms ~]# ./test2.rb http://192.168.101.81:5000/v3/auth/tokens 59fef26e2edb49a381560f53651a494e admin xxxx demo-vms 9061d5f7-71db-4254-8e84-587ff0cdac13 I, [2016-11-15T22:17:46.146016 #45792] INFO -- : Allocated <Fog::Compute::OpenStack::Address id="09569946-6c41-4294-8477-da3ffd8fed0d", ip="10.32.105.137", pool="guests", fixed_ip=nil, instance_id=nil > I, [2016-11-15T22:17:47.125436 #45792] INFO -- : Associate: Response: #<Excon::Response:0x00000001b184c8 @data={:body=>"", :cookies=>[], :host=>"192.168.101.81", :headers=>{"Content-Type"=>"text/html; charset=UTF-8", "Content-Length"=>"0", "X-Openstack-Nova-Api-Version"=>"2.1", "Vary"=>"X-OpenStack-Nova-API-Version", "X-Compute-Request-Id"=>"req-c14c7c38-e899-4260-9b5a-181536b1e3b8", "Date"=>"Tue, 15 Nov 2016 21:17:49 GMT"}, :path=>"/v2.1/a9374b0dde9c4b14b0b4661cb3402e27/servers/9061d5f7-71db-4254-8e84-587ff0cdac13/action.json", :port=>8774, :status=>202, :status_line=>"HTTP/1.1 202 Accepted\r\n", :reason_phrase=>"Accepted", :remote_ip=>"192.168.101.81", :local_port=>50964, :local_address=>"192.168.101.17"}, @body="", @headers={"Content-Type"=>"text/html; charset=UTF-8", "Content-Length"=>"0", "X-Openstack-Nova-Api-Version"=>"2.1", "Vary"=>"X-OpenStack-Nova-API-Version", "X-Compute-Request-Id"=>"req-c14c7c38-e899-4260-9b5a-181536b1e3b8", "Date"=>"Tue, 15 Nov 2016 21:17:49 GMT"}, @status=202, @remote_ip="192.168.101.81", @local_port=50964, @local_address="192.168.101.17"> Additional info: When tracing the network traffic via Wireshark you can see that the interaction works as follows: 1) Fog authenticates the user using username+pw and receives a token and user ID (POST /v3/auth/tokens) 2) Fog retrieves the list of tenants available to the user (GET /v3/users/a12b9e2ec11b7fee4f21d0f3b3a4c643c531fa8d0625311b04af0228c833adf0/projects) 3) Fog requests a token for the FIRST project retrieved in step 2 (instead of the tenant/project supplied during connection creation): POST /v3/auth/tokens HTTP/1.1 User-Agent: fog-core/1.42.0 Content-Type: application/json Host: 192.168.101.81:5000 Content-Length: 155 {"auth":{"identity":{"methods":["token"],"token":{"id":"bd63498f437149909ce69a3640c61446"}},"scope":{"project":{"id":"44322c412882479f86fcf45b4d5b8193"}}}} --- BEGIN TEST SCRIPT --- #!/usr/bin/env ruby require 'rubygems' require 'fog/openstack' require 'logger' auth_url = ARGV[0] domain = ARGV[1] username = ARGV[2] password = ARGV[3] tenant = ARGV[4] instance_uuid = ARGV[5] logger = Logger.new(STDOUT) logger.level = Logger::INFO credentials={ :provider => "OpenStack", :openstack_auth_url => auth_url, :openstack_domain_id => domain, :openstack_username => username, :openstack_api_key => password, :openstack_tenant => tenant } conn = Fog::Compute.new(credentials) #address = conn.allocate_address("guests").body #logger.info("Allocated #{address['floating_ip'].inspect}") #res = conn.associate_address(instance_uuid, "#{address['floating_ip']['ip']}") address = conn.addresses.create pool: "guests" logger.info("Allocated #{address.inspect}") res = conn.associate_address(instance_uuid, "#{address.ip}") logger.info("Associate: Response: #{res.inspect}") --- END TEST SCRIPT ---
While this could be seen from Automate, the bug is in the provider interactions.
Petr, would your fix in https://github.com/ManageIQ/manageiq/pull/12486 also apply here?
Please assess the impact of this issue and update the severity accordingly. Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity for a reminder on each severity's definition.
setting prio to medium since there is a 50% chance of things working when two tenants are present (33% for 3 tenants, etc...).
Please assess the importance of this issue and update the priority accordingly. Somewhere it was missed in the bug triage process. Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#priority for a reminder on each priority's definition. If it's something like a tracker bug where it doesn't matter, please set it to Low/Low.
Reseting needinfo flag back to Petr.
Bug Closure Dear customer, The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. If you have any concerns about this, please let us know. Thanks and regards!