Bug 1375639

Hi Jerome,

I'm not sure I fully understand what you're trying to do.  I don't believe we collect the Ansible Tower Host variables.  Are you connecting to the provider object, parsing these variables and trying to call Ansible Tower with this IP address?

I am trying to run an Ansible Template Job on a AWS instance.

1. Setup CloudForms with AWS provider
2. Setup Ansible Tower with the new CloudForms inventory
3. Create a template job for this inventory
4. Run the template job on a AWS instance on the inventory

Issue: 
The AWS instance in the inventory (coming from CloudForms) has 'ansible_ssh_host' variable set to the internal AWS IP address. This result in Ansible Tower not being able to reach out to the AWS instance to run the playbook.

Jerome, has the aws instance a public ip associated with it?

Hi Marcel, yes, the aws instance has a public IP. However, this public IP is not seen in CloudForms on the instance screen.

I just checked with the latest master upstream version and I could see both, the internal and external ip for an instance. 
There is already a BZ for this https://bugzilla.redhat.com/show_bug.cgi?id=1376516

Could you check with a nightly build if the original ansible_ssh_host problem still persists with a more recent build?

I have applied the PR [1] in my CF4.1 environment (changes to /var/www/miq/vmdb/gems/manageiq-providers-amazon/app/models/manageiq/providers/amazon/network_manager/refresh_parser.rb).

I can confirm that I now see both IP addresses showing up in the instance summary view in CloudForms.

In Ansible Tower, after a refresh of the inventory, I can see both addresses in "ipaddresses" variable: ["172.31.27.175", "52.35.146.97"] but ansible_ssh_host" is still set to the local IP: "172.31.27.175".

Full variable list on AWS instance:
{"ansible_ssh_host": "172.31.27.175", "cloudforms": {"last_perf_capture_on": "2016-09-21T22:09:00Z", "retired": false, "availability_zone_id": 10000000000005, "previous_state": "stopped", "created_on": "2016-09-16T18:51:50Z", "href": "https://10.9.62.88/api/vms/10000000000950", "guid": "a043ab28-7c3e-11e6-872f-005056b1e4d4", "id": 10000000000950, "cloud": true, "retirement_warn": 7, "state_changed_on": "2016-09-21T22:02:10Z", "ansible_ssh_host": "172.31.27.175", "raw_power_state": "running", "location": "ec2-52-35-146-97.us-west-2.compute.amazonaws.com", "power_state": "on", "type": "ManageIQ::Providers::Amazon::CloudManager::Vm", "uid_ems": "i-05091373001555722", "vendor": "amazon", "template": false, "tags": [{"href": "https://10.9.62.88/api/vms/10000000000950/tags/10000000000023", "id": 10000000000023, "name": "/managed/environment/dev"}, {"href": "https://10.9.62.88/api/vms/10000000000950/tags/10000000000185", "id": 10000000000185, "name": "/managed/function/app"}], "boot_time": "2016-09-21T21:55:01Z", "ems_id": 10000000000008, "name": "bz1375639", "miq_group_id": 10000000000002, "tenant_id": 10000000000001, "evm_owner_id": 10000000000001, "flavor_id": 10000000000010, "ipaddresses": ["172.31.27.175", "52.35.146.97"], "ems_ref": "i-05091373001555722", "updated_on": "2016-09-21T22:16:36Z", "retires_on": "2016-10-16"}}

How does the inventory script in Ansible Tower determine which IP to set ansible_ssh_host to? Can this be changed?

[1] ManageIQ - Fix collection of public ip addresses #36
https://github.com/ManageIQ/manageiq-providers-amazon/pull/36/files

I'm seen the same issue on on Openstack provider. ansible_ssh_host is set to the tenant IP not the floating for the instance, and I was able to work around that by updating cloudfrom plugin 
/var/lib/awx/venv/tower/lib/python2.7/site-packages/awx/plugins/inventory/cloudforms.py 

Change to 
host['ansible_ssh_host'] = host['ipaddresses'][1]
I think we need way for cloudfrom to change ansible_ssh_host to the floating IP for openstack instance, and the public IP for AWS instances.

After further investigation on the Ansible Tower side, it appears that the CloudForms inventory script set ansible_ssh_host to the first available ip address:

   # Set ansible_ssh_host to the first available ip address
   host['ansible_ssh_host'] = host['ipaddresses'][0]

in /var/lib/awx/venv/tower/lib/python2.7/site-packages/awx/plugins/inventory/cloudforms.py

Clearly this is a limitation and will not work for all providers (especially public clouds).

As a workaround (hack) in my environment, I have modified the Ansible Tower inventory script to check for the host type and set ansible_ssh_host to the second ip returned by CloudForms:

   # Workaround for BZ 1375639
   if host['type'] is "ManageIQ::Providers::Amazon::CloudManager::Vm":
       host['ansible_ssh_host'] = host['ipaddresses'][1]

ERRATA: Based on Comment 8 & 9, workaround (hack) is:

   # Workaround for BZ 1375639
   if len(host['ipaddresses']) > 1 and (host['type'] == "ManageIQ::Providers::Amazon::CloudManager::Vm" or host['type'] == "ManageIQ::Providers::Openstack::CloudManager::Vm"):
       host['ansible_ssh_host'] = host['ipaddresses'][1]

in /var/lib/awx/venv/tower/lib/python2.7/site-packages/awx/plugins/inventory/cloudforms.py

Jerome, Saif, I guess the python script is part of ansible tower, right?
If thats the case, can you open an issue on their side and close this one?

Both teams need to work on this together. How will Ansible team determine which IP address to choose to set ansible_ssh_host variable to? Should CloudForms provide this information in the results returned from its API? 

From what I can see, there is no clear way to determine which IP address is public in the API results coming from CloudForms.

Jerome, you are absolutely right. But I have no idea how ansible is consuming our API. Via REST? What calls do they make?

We can use this BZ as a touch point, in case you would like to.

In more advanced cases, not even ManageIQ can know, which IP address is public or has port 22 opened, in general, it has to be accessible to the Tower.

What we should return in the API the is the list of all the interfaces, listing what subnets they connect to and what public/floating ips they have associated.

Then you can pick the IP on the ansible side, using your knowledge https://github.com/atyronesmith/OPNFV-demo-ansible/blob/master/roles/cloudrouter-clients/tasks/main.yml#L5

My knowledge in this ^ case is, that net_mgmt subnet is accessible to Tower and the port 22 is allowed there.

The format I would like to have in MIQ API for a VM would be:
    CloudExternal:
      fixed_ips:
        - 10.3.0.36
      floating_ips: []
      interface: eth0
    VL1:
      fixed_ips:
        - 10.20.0.3
      floating_ips: []
      interface: eth1
    VL2:
      fixed_ips:
        - 10.4.0.3
      floating_ips: []
      interface: eth2
    net_mgmt:
      fixed_ips:
        - 10.0.13.44
      floating_ips: []
      interface: eth3

But we should also have just a list of all fixed_ips and floating_ips(public_ips), for the simple cases. Then the host['floating_ips'][0] should just work in normal cases.

https://github.com/ManageIQ/manageiq/pull/11491

should be enough for this BZ. I'll do another PR, that will expose a detailed list I describe #14 as another attribute or subcollection.

After this PR, we will need an Ansible Tower BZ for changing:
https://github.com/ansible/ansible/blob/cf4d436e07009e91e74664789953a8a5d6b2d823/contrib/inventory/cloudforms.py#L264

To:
ret = self._get_json("%s/api/vms?offset=%s&limit=%s&expand=resources,tags,hosts,&attributes=ipaddresses,floating_ip_addresses,fixed_ip_addresses" % (self.cloudforms_url, offset, limit))


And then the logic for ansible_ssh_host, needs to check if there is host['floating_ip_addresses'][0] and use that, otherwise use host['ipaddresses'][0]

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/cd3535816e2509760a4598256f39c99c89333e67

commit cd3535816e2509760a4598256f39c99c89333e67
Author:     Ladislav Smola <lsmola>
AuthorDate: Fri Sep 23 17:58:34 2016 +0200
Commit:     Ladislav Smola <lsmola>
CommitDate: Tue Oct 18 14:49:46 2016 +0200

    Allow Vm to show floating and fixed ip addresses
    
    Allow Vm to show floating and fixed ip addresses, since it is
    a virtual column, it's also usable in the API.
    
    Fixes:
    https://bugzilla.redhat.com/show_bug.cgi?id=1375639

 app/models/manageiq/providers/cloud_manager/vm.rb | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/9b3a00114dabb9b979a675f85d59ef70dfe0a3d8

commit 9b3a00114dabb9b979a675f85d59ef70dfe0a3d8
Author:     Greg Blomquist <blomquisg>
AuthorDate: Tue Oct 18 13:14:47 2016 -0400
Commit:     Oleg Barenboim <chessbyte>
CommitDate: Tue Oct 18 23:05:11 2016 -0500

    Merge pull request #11491 from Ladas/expose_fixed_and_floating_ips_usable_by_api
    
    Allow Vm to show floating and fixed ip addresses
    (cherry picked from commit 6a6271c3e32dcfb39587fb97253fe311f23a924d)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1375639

 app/models/manageiq/providers/cloud_manager/vm.rb            | 12 +++++++++++-
 .../manageiq/providers/azure/cloud_manager/refresher_spec.rb |  5 +++++
 .../providers/openstack/cloud_manager/refresh_spec_common.rb |  3 +++
 3 files changed, 19 insertions(+), 1 deletion(-)

https://github.com/ManageIQ/manageiq/pull/12119

How about editing ansible_ssh_host in /var/libr/awx/venv/tower/lib/python2.7/site-packages/awx/plugins/inventory/openstack.py like this?

['ansible_ssh_host' ] = server_vars['public_v4']

Especially for cloud providers, as long as Ansible Tower is cannot access the internal cloud tenant networks, this seems to be a workaround.

That would apply if you would fetch inventory directly from OpenStack, in this BZ, the Ansible Tower fetches inventory from Cloudforms.

My workaround is to add a custom property via automate.
The value of the custom property comes from a service dialog (wich is the ip address itself, so it will be ok)

I modified the inventory script to fetch custom properties. If a VM/Instance has the desired custom propery, it loads its value for ansible_ssh_host. If the VM/instance does not have the custom property, then it does what it is currently doing which is use the first ip address.

With this method, i will have no issues with VM/Instances that were provisioned via service dialog. For other VM/Instances, i can set it manually via a python program (wich is the same program called from automate as an external script), so with the next inventory sync it will be ok. 

It is not a perfect solution, but it helps a lot.

Verified in 5.8.0.13-rc2.20170502165848_0f98658. "floating_ip_addresses" and "fixed_ip_addresses" properties are exposed in REST API, however ansible team hasn't changed their code yet and "ansible_ssh_host" variable shows the internal ip.

Has anyone told the Ansible team?  I don't see any GitHub issues (open or closed) referencing "floating_ip_addresses" or "fixed_ip_addresses":

* https://github.com/ansible/ansible/issues?utf8=%E2%9C%93&q=is%3Aissue%20floating_ip_addresses%20

* https://github.com/ansible/ansible/issues?utf8=%E2%9C%93&q=is%3Aissue%20fixed_ip_addresses%20

I've submitted the issue https://github.com/ansible/ansible/issues/24920