Bug 1298476 - Deployment failed when using IPv6 templates with HA setup
Deployment failed when using IPv6 templates with HA setup
Status: CLOSED WORKSFORME
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
All Linux
urgent Severity urgent
: y3
: 7.0 (Kilo)
Assigned To: Steven Hardy
Udi Shkalim
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-14 03:48 EST by Udi Shkalim
Modified: 2016-09-17 00:11 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-04 02:52:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ushkalim: needinfo+


Attachments (Terms of Use)
templates dir (1.69 MB, application/x-gzip)
2016-01-14 03:48 EST, Udi Shkalim
no flags Details

  None (edit)
Description Udi Shkalim 2016-01-14 03:48:52 EST
Created attachment 1114720 [details]
templates dir

Description of problem:
RHEL 7.2 
When I'm trying to deploy the overcloud HA 3 controllers and 1 compute and I'm getting the following error:
2016-01-13 16:01:50 [MysqlRootPassword]: CREATE_COMPLETE  state changed
2016-01-13 16:01:50 [PcsdPassword]: CREATE_COMPLETE  state changed
2016-01-13 16:01:50 [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.InternalNetwork: resources.Networks.Property error: resources.InternalApiSubnet.properties.allocation_pools[0].start: "fd00: fd00: fd00: 2000: 0000: 0000: 0000: 10" does not validate ip_addr (constraint not found)
Stack overcloud CREATE_IN_PROGRESS
Stack overcloud CREATE_FAILED
Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 977, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 344, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 276, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.
clean_up DeployOvercloud: Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 112, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 255, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 977, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 344, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 276, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.
 
END return value: 1


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Try to deploy overcloud HA with 3 Controllers + 1 Compute 
2.
3.

Actual results:
Deployment failed

Expected results:
Deployment passed

Additional info:
Deploy command is: openstack overcloud deploy --templates $THT -e $THT/environments/network-isolation-v6.yaml -e $THT/environments/net-single-nic-with-vlans.yaml -e /home/stack/templates/network-environment.yaml --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --libvirt-type qemu --timeout 180 --ntp-server clock.redhat.com

Templates Dir is attached.
Comment 2 Hugh Brock 2016-01-14 04:13:56 EST
Steve, could you have a look at this, it seems like it might be a simple parameter validation thing.
Comment 3 Steven Hardy 2016-01-14 04:45:36 EST
Ok, so from looking at the heat code, it seems we validate this via oslo.netutils:

https://github.com/openstack/oslo.utils/blob/master/oslo_utils/netutils.py#L168

So one thing to confirm is the version if oslo.utils, and check by inspection that netutils.py on the platform has the expected IPv6 functions (don't look like recent additions from the git history so it should).


However, the log above looks suspicious:

resources.InternalApiSubnet.properties.allocation_pools[0].start: "fd00: fd00: fd00: 2000: 0000: 0000: 0000: 10" does not validate ip_addr (constraint not found)

The IPv6 address is split by spaces, which would cause this:

>>> from oslo_utils import netutils
>>> a = "fd00: fd00: fd00: 2000: 0000: 0000: 0000: 10"
>>> netutils.is_valid_ip(a)
False
>>> b = "fd00:fd00:fd00:2000:0000:0000:0000:10"
>>> netutils.is_valid_ip(b)
True

So, we need to figure out where those spaces are coming from, in the template the parameter looks like this, and from the attached tarball, the error appears to be from the defaults vs a user-provided value:

  InternalApiAllocationPools:
    default: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]
    description: Ip allocation pool range for the internal API network.
    type: json
Comment 4 Steven Hardy 2016-01-14 05:19:01 EST
Hi Udi,

I've tried reproducing this with the attached minimal reproducer (just the network/subnet), on upstream master it works fine and I cannot reproduce.

Also, the report is against OSP7/Kilo, but the constraint failing does not exist in kilo heat, which is confusing:

https://github.com/openstack/heat/blob/stable/kilo/heat/engine/resources/openstack/neutron/subnet.py#L132

vs 

https://github.com/openstack/heat/blob/stable/liberty/heat/engine/resources/openstack/neutron/subnet.py#L145

Please confirm, is the undercloud OSP7 (kilo) or OSP8 (liberty)?

Also, please confirm the version of oslo.utils and the version of all heat packages so I can be sure what version you're testing, thanks!
Comment 5 Steven Hardy 2016-01-14 05:20:05 EST
Minimal reproducer:

http://paste.fedoraproject.org/310613/66427145/raw/

Try downloading that, then do:

heat stack-create test -f the_template.yaml

Ensure the overcloud has been deleted before trying this - on master TripleO this works fine for me.
Comment 6 Marius Cornea 2016-01-14 05:33:01 EST
(In reply to Steven Hardy from comment #5)
> Minimal reproducer:
> 
> http://paste.fedoraproject.org/310613/66427145/raw/
> 
> Try downloading that, then do:
> 
> heat stack-create test -f the_template.yaml
> 
> Ensure the overcloud has been deleted before trying this - on master TripleO
> this works fine for me.

It worked ok on a 7.2 director environment:

openstack-heat-engine-2015.1.2-4.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch
openstack-heat-api-2015.1.2-4.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch
openstack-heat-common-2015.1.2-4.el7ost.noarch
openstack-heat-api-cfn-2015.1.2-4.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.2-4.el7ost.noarch


stack@instack:~>>> heat stack-show test
+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| Property              | Value                                                                                                                                |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| capabilities          | []                                                                                                                                   |
| creation_time         | 2016-01-14T10:23:38Z                                                                                                                 |
| description           | Internal API network. Used for most APIs, Database,                                                                                  |
|                       | RPC.                                                                                                                                 |
| disable_rollback      | True                                                                                                                                 |
| id                    | 185ebc16-cf6b-4277-9303-6fd6eb60bfc7                                                                                                 |
| links                 | http://192.0.2.1:8004/v1/bba90a9c3682484ea6acf64484f56334/stacks/test/185ebc16-cf6b-4277-9303-6fd6eb60bfc7 (self)                    |
| notification_topics   | []                                                                                                                                   |
| outputs               | [                                                                                                                                    |
|                       |   {                                                                                                                                  |
|                       |     "output_value": "5f2151b1-0eef-49ff-84c6-96928eded09e",                                                                          |
|                       |     "description": "Neutron internal network",                                                                                       |
|                       |     "output_key": "OS::stack_id"                                                                                                     |
|                       |   }                                                                                                                                  |
|                       | ]                                                                                                                                    |
| parameters            | {                                                                                                                                    |
|                       |   "OS::project_id": "bba90a9c3682484ea6acf64484f56334",                                                                              |
|                       |   "InternalApiSubnetName": "internal_api_subnet",                                                                                    |
|                       |   "OS::stack_id": "185ebc16-cf6b-4277-9303-6fd6eb60bfc7",                                                                            |
|                       |   "OS::stack_name": "test",                                                                                                          |
|                       |   "InternalApiNetCidr": "fd00:fd00:fd00:2000::/64",                                                                                  |
|                       |   "InternalApiAllocationPools": "[{\"start\": \"fd00:fd00:fd00:2000::10\", \"end\": \"fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe\"}]",  |
|                       |   "InternalApiNetValueSpecs": "{\"provider:physical_network\": \"internal_api\", \"provider:network_type\": \"flat\"}",              |
|                       |   "InternalApiNetShared": "False",                                                                                                   |
|                       |   "InternalApiNetAdminStateUp": "False",                                                                                             |
|                       |   "InternalApiNetName": "internal_api"                                                                                               |
|                       | }                                                                                                                                    |
| parent                | None                                                                                                                                 |
| stack_name            | test                                                                                                                                 |
| stack_owner           | admin                                                                                                                                |
| stack_status          | CREATE_COMPLETE                                                                                                                      |
| stack_status_reason   | Stack CREATE completed successfully                                                                                                  |
| stack_user_project_id | bfcf4eba3981475da17d241a062f448c                                                                                                     |
| template_description  | Internal API network. Used for most APIs, Database,                                                                                  |
|                       | RPC.                                                                                                                                 |
| timeout_mins          | None                                                                                                                                 |
| updated_time          | None                                                                                                                                 |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+
Comment 7 Udi Shkalim 2016-01-21 11:33:17 EST
Hi,

The version is OSP 7.
I'll update the bug with the required information once we will start testing on OSP8

Thanks,
Udi
Comment 8 Jaromir Coufal 2016-02-04 02:52:13 EST
Please re-open if this appears to be an issue. Based on tested version and Marius's comment, I am closing for now.
Comment 9 srivatsa 2016-09-16 04:30:07 EDT
Hi, 

I'm getting same error in open stack liberty.

[root@controller prerequisites]# heat stack-create -f abc.hot.orc.yaml -Pf orc=abc.hot.orc.json -P prefix=abcPreReqStack prerequisiteStack
ERROR: Property error: : resources.internal_network_for_ipv6_subnet.properties.allocation_pools[0].start: : "192.168.4.100" does not validate ip_addr (constraint not found)

on heat-engine.log 

2016-09-16 19:35:13.141 23433 INFO heat.engine.resource [req-6dd072b1-560d-41af-a4eb-6045fc164c82 7d2a7dcaa26040e189cce72b412e35d8 b6bafa4dc1c948e99dc2afadfa4c68bb] Validating Subnet "internal_network_for_ipv6_subnet"


error is from the following abc.hot.orc.yaml content 

  internal_network_for_ipv6_subnet:
    type: OS::Neutron::Subnet
    depends_on: [ internal_network_for_ipv6 ]
    properties:
      name: { list_join: [ "-", [ { get_param: prefix }, ipv6 ] ] }
      network: { get_resource: internal_network_for_ipv6 }
      cidr: { get_param: [orc, externalVirtualLinks, ipv6, cidr ] }
      allocation_pools: [
        {"start": { get_param: [orc, externalVirtualLinks, ipv6, pool_start ] },
        "end": { get_param: [orc, externalVirtualLinks, ipv6, pool_end ] } }]
      gateway_ip: null

Let me know if more information is needed.
Comment 10 srivatsa 2016-09-16 04:38:52 EDT
contents of json file
{
	"externalVirtualLinks": 
	{

		"ipv6":
		{
		    "cidr": "192.168.4.0/24",
		    "pool_start" : "192.168.4.100",
		    "pool_end" : "192.168.4.253"
		}
	}
}
Comment 11 srivatsa 2016-09-17 00:11:57 EDT
heat-engine logs show following
=======================================
2016-09-17 14:57:44.789 8068 ERROR stevedore.extension [-] Could not load 'zaqar': No module named zaqarclient.queues.v1
2016-09-17 14:57:44.805 8068 ERROR stevedore.extension [-] Could not load 'mistral': No module named mistralclient.api
2016-09-17 14:57:44.812 8068 ERROR stevedore.extension [-] Could not load 'manila': No module named manilaclient
2016-09-17 14:57:44.813 8068 ERROR stevedore.extension [-] Could not load 'magnum': No module named magnumclient.openstack.common.apiclient
2016-09-17 14:57:44.815 8068 ERROR stevedore.extension [-] Could not load 'barbican': No module named barbicanclient
2016-09-17 14:57:44.845 8068 ERROR stevedore.extension [-] Could not load 'designate': No module named designateclient
2016-09-17 14:57:44.846 8068 ERROR stevedore.extension [-] Could not load 'designate.domain': No module named designateclient
2016-09-17 14:57:44.847 8068 ERROR stevedore.extension [-] Could not load 'ip_addr': No module named croniter
2016-09-17 14:57:44.848 8068 ERROR stevedore.extension [-] Could not load 'manila.share_type': No module named manilaclient
2016-09-17 14:57:44.848 8068 ERROR stevedore.extension [-] Could not load 'timezone': No module named croniter
2016-09-17 14:57:44.849 8068 ERROR stevedore.extension [-] Could not load 'net_cidr': No module named croniter
2016-09-17 14:57:44.850 8068 ERROR stevedore.extension [-] Could not load 'iso_8601': No module named croniter
2016-09-17 14:57:44.850 8068 ERROR stevedore.extension [-] Could not load 'manila.share_network': No module named manilaclient
2016-09-17 14:57:44.851 8068 ERROR stevedore.extension [-] Could not load 'manila.share_snapshot': No module named manilaclient
2016-09-17 14:57:44.852 8068 ERROR stevedore.extension [-] Could not load 'mac_addr': No module named croniter


ip_addr plugin not present.. installing python-crontier and restarting heat engine solves the issue

Note You need to log in before you can comment on or make changes to this bug.