Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1366254

Summary: New deployments with IPv6 fails post-deployment initialization
Product: Red Hat OpenStack Reporter: Marian Krcmarik <mkrcmari>
Component: python-tripleoclientAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Tomas Jamrisko <tjamrisk>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: hbrock, jason.dobies, jslagle, mburns, mcornea, oblaut, ohochman, rhel-osp-director-maint, sclewis, tvignaud
Target Milestone: gaKeywords: Regression, Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-2.0.0-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1367443 (view as bug list) Environment:
Last Closed: 2016-08-24 13:31:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
external-lb.yml
none
openstack stack show overcloud none

Description Marian Krcmarik 2016-08-11 11:50:42 UTC
Created attachment 1190037 [details]
external-lb.yml

Description of problem:
I am deploying RHOS9 on IPv6 topology with external loadbalancer - the overcloud deploy fails on following error:
Error: Parameter name failed on Ring_object_device[fd00:fd00:fd00:4000::fb:6000/d1]: Validate method failed for class name: the scheme http does not accept registry part: fd00:fd00:fd00:4000::fb:6000 (or bad hostname?) at /var/lib/heat-config/heat-config-puppet/32cc6e6f-c4c2-492c-88f3-8b516459e9fa.pp:42
Wrapped exception:
Validate method failed for class name: the scheme http does not accept registry part: fd00:fd00:fd00:4000::fb:6000 (or bad hostname?)

Based on the past bug - https://bugzilla.redhat.com/show_bug.cgi?id=1296701 It seems that IPv6 should be wrapped in brackets [] but It did not happen in this case when static single IPv6 is assigned and taken from external-lb.yaml passed to ovecloud deploy (attached to bug).

# cat /etc/puppet/hieradata/swift_devices_and_proxy.yaml swift::proxy::cache::memcache_servers: ['fd00:fd00:fd00:2000::fb:11211','fd00:fd00:fd00:2000::fc:11211','fd00:fd00:fd00:2000::fd:11211']
tripleo::ringbuilder::devices: r1z1-fd00:fd00:fd00:4000::fb:%PORT%/d1, r1z1-fd00:fd00:fd00:4000::fc:%PORT%/d1, r1z1-fd00:fd00:fd00:4000::fd:%PORT%/d1,
- The IPv6 is not in brackets, on setup without external loadbalancer and with IPv6 allocation range of IPs to be assigned these IPv6 addresses are wrapped by brackets.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-liberty-2.0.0-30.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-30.el7ost.noarch
python-uri-templates-0.6-5.el7ost.noarch
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy overcloud on IPv6 with external loadbalancer and single statis IPv6 assigned and specified in external-lb.yml example (in attachment)

Actual results:
Ring_object_device[fd00:fd00:fd00:4000::fb:6000/d1]: Validate method failed for class name: the scheme http does not accept registry part: fd00:fd00:fd00:4000::fb:6000 (or bad hostname?) at /var/lib/heat-config/heat-config-puppet/32cc6e6f-c4c2-492c-88f3-8b516459e9fa.pp:42

Expected results:
Passed deploy

Additional info:

Comment 2 Marian Krcmarik 2016-08-11 12:01:47 UTC
Created attachment 1190038 [details]
openstack stack show overcloud

Comment 3 Giulio Fidente 2016-08-11 15:34:56 UTC
From the stack show output attached it looks like none of the IPs in the EndpointMap had brackets either.

Can you paste the deploy command line? I suspect the order in which the environments file are passed could get some resources overridden with the wrong template.

Comment 4 Marian Krcmarik 2016-08-11 15:41:53 UTC
(In reply to Giulio Fidente from comment #3)
> From the stack show output attached it looks like none of the IPs in the
> EndpointMap had brackets either.
> 
> Can you paste the deploy command line? I suspect the order in which the
> environments file are passed could get some resources overridden with the
> wrong template.

openstack overcloud deploy --debug \
 --templates \
 --libvirt-type qemu \
 --ntp-server clock.redhat.com \
 --neutron-network-type vxlan \
 --neutron-tunnel-types vxlan \
 --control-scale 3 \
 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \
 --compute-scale 1 \
 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
 -e /home/stack/virt/network/network-environment-v6.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/external-loadbalancer-vip.yaml \
 -e ~/external-lb.yaml \
 -e /home/stack/virt/hostnames.yml \
 --log-file overcloud_deployment_9.log

Comment 5 Giulio Fidente 2016-08-11 15:49:13 UTC
I think I spotted it, you should be using:

external-loadbalancer-vip-v6.yaml

not

external-loadbalancer-vip.yaml

Comment 6 Marian Krcmarik 2016-08-11 15:58:27 UTC
(In reply to Giulio Fidente from comment #5)
> I think I spotted it, you should be using:
> 
> external-loadbalancer-vip-v6.yaml
> 
> not
> 
> external-loadbalancer-vip.yaml

Oh, let me rerun the deploy and see the result, will close the bug If It passes.

Comment 7 Marian Krcmarik 2016-08-11 21:55:27 UTC
(In reply to Giulio Fidente from comment #5)
> I think I spotted it, you should be using:
> 
> external-loadbalancer-vip-v6.yaml
> 
> not
> 
> external-loadbalancer-vip.yaml

Giulio,

That was the problem, thanks for spotting it but while we are on it, I got another (maybe opposite) problem, overcloud deploy fails on following error:
2016-08-11 15:01:33.311 1839 ERROR openstack [ admin admin] Command '['ssh', '-oStrictHostKeyChecking=no', '-t', '-l', 'heat-admin', u'[fd00:fd00:fd00:2000::250]', 'sudo', 'key
stone-manage', 'pki_setup', '--keystone-user', "$(getent passwd | grep '^keystone' | cut -d: -f1)", '--keystone-group', "$(getent group | grep '^keystone' | cut -d: -f1)"]' ret
urned non-zero exit status 255

I tried to run it manually from undercloud (hopefully I made ssh command correctly):
$ ssh -oStrictHostKeyChecking=no -t -l heat-admin [fd00:fd00:fd00:2000::250] sudo keystone-manage pki_setup --keystone-user "$(getent passwd | grep '^keystone' | cut -d: -f1)" --keystone-group "$(getent group | grep '^keystone' | cut -d: -f1)"
ssh: Could not resolve hostname [fd00:fd00:fd00:2000::250]: Name or service not known

If I remove brackets around the IPv6:
$ ssh -oStrictHostKeyChecking=no -t -l heat-admin fd00:fd00:fd00:2000::250 sudo keystone-manage pki_setup --keystone-user "$(getent passwd | grep '^keystone' | cut -d: -f1)" --keystone-group "$(getent group | grep '^keystone' | cut -d: -f1)"
Add correct host key in /home/stack/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/stack/.ssh/known_hosts:2
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to fd00:fd00:fd00:2000::250 closed.

Is that a bug or I missed something somewhere again?

Thanks.

Comment 8 Giulio Fidente 2016-08-11 23:23:12 UTC
I suppose the stack is in CREATE_COMPLETE at this stage, it's just the post-install command which fails

The Swift error wasn't an error, but the post-deploy error is a real bug, introduced with tripleoclient commit d67ccaf7baf7beb7ea3559c84ae0c645d0452dd8

I think we can implement a change in tripleoclient to support both TLS and IPv6, working on it

Comment 9 Giulio Fidente 2016-08-11 23:38:58 UTC
Note, the tripleoclient change has been ported to upstream liberty as well so we'll need to fix it there too.

Comment 11 Marian Krcmarik 2016-08-12 09:53:26 UTC
(In reply to Giulio Fidente from comment #8)
> I suppose the stack is in CREATE_COMPLETE at this stage, it's just the
> post-install command which fails
> 
Yes
> The Swift error wasn't an error, but the post-deploy error is a real bug,
> introduced with tripleoclient commit d67ccaf7baf7beb7ea3559c84ae0c645d0452dd8
> 
> I think we can implement a change in tripleoclient to support both TLS and
> IPv6, working on it
Not sure what TLS support is meant here but I did not enable TLS support for openstack overcloud (AFAIK) - should be that pki_setup run anyway?

Comment 14 Marian Krcmarik 2016-08-13 00:08:02 UTC
FYI: I tried Patch Set 4 from the attached gerrit change and overcloud deployment passed successfully

Comment 19 errata-xmlrpc 2016-08-24 13:31:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1766.html