Sridhar,

Can you please confirm if this issue is seen only when extra DHCP options are being used? 

Thanks,
Nir

HI Nir , 
I tried to reproduce it without extra_dhcp_opt  and I  didnt success, 
With extra_dhcp_opt it reproduce all the times .

The  bug reproduce without extrra_dhcp_opt

1. Create network 

#neutron net-create internal_ipv4_6 --shared

2. Create 4 subnets 1 IPv4 and 3 IPv6 Stateful for the network  

# neutron subnet-create <internal_ipv4_6> 192.168.1.0/24 --name internal_ipv4_subnet 

#neutron subnet-create <internal_ipv4_6> 2001:db1:0::2/64 --name dhcpv6_stateful_subnet1 --ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful --ip-version 6

#neutron subnet-create <internal_ipv4_6> 2002:db2:0::2/64 --name dhcpv6_stateful_subnet2 --ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful --ip-version 6

#neutron subnet-create <internal_ipv4_6> 2003:db3:0::2/64 --name dhcpv6_stateful_subnet3 --ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful --ip-version 6

3. Create router and port and attach the port to router 

# neutron router-create router 

# neutron port-create <internal_ipv4_6>

# neutron router-interface-add router port=<port-id>

# neutron router-interface-add router <dhcpv6_stateful_subnet1 id>

4. Launch VM with network 

# nova boot --flavor 3 --image <image-ID> VM1 --nic net-id=<net-id>

anilvenkata_vkommadi  saw it in my ENV

Nir, without extra_dhcp_opts, I was unable to reproduce the issue locally.
Thanks to Anil who was also investigating this issue in parallel and helping Eran. 

Basically the version of dnsmasq on the platform is a bit old (i.e., 2.66) instead of the recommended minimum version of 2.67.

Output of neutron-sanity-check on the platform:
#neutron-sanity-check --dnsmasq_version
2015-06-11 14:23:41.366 9164 ERROR neutron.cmd.sanity_check [-] The installed version of dnsmasq is too old. Please update to at least version 2.67.

Eran, please try with an updated version of dnsmasq and let us know if you still see the issue.

Hi Sridhar, 
I updated dnsmasq version to 
dnsmasq-utils-2.67-1.fc20.x86_64
dnsmasq-2.67-1.fc20.x86_64
And I see that VM get ip addresses as I expected IPv4 & IPv6 .
I have new issue with dnsmasq version 2.67 :

1. dnsmasq advertise incorrect default gateway address 
router get  ipv4 192.168.1.3 & 2003:db3::3

When I check in VM the routing table I see other details : ipv4 192.168.1.1 

2. I also saw that VM  lose his IPv6 address after few minutes and it dont get new one until I don't run "dhclient -6 " . 


please advise .

Hello Eran,

I would like to understand the exact steps you have followed for this issue. 
Can you please give the complete steps (as you seem to invoke neutron router-interface-add twice in Step3 of Comment#5 above) and the observations. 

Thanks,
--Sridhar.

Sridahr , I got your mail and I wrote to you what exactly I did.

Hello Eran,

As discussed over the email, there are two issues which you have reported after moving to dnsmasq version 2.67

Since the new issues are different to the one which you have reported in this bug report, IMO raising separate bugs to track them would be ideal.

Please confirm if you still see the original issue you reported in this Bug even after moving to dnsmasq 2.67. 

Thanks,
--Sridhar.

Sridhar, 
The original issue I reported in this Bug does not reproduce after moving to dnsmasq 2.67. The other issue that I saw in 2.67 as we discussed  I opened a bug:
https://bugs.launchpad.net/neutron/+bug/1465513

RHEL 7.1 ships with dnsmasq 2.66-13 that has the bug fixed. Why not using the package from the repo?

To clarify the matter, there are two separate issues here.

1. dnsmasq 2.66 considers ipv6 entry with an address (stateful) for ipv4, and vice versa. This has nothing to do with extra_dhcp_opts. The issue is fixed by bumping to dnsmasq 2.67. I am pretty sure the following patch is the culprit:

http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=24b5a5d50bc793f7e218ef6018ad210eab28c686

We may want to backport it into next el7 z-release for dnsmasq.

2. dnsmasq of any version quickly bailing out on addressless entry for ipv4 clients, even if there are more dhcp-host entries in host file. This is a limitation of dnsmasq that is going to be fixed by: https://review.openstack.org/#/c/185066/

The limitation shows up in neutron context when there is a ipv6 stateless subnet with an extra dhcp option set. In that case we insert an address-less entry in host file that triggers for ipv4 clients too.

We should clone the bug into dnsmasq component asking to provide a test build of the package with the above mentioned backport included, so that we can locally test it and/or provide to those affected.

@Jon, 2.66-13 does not fix the problem. We also have neutron side of the patch in progress.

Upstream Neutron patch is now merged - https://review.openstack.org/#/c/185066/
Patch for stable/kilo is currently under review - https://review.openstack.org/#/c/197469/

The fix is included in latest rebase.

I deployed ENV by packstack  using the latest version of OSP7 on rhel 7
I run same steps as described in the description &  comment 5 and the bug reproduce again .I verified that  instance image (rhel 7) enabled with IPv6 configuration in ifcfg files .
verified that dnsmasq is running all agents & relevant processes are running on my OSP env.
[root@puma15 ~(keystone_admin)]# rpm -qa |grep dnsmasq
dnsmasq-2.66-14.el7_1.x86_64
dnsmasq-utils-2.66-14.el7_1.x86_64
[root@puma15 ~(keystone_admin)]# rpm -qa |grep neutron 
openstack-neutron-openvswitch-2015.1.1-2.el7ost.noarch
openstack-neutron-common-2015.1.1-2.el7ost.noarch
python-neutronclient-2.4.0-2.el7ost.noarch
openstack-neutron-2015.1.1-2.el7ost.noarch
openstack-neutron-ml2-2015.1.1-2.el7ost.noarch
python-neutron-2015.1.1-2.el7ost.noarch

OK, here is what I did to reproduce the bug:

- set up cloud;
- follow steps from comment 5;
- start f22 cloud image;
- added security group rules for ssh, icmp, icmpv6;
- logged in the instance with ssh;
- added NETWORKING_IPV6=yes in /etc/sysconfig/network;
- added IPV6INIT=yes and DHCPV6C=yes in /etc/sysconfig/network-scripts/ifcfg-eth0;
- killed all dhclients;
- issued the networking script: sudo /etc/sysconfig/network-scripts/ifup-eth eth0.

Here is what I got on console:

[fedora@vm1 ~]$ sudo /etc/sysconfig/network-scripts/ifup-eth eth0

Determining IP information for eth0... done.

Determining IPv6 information for eth0... done.
[fedora@vm1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:91:a9:b0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.12/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 86216sec preferred_lft 86216sec
    inet6 2003:db3::c/128 scope global dynamic 
       valid_lft 7027sec preferred_lft 7027sec
    inet6 fe80::f816:3eff:fe91:a9b0/64 scope link 
       valid_lft forever preferred_lft forever

Note the 2003:db3::c address assigned, as well as 192.168.1.12.

And here is tcpdump output on the instance tap device:

16:07:15.540653 IP6 fe80::f816:3eff:fe4b:4b58 > ff02::1: ICMP6, router advertisement, length 24
...
16:07:18.273892 IP 0.0.0.0.bootpc > broadcasthost.localdomain.bootps: BOOTP/DHCP, Request from fa:16:3e:91:a9:b0 (oui Unknown), length 300
16:07:18.279027 IP 192.168.1.2.bootps > 192.168.1.12.bootpc: BOOTP/DHCP, Reply, length 347
16:07:19.968963 ARP, Request who-has 192.168.1.12 (Broadcast) tell 0.0.0.0, length 28
16:07:20.973270 ARP, Request who-has 192.168.1.12 (Broadcast) tell 0.0.0.0, length 28
...
16:07:29.615213 IP6 fe80::f816:3eff:fe4b:4b58 > ff02::1: ICMP6, router advertisement, length 24
16:07:30.182793 IP6 fe80::f816:3eff:fe91:a9b0.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 confirm
16:07:30.184316 IP6 fe80::f816:3eff:fe00:9cc9.dhcpv6-server > fe80::f816:3eff:fe91:a9b0.dhcpv6-client: dhcp6 reply
16:07:30.350819 IP6 fe80::f816:3eff:fe91:a9b0.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
16:07:30.351074 IP6 fe80::f816:3eff:fe00:9cc9.dhcpv6-server > fe80::f816:3eff:fe91:a9b0.dhcpv6-client: dhcp6 advertise
16:07:31.368580 IP6 fe80::f816:3eff:fe91:a9b0.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 request
16:07:31.372733 IP6 fe80::f816:3eff:fe00:9cc9.dhcpv6-server > fe80::f816:3eff:fe91:a9b0.dhcpv6-client: dhcp6 reply
...

Again, first ipv4 address is issued, then ipv6.

Moving back to MODIFIED since the issue is fixed.

openstack-neutron-2015.1.1-6.el7ost.noarch
dnsmasq-2.66-14.el7_1.x86_64

Followed steps in comment#25:

Before:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:b0:6b:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.5/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 86058sec preferred_lft 86058sec
    inet6 fe80::f816:3eff:feb0:6b35/64 scope link 
       valid_lft forever preferred_lft forever

After:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:b0:6b:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.5/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 86385sec preferred_lft 86385sec
    inet6 2002:db2::5/128 scope global dynamic 
       valid_lft 7187sec preferred_lft 7187sec
    inet6 fe80::f816:3eff:feb0:6b35/64 scope link 
       valid_lft forever preferred_lft forever

Marius I tested the issue with image of rhel 7.1 . 
can you please check if the problem reproduce with rhel 7.1  ??

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1866