Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1279505

Summary:

Instance rebuild removes some interfaces & also changes interface mac address of the rest

Product:

Red Hat OpenStack

Reporter:

Jaison Raju <jraju>

Component:

openstack-nova

Assignee:

Sahid Ferdjaoui <sferdjao>

Status:

CLOSED ERRATA

QA Contact:

Prasanth Anbalagan <panbalag>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

6.0 (Juno)

CC:

adahms, berrange, dasmith, dmaley, eglynn, jraju, kchamart, ndipanov, ochalups, panbalag, pbandark, rcernin, sbauza, sferdjao, sgordon, vromanso, yeylon

Target Milestone:

async

Keywords:

Unconfirmed, ZStream

Target Release:

6.0 (Juno)

Flags:

pbandark: needinfo-

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

openstack-nova-2014.2.3-52.el7ost

Doc Type:

Bug Fix

Doc Text:

Previously, a race condition would be encountered when instances were assigned several network interfaces, causing some interfaces to not be correctly initialized during boot time, or removed during the build process. With this update, a lock has now been added to the method that maintains the network interfaces for instances, resolving the race condition so that you can correctly boot instances with multiple network interfaces or rebuild instances without losing any network interfaces.

Story Points:

---

Clone Of:

Clones:

1297635 (view as bug list)

Environment:

Last Closed:

2016-02-22 12:33:02 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1297635

Attachments:

Description	Flags
before xml	none
after xml	none
before hardreboot	none
after hardreboot	none

Description Jaison Raju 2015-11-09 15:24:52 UTC

Description of problem:
Instance rebuild removes some interfaces & also changes interface mac address of the rest .
This same behavior is noticed after instance hard reboot too .

Version-Release number of selected component (if applicable):
openstack-nova-conductor-2014.2.3-32.el7ost.noarch
openstack-nova-scheduler-2014.2.3-32.el7ost.noarch
openstack-nova-api-2014.2.3-32.el7ost.noarch
openstack-nova-cert-2014.2.3-32.el7ost.noarch
openstack-nova-common-2014.2.3-32.el7ost.noarch
openstack-nova-console-2014.2.3-32.el7ost.noarch
openstack-nova-novncproxy-2014.2.3-32.el7ost.noarch

How reproducible:
Always on customer environment 

Steps to Reproduce:
1. Rebuild instance (on / off state) with multiple interface .
Or hard reboot instance.
2. 
3.

Actual results:
All Interfaces are present on instance are not present after rebuild .
The ones present also have different mac address than before .

Expected results:
All Interfaces are present on instance after rebuild & have same mac address .

Additional info:

Comment 2 Jaison Raju 2015-11-09 15:26:34 UTC

Created attachment 1091799 [details]
before xml

Comment 3 Jaison Raju 2015-11-09 15:26:58 UTC

Created attachment 1091800 [details]
after xml

Comment 4 Jaison Raju 2015-11-09 15:28:46 UTC

Created attachment 1091801 [details]
before hardreboot

Comment 5 Jaison Raju 2015-11-09 15:29:13 UTC

Created attachment 1091802 [details]
after hardreboot

Comment 8 Eoghan Glynn 2015-11-13 15:09:05 UTC

*** Bug 1279504 has been marked as a duplicate of this bug. ***

Comment 9 Vladik Romanovsky 2015-11-13 16:53:54 UTC

Hi Jaison,

Could you please provide logs from the compute node and the controller,
right after the rebuild?

Thanks,
Vladik

Comment 30 Prasanth Anbalagan 2016-02-12 13:34:11 UTC

Verified as follows - Interface list of an instance remains intact after rebuid.
**********
Version
**********

[root@rhos-compute-node-02 ~(keystone_admin)]# yum list installed  grep openstack-nova
openstack-nova-api.noarch            2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-cert.noarch           2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-common.noarch         2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-compute.noarch        2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-conductor.noarch      2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-console.noarch        2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-novncproxy.noarch     2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-scheduler.noarch      2014.2.3-54.el7ost      @rhelosp-6.0-puddle

************
Logs
************
[root@rhos-compute-node-02 ~(keystone_admin)]# nova boot --flavor 1 --image cirros vm01
+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | 3XcxXs8Z48d7                                  |
| config_drive                         |                                               |
| created                              | 2016-02-12T13:21:54Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | 7148d041-78f2-4059-bc6b-2a626d4f4916          |
| image                                | cirros (41ced790-3280-4beb-86e9-ead787f71884) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | vm01                                          |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tenant_id                            | 30c68d93471f406481bd9cb69034c1b9              |
| updated                              | 2016-02-12T13:21:55Z                          |
| user_id                              | 269f954a9e2b42d2a303332929bc3221              |
+--------------------------------------+-----------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| 7148d041-78f2-4059-bc6b-2a626d4f4916 | vm01 | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 201fb6f8-c3d3-4021-94e3-b8bc548a1c4c vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 3af6683e-6111-4a0a-81cd-0c743bd220c8 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 64940b4a-d3b9-4112-87c8-93eb52fc0122 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 669fb10e-7d26-438b-a03a-e69027f96180 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 6d11f5f4-89f3-473b-b03b-84bb22b55374 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 83c41771-5e3e-46f6-a9d9-56be50953a7e vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id cd5b0039-87d1-4e90-a118-cc6f9f5cc9b0 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id df5ddacd-84df-41f6-aead-4b72520168aa vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id e88764c9-e1b0-434b-9bed-d27b889a1c0d vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     instance-00000001              running

[root@rhos-compute-node-02 ~(keystone_admin)]# virsh dumpxml 2 | grep mac
    <partition>/machine</partition>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
      <mac address='fa:16:3e:ed:fc:d0'/>
      <mac address='fa:16:3e:ef:2a:63'/>
      <mac address='fa:16:3e:80:b7:3f'/>
      <mac address='fa:16:3e:ca:35:bf'/>
      <mac address='fa:16:3e:d4:c4:ac'/>
      <mac address='fa:16:3e:79:47:8e'/>
      <mac address='fa:16:3e:72:eb:7b'/>
      <mac address='fa:16:3e:56:01:f7'/>
      <mac address='fa:16:3e:ff:a6:21'/>
      <mac address='fa:16:3e:8c:ae:e5'/>
[root@rhos-compute-node-02 ~(keystone_admin)]# 

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova rebuild vm01 cirros
+-------------------+----------------------------------------------------------+
| Property          | Value                                                    |
+-------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL                                                   |
| accessIPv4        |                                                          |
| accessIPv6        |                                                          |
| adminPass         | ipvyZYZmvZ2Z                                             |
| created           | 2016-02-12T13:21:54Z                                     |
| flavor            | m1.tiny (1)                                              |
| hostId            | 9bc8bac5c152ff7731522ac40b13db671f083abdc30f24b03a35e4f5 |
| id                | 7148d041-78f2-4059-bc6b-2a626d4f4916                     |
| image             | cirros (41ced790-3280-4beb-86e9-ead787f71884)            |
| metadata          | {}                                                       |
| name              | vm01                                                     |
| net1 network      |                                                          |
| net2 network      |                                                          |
| net3 network      |                                                          |
| net4 network      |                                                          |
| net5 network      |                                                          |
| net6 network      |                                                          |
| net7 network      |                                                          |
| net8 network      |                                                          |
| net9 network      |                                                          |
| progress          | 0                                                        |
| public network    | 172.24.4.227                                             |
| status            | REBUILD                                                  |
| tenant_id         | 30c68d93471f406481bd9cb69034c1b9                         |
| updated           | 2016-02-12T13:31:07Z                                     |
| user_id           | 269f954a9e2b42d2a303332929bc3221                         |
+-------------------+----------------------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+---------+------------+-------------+---------------------+
| ID                                   | Name | Status  | Task State | Power State | Networks            |
+--------------------------------------+------+---------+------------+-------------+---------------------+
| 7148d041-78f2-4059-bc6b-2a626d4f4916 | vm01 | REBUILD | rebuilding | Running     | public=172.24.4.227 |
+--------------------------------------+------+---------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| 7148d041-78f2-4059-bc6b-2a626d4f4916 | vm01 | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     instance-00000001              running

[root@rhos-compute-node-02 ~(keystone_admin)]# virsh dumpxml 3 | grep mac
    <partition>/machine</partition>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
      <mac address='fa:16:3e:ed:fc:d0'/>
      <mac address='fa:16:3e:ef:2a:63'/>
      <mac address='fa:16:3e:80:b7:3f'/>
      <mac address='fa:16:3e:ca:35:bf'/>
      <mac address='fa:16:3e:d4:c4:ac'/>
      <mac address='fa:16:3e:79:47:8e'/>
      <mac address='fa:16:3e:72:eb:7b'/>
      <mac address='fa:16:3e:56:01:f7'/>
      <mac address='fa:16:3e:ff:a6:21'/>
      <mac address='fa:16:3e:8c:ae:e5'/>
[root@rhos-compute-node-02 ~(keystone_admin)]#

Comment 32 errata-xmlrpc 2016-02-22 12:33:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0283.html