Bug 1279505 - Instance rebuild removes some interfaces & also changes interface mac address of the rest
Instance rebuild removes some interfaces & also changes interface mac address...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
6.0 (Juno)
All Linux
urgent Severity urgent
: async
: 6.0 (Juno)
Assigned To: Sahid Ferdjaoui
Prasanth Anbalagan
: Unconfirmed, ZStream
: 1279504 (view as bug list)
Depends On:
Blocks: 1297635
  Show dependency treegraph
 
Reported: 2015-11-09 10:24 EST by Jaison Raju
Modified: 2018-02-08 06:02 EST (History)
18 users (show)

See Also:
Fixed In Version: openstack-nova-2014.2.3-52.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, a race condition would be encountered when instances were assigned several network interfaces, causing some interfaces to not be correctly initialized during boot time, or removed during the build process. With this update, a lock has now been added to the method that maintains the network interfaces for instances, resolving the race condition so that you can correctly boot instances with multiple network interfaces or rebuild instances without losing any network interfaces.
Story Points: ---
Clone Of:
: 1297635 (view as bug list)
Environment:
Last Closed: 2016-02-22 07:33:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
pbandark: needinfo-


Attachments (Terms of Use)
before xml (12.06 KB, text/plain)
2015-11-09 10:26 EST, Jaison Raju
no flags Details
after xml (10.87 KB, text/plain)
2015-11-09 10:26 EST, Jaison Raju
no flags Details
before hardreboot (12.06 KB, text/plain)
2015-11-09 10:28 EST, Jaison Raju
no flags Details
after hardreboot (10.28 KB, text/plain)
2015-11-09 10:29 EST, Jaison Raju
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1522112 None None None 2015-12-16 07:40 EST
OpenStack gerrit 252565 None None None 2015-12-16 07:42 EST

  None (edit)
Description Jaison Raju 2015-11-09 10:24:52 EST
Description of problem:
Instance rebuild removes some interfaces & also changes interface mac address of the rest .
This same behavior is noticed after instance hard reboot too .

Version-Release number of selected component (if applicable):
openstack-nova-conductor-2014.2.3-32.el7ost.noarch
openstack-nova-scheduler-2014.2.3-32.el7ost.noarch
openstack-nova-api-2014.2.3-32.el7ost.noarch
openstack-nova-cert-2014.2.3-32.el7ost.noarch
openstack-nova-common-2014.2.3-32.el7ost.noarch
openstack-nova-console-2014.2.3-32.el7ost.noarch
openstack-nova-novncproxy-2014.2.3-32.el7ost.noarch

How reproducible:
Always on customer environment 

Steps to Reproduce:
1. Rebuild instance (on / off state) with multiple interface .
Or hard reboot instance.
2. 
3.

Actual results:
All Interfaces are present on instance are not present after rebuild .
The ones present also have different mac address than before .

Expected results:
All Interfaces are present on instance after rebuild & have same mac address .

Additional info:
Comment 2 Jaison Raju 2015-11-09 10:26 EST
Created attachment 1091799 [details]
before xml
Comment 3 Jaison Raju 2015-11-09 10:26 EST
Created attachment 1091800 [details]
after xml
Comment 4 Jaison Raju 2015-11-09 10:28 EST
Created attachment 1091801 [details]
before hardreboot
Comment 5 Jaison Raju 2015-11-09 10:29 EST
Created attachment 1091802 [details]
after hardreboot
Comment 8 Eoghan Glynn 2015-11-13 10:09:05 EST
*** Bug 1279504 has been marked as a duplicate of this bug. ***
Comment 9 Vladik Romanovsky 2015-11-13 11:53:54 EST
Hi Jaison,

Could you please provide logs from the compute node and the controller,
right after the rebuild?

Thanks,
Vladik
Comment 30 Prasanth Anbalagan 2016-02-12 08:34:11 EST
Verified as follows - Interface list of an instance remains intact after rebuid.
**********
Version
**********

[root@rhos-compute-node-02 ~(keystone_admin)]# yum list installed  grep openstack-nova
openstack-nova-api.noarch            2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-cert.noarch           2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-common.noarch         2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-compute.noarch        2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-conductor.noarch      2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-console.noarch        2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-novncproxy.noarch     2014.2.3-54.el7ost      @rhelosp-6.0-puddle
openstack-nova-scheduler.noarch      2014.2.3-54.el7ost      @rhelosp-6.0-puddle

************
Logs
************
[root@rhos-compute-node-02 ~(keystone_admin)]# nova boot --flavor 1 --image cirros vm01
+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | 3XcxXs8Z48d7                                  |
| config_drive                         |                                               |
| created                              | 2016-02-12T13:21:54Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | 7148d041-78f2-4059-bc6b-2a626d4f4916          |
| image                                | cirros (41ced790-3280-4beb-86e9-ead787f71884) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | vm01                                          |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tenant_id                            | 30c68d93471f406481bd9cb69034c1b9              |
| updated                              | 2016-02-12T13:21:55Z                          |
| user_id                              | 269f954a9e2b42d2a303332929bc3221              |
+--------------------------------------+-----------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| 7148d041-78f2-4059-bc6b-2a626d4f4916 | vm01 | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 201fb6f8-c3d3-4021-94e3-b8bc548a1c4c vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 3af6683e-6111-4a0a-81cd-0c743bd220c8 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 64940b4a-d3b9-4112-87c8-93eb52fc0122 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 669fb10e-7d26-438b-a03a-e69027f96180 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 6d11f5f4-89f3-473b-b03b-84bb22b55374 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id 83c41771-5e3e-46f6-a9d9-56be50953a7e vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id cd5b0039-87d1-4e90-a118-cc6f9f5cc9b0 vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id df5ddacd-84df-41f6-aead-4b72520168aa vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# nova interface-attach --port-id e88764c9-e1b0-434b-9bed-d27b889a1c0d vm01
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     instance-00000001              running

[root@rhos-compute-node-02 ~(keystone_admin)]# virsh dumpxml 2 | grep mac
    <partition>/machine</partition>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
      <mac address='fa:16:3e:ed:fc:d0'/>
      <mac address='fa:16:3e:ef:2a:63'/>
      <mac address='fa:16:3e:80:b7:3f'/>
      <mac address='fa:16:3e:ca:35:bf'/>
      <mac address='fa:16:3e:d4:c4:ac'/>
      <mac address='fa:16:3e:79:47:8e'/>
      <mac address='fa:16:3e:72:eb:7b'/>
      <mac address='fa:16:3e:56:01:f7'/>
      <mac address='fa:16:3e:ff:a6:21'/>
      <mac address='fa:16:3e:8c:ae:e5'/>
[root@rhos-compute-node-02 ~(keystone_admin)]# 

[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova rebuild vm01 cirros
+-------------------+----------------------------------------------------------+
| Property          | Value                                                    |
+-------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL                                                   |
| accessIPv4        |                                                          |
| accessIPv6        |                                                          |
| adminPass         | ipvyZYZmvZ2Z                                             |
| created           | 2016-02-12T13:21:54Z                                     |
| flavor            | m1.tiny (1)                                              |
| hostId            | 9bc8bac5c152ff7731522ac40b13db671f083abdc30f24b03a35e4f5 |
| id                | 7148d041-78f2-4059-bc6b-2a626d4f4916                     |
| image             | cirros (41ced790-3280-4beb-86e9-ead787f71884)            |
| metadata          | {}                                                       |
| name              | vm01                                                     |
| net1 network      |                                                          |
| net2 network      |                                                          |
| net3 network      |                                                          |
| net4 network      |                                                          |
| net5 network      |                                                          |
| net6 network      |                                                          |
| net7 network      |                                                          |
| net8 network      |                                                          |
| net9 network      |                                                          |
| progress          | 0                                                        |
| public network    | 172.24.4.227                                             |
| status            | REBUILD                                                  |
| tenant_id         | 30c68d93471f406481bd9cb69034c1b9                         |
| updated           | 2016-02-12T13:31:07Z                                     |
| user_id           | 269f954a9e2b42d2a303332929bc3221                         |
+-------------------+----------------------------------------------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+---------+------------+-------------+---------------------+
| ID                                   | Name | Status  | Task State | Power State | Networks            |
+--------------------------------------+------+---------+------------+-------------+---------------------+
| 7148d041-78f2-4059-bc6b-2a626d4f4916 | vm01 | REBUILD | rebuilding | Running     | public=172.24.4.227 |
+--------------------------------------+------+---------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| 7148d041-78f2-4059-bc6b-2a626d4f4916 | vm01 | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# 
[root@rhos-compute-node-02 ~(keystone_admin)]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     instance-00000001              running

[root@rhos-compute-node-02 ~(keystone_admin)]# virsh dumpxml 3 | grep mac
    <partition>/machine</partition>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
      <mac address='fa:16:3e:ed:fc:d0'/>
      <mac address='fa:16:3e:ef:2a:63'/>
      <mac address='fa:16:3e:80:b7:3f'/>
      <mac address='fa:16:3e:ca:35:bf'/>
      <mac address='fa:16:3e:d4:c4:ac'/>
      <mac address='fa:16:3e:79:47:8e'/>
      <mac address='fa:16:3e:72:eb:7b'/>
      <mac address='fa:16:3e:56:01:f7'/>
      <mac address='fa:16:3e:ff:a6:21'/>
      <mac address='fa:16:3e:8c:ae:e5'/>
[root@rhos-compute-node-02 ~(keystone_admin)]#
Comment 32 errata-xmlrpc 2016-02-22 07:33:02 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0283.html

Note You need to log in before you can comment on or make changes to this bug.