Bug 1244139

Summary:	Updating overcloud-full image breaks deployments
Product:	Red Hat OpenStack	Reporter:	Steven Hardy <shardy>
Component:	rhosp-director	Assignee:	Marek Aufart <maufart>
Status:	CLOSED DUPLICATE	QA Contact:	yeylon <yeylon>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	unspecified	CC:	calfonso, jslagle, maufart, mburns, rhel-osp-director-maint, srevivo
Target Milestone:	z1	Keywords:	Triaged
Target Release:	Director
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-07-24 16:29:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Steven Hardy 2015-07-17 09:46:22 UTC

Description of problem:

I'm not quite sure if this is a docs issue, a bug in the oscplugin or an actual bug in the underlying services, but atm I'm unable to find a working sequence which allows replacing the overcloud-full image with a new one and have nova/ironic work again afterwards.

Any attempt to update the image results in nova servers which go immediately to ERROR, saying no valid host found, so I assume the image ID is persisted somewhere and I'm not updating it after switching images in glance.

I'm not changing the kernel or ramdisk, and AFAICT openstack baremetal configure boot doesn't fix it (that only consumes the kernel and ramdisk anyway, right?)


Version-Release number of selected component (if applicable):
Poodle from yesterday

How reproducible:
Always

Steps to Reproduce:

Overcloud Deployed
[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| fe82819d-cccf-4cec-98c3-0a67d64df0ab | overcloud  | CREATE_COMPLETE | 2015-07-17T08:37:28Z |
+--------------------------------------+------------+-----------------+----------------------+
[stack@instack ~]$ glance image-list
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                   | Disk Format | Container Format | Size      | Status |
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
| d0b2a5bc-c173-4958-9c8f-adaa031cdec7 | bm-deploy-kernel       | aki         | aki              | 5026624   | active |
| 93daaad4-e7ad-46ee-8ba3-0d14b91bfdab | bm-deploy-ramdisk      | ari         | ari              | 58839135  | active |
| 5eb0700e-7ebe-4482-9bcf-6515647abd00 | overcloud-full         | qcow2       | bare             | 974060544 | active |
| 53602412-f522-42ef-ab07-654eec764b14 | overcloud-full-initrd  | ari         | ari              | 34147388  | active |
| 82eb498e-b1b2-423a-9c91-338153d17ee9 | overcloud-full-vmlinuz | aki         | aki              | 5027584   | active |
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
[stack@instack ~]$ heat stack-delete overcloud
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| fe82819d-cccf-4cec-98c3-0a67d64df0ab | overcloud  | DELETE_IN_PROGRESS | 2015-07-17T08:37:28Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@instack ~]$ glance image-delete 5eb0700e-7ebe-4482-9bcf-6515647abd00
[stack@instack ~]$ glance image-list
+--------------------------------------+------------------------+-------------+------------------+----------+--------+
| ID                                   | Name                   | Disk Format | Container Format | Size     | Status |
+--------------------------------------+------------------------+-------------+------------------+----------+--------+
| d0b2a5bc-c173-4958-9c8f-adaa031cdec7 | bm-deploy-kernel       | aki         | aki              | 5026624  | active |
| 93daaad4-e7ad-46ee-8ba3-0d14b91bfdab | bm-deploy-ramdisk      | ari         | ari              | 58839135 | active |
| 53602412-f522-42ef-ab07-654eec764b14 | overcloud-full-initrd  | ari         | ari              | 34147388 | active |
| 82eb498e-b1b2-423a-9c91-338153d17ee9 | overcloud-full-vmlinuz | aki         | aki              | 5027584  | active |
+--------------------------------------+------------------------+-------------+------------------+----------+--------+
[stack@instack ~]$ openstack overcloud image upload
[stack@instack ~]$ glance image-list
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                   | Disk Format | Container Format | Size      | Status |
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
| d0b2a5bc-c173-4958-9c8f-adaa031cdec7 | bm-deploy-kernel       | aki         | aki              | 5026624   | active |
| 230e9704-4178-4f2d-a3a8-99b82633df6a | bm-deploy-kernel       | aki         | aki              | 5026624   | active |
| 93daaad4-e7ad-46ee-8ba3-0d14b91bfdab | bm-deploy-ramdisk      | ari         | ari              | 58839135  | active |
| 951fb565-4bd3-46ff-84a6-f5a3713498bc | bm-deploy-ramdisk      | ari         | ari              | 58839135  | active |
| 4b96c302-fbc2-417b-a45a-b199674d3868 | overcloud-full         | qcow2       | bare             | 974060544 | active |
| 53602412-f522-42ef-ab07-654eec764b14 | overcloud-full-initrd  | ari         | ari              | 34147388  | active |
| b1cab2de-42a2-4fdc-b425-b43509fedb9e | overcloud-full-initrd  | ari         | ari              | 34147388  | active |
| 82eb498e-b1b2-423a-9c91-338153d17ee9 | overcloud-full-vmlinuz | aki         | aki              | 5027584   | active |
| 38739910-ea86-4758-a0c9-66580b28c639 | overcloud-full-vmlinuz | aki         | aki              | 5027584   | active |
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
# Delete duplicates due to bz#1243829
[stack@instack ~]$ glance image-delete 230e9704-4178-4f2d-a3a8-99b82633df6a 951fb565-4bd3-46ff-84a6-f5a3713498bc b1cab2de-42a2-4fdc-b425-b43509fedb9e 38739910-ea86-4758-a0c9-66580b28c639
[stack@instack ~]$ glance image-list
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                   | Disk Format | Container Format | Size      | Status |
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
| d0b2a5bc-c173-4958-9c8f-adaa031cdec7 | bm-deploy-kernel       | aki         | aki              | 5026624   | active |
| 93daaad4-e7ad-46ee-8ba3-0d14b91bfdab | bm-deploy-ramdisk      | ari         | ari              | 58839135  | active |
| 4b96c302-fbc2-417b-a45a-b199674d3868 | overcloud-full         | qcow2       | bare             | 974060544 | active |
| 53602412-f522-42ef-ab07-654eec764b14 | overcloud-full-initrd  | ari         | ari              | 34147388  | active |
| 82eb498e-b1b2-423a-9c91-338153d17ee9 | overcloud-full-vmlinuz | aki         | aki              | 5027584   | active |
+--------------------------------------+------------------------+-------------+------------------+-----------+--------+
[stack@instack ~]$ heat stack-list

+----+------------+--------------+---------------+
| id | stack_name | stack_status | creation_time |
+----+------------+--------------+---------------+
+----+------------+--------------+---------------+
[stack@instack ~]$ 
[stack@instack ~]$ openstack overcloud deploy --templates
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
ERROR: openstack Heat Stack create failed.
[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+----------+
| ID                                   | Name                   | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------+--------+------------+-------------+----------+
| bca16d59-bd2a-4aaa-8896-1985d7dc84a3 | overcloud-compute-0    | ERROR  | -          | NOSTATE     |          |
| 9a5275ce-d55c-453e-af66-814d48a1788f | overcloud-controller-0 | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+------------------------+--------+------------+-------------+----------+
[stack@instack ~]$ nova show bca16d59-bd2a-4aaa-8896-1985d7dc84a3
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                                                                                                                                      |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                                       |
| OS-EXT-SRV-ATTR:host                 | instack.localdomain                                                                                                                                                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | 3b013b04-a7df-4674-977c-4016007cf22a                                                                                                                                                                       |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000000f                                                                                                                                                                                          |
| OS-EXT-STS:power_state               | 0                                                                                                                                                                                                          |
| OS-EXT-STS:task_state                | -                                                                                                                                                                                                          |
| OS-EXT-STS:vm_state                  | error                                                                                                                                                                                                      |
| OS-SRV-USG:launched_at               | -                                                                                                                                                                                                          |
| OS-SRV-USG:terminated_at             | -                                                                                                                                                                                                          |
| accessIPv4                           |                                                                                                                                                                                                            |
| accessIPv6                           |                                                                                                                                                                                                            |
| config_drive                         |                                                                                                                                                                                                            |
| created                              | 2015-07-17T09:25:27Z                                                                                                                                                                                       |
| fault                                | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 671, in build_instances |
|                                      |     request_spec, filter_properties)                                                                                                                                                                       |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 337, in wrapped                                                                                                                  |
|                                      |     return func(*args, **kwargs)                                                                                                                                                                           |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 52, in select_destinations                                                                                             |
|                                      |     context, request_spec, filter_properties)                                                                                                                                                              |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method                                                                                                    |
|                                      |     return getattr(self.instance, __name)(*args, **kwargs)                                                                                                                                                 |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 34, in select_destinations                                                                                                |
|                                      |     context, request_spec, filter_properties)                                                                                                                                                              |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 120, in select_destinations                                                                                                     |
|                                      |     request_spec=request_spec, filter_properties=filter_properties)                                                                                                                                        |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 156, in call                                                                                                                |
|                                      |     retry=self.retry)                                                                                                                                                                                      |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 90, in _send                                                                                                                 |
|                                      |     timeout=timeout, retry=retry)                                                                                                                                                                          |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 350, in send                                                                                                       |
|                                      |     retry=retry)                                                                                                                                                                                           |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 341, in _send                                                                                                      |
|                                      |     raise result                                                                                                                                                                                           |
|                                      | ", "created": "2015-07-17T09:25:44Z"}                                                                                                                                                                      |
| flavor                               | baremetal (4f7afe55-5586-410a-bbd8-757f5e021d27)                                                                                                                                                           |
| hostId                               | a837751017a123cd36c035f1e3d5d7ab56de2463274b7bb7ab629315                                                                                                                                                   |
| id                                   | bca16d59-bd2a-4aaa-8896-1985d7dc84a3                                                                                                                                                                       |
| image                                | overcloud-full (4b96c302-fbc2-417b-a45a-b199674d3868)                                                                                                                                                      |
| key_name                             | default                                                                                                                                                                                                    |
| metadata                             | {}                                                                                                                                                                                                         |
| name                                 | overcloud-compute-0                                                                                                                                                                                        |
| os-extended-volumes:volumes_attached | []                                                                                                                                                                                                         |
| status                               | ERROR                                                                                                                                                                                                      |
| tenant_id                            | 0802efafbaf6412b9f78abe1e37d86b1                                                                                                                                                                           |
| updated                              | 2015-07-17T09:25:43Z                                                                                                                                                                                       |
| user_id                              | b012eea4ba84451bbacafaee35f7d029                                                                                                                                                                           |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+





Actual results:
Nova servers always go to ERROR state after updating the image, resultinh in the deployment failing in a pretty non-obvious way.  

The experience is made worse because when you then try to delete the stack, it fails due to bz#1242796


Expected results:
We should have a series of steps which allows operators to update the openstack-full image and retain a functional deploy capability.

I hit this when I was attempting to patch my image (using virt-copy-in) and upload a version with a fix for testing.  I assume this will be a pretty common requirement, in addition to consuming updated images via CDN.

Additional info:

Comment 3 James Slagle 2015-07-17 11:29:02 UTC

the overcloud-full kernel and ramdisk id's are saved as glance properties on the qcow2 image:

[stack@instack ~]$ glance image-show 6e5c161b-d31e-47e2-82c9-8cb80ae8ce04
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| Property 'kernel_id'  | 00d1c4ab-137a-4135-8095-c78a4af3dcc3 |
| Property 'ramdisk_id' | a5bd6e54-0274-45f8-af87-0d8c42364b5f |
| checksum              | fae896e079de34860c78d39baca3301a     |
| container_format      | bare                                 |
| created_at            | 2015-07-14T21:39:25.000000           |
| deleted               | False                                |
| disk_format           | qcow2                                |
| id                    | 6e5c161b-d31e-47e2-82c9-8cb80ae8ce04 |
| is_public             | True                                 |
| min_disk              | 0                                    |
| min_ram               | 0                                    |
| name                  | overcloud-full                       |
| owner                 | 8027be3dc82b4bf7a91c3e25c2622aad     |
| protected             | False                                |
| size                  | 914223616                            |
| status                | active                               |
| updated_at            | 2015-07-14T21:39:31.000000           |
+-----------------------+--------------------------------------+


I think this is probably your issue since you're trying to reuse the old vmlinuz/initrd with the new image.

Totally agree this needs documentation.

Comment 4 Steven Hardy 2015-07-17 15:50:31 UTC

@James: Aha!!  Thanks, I totally missed that.

I guess this could be considered a dupe of bz#1243829 then, as if we correctly handle reloading the image without creating the dupes, I probably wouldn't have got into this state.

Docs patch also sounds like a good plan tho.

Comment 5 Mike Burns 2015-07-17 17:08:30 UTC

This has impact on partners and needs to be resolved for any updates in an A1 timeframe

Comment 6 chris alfonso 2015-07-24 15:22:17 UTC

Marek, what are you doing with this bug given comment 4 notes a potential dupe.

Comment 7 Marek Aufart 2015-07-24 16:29:53 UTC

(In reply to chris alfonso from comment #6)
> Marek, what are you doing with this bug given comment 4 notes a potential
> dupe.

I am working on bz#1243829. It is duplicate from functional point of view. I considered if this bug would result to docs update, but propably no (it should not happend when duplicate one is fixed).

Going to close this one as duplicate of bz#1243829 and mark the duplicate one as blocker since this one has blocker ?.

*** This bug has been marked as a duplicate of bug 1243829 ***