Bug 1539703

Summary:	By rebuilding twice with the same "forbidden" image one can circumvent scheduler rebuild restrictions
Product:	Red Hat OpenStack	Reporter:	Artom Lifshitz <alifshit>
Component:	openstack-nova	Assignee:	Artom Lifshitz <alifshit>
Status:	CLOSED ERRATA	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	12.0 (Pike)	CC:	alifshit, berrange, dasmith, eglynn, jamsmith, kchamart, lyarwood, mbooth, mriedem, sbauza, sferdjao, sgordon, slinaber, srevivo, vromanso
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-16.1.4-2.el7ost	Doc Type:	Bug Fix
Doc Text:	This update prevents an unintended bypass of the schedule filters that could occur after the scheduler refused a rebuild request sent by nova. If a user rebuilds an instance with a new image, the change from old image to new image causes nova to send the rebuild request to the scheduler to make sure it is allowed according to the scheduler filters. Prior to this update, if the scheduler refused the request, the instance's image reference was not rolled back to the original image. This caused an inconsistency between the original image actually in use by the instance, and the new image reference saved in the database. As a result, a second rebuild request with the same new image would bypass the scheduler and be allowed, because the image in the rebuild request was the same as the instance's image in the database, even though the real image in use by the instance was the old original image. This bypass of scheduler filters was considered a security flaw. As of this update, when a rebuild request is refused by the scheduler,the image reference is rolled back to the original. If another rebuild request is made with the same new image, it is correctly identified as being different from the instance's current image and the request is send to the scheduler.	Story Points:	---
Clone Of:
Clones:	1540368 (view as bug list)		Environment:
Last Closed:	2018-08-20 12:55:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1540368, 1540369

Description Artom Lifshitz 2018-01-29 13:39:39 UTC

Description of problem:

Since bz 1508689 (CVE-2017-16239), we call to the scheduler when doing a rebuild with a new image. If the scheduler refuses a rebuild because a filter forbids the new image on the instance's host (for example, IsolatedHostsFilter), at first there was no indication of this in the API (bz 1536150). Currently, with the fix for bz 1536150 merged, the instance goes to ERROR to indicate the refused rebuild. However, by rebuilding again with the same "forbidden" image it is possible to circumvent scheduler restrictions.

Version-Release number of selected component (if applicable):

OSP 10 through 12

How reproducible
================

Always.

Steps to Reproduce
==================

1. Configure IsolatedHostsFilter:

   [filter_scheduler]
   enabled_filters = [...],IsolatedHostsFilter
   isolated_images = 41d3e5ca-14cf-436c-9413-4826b5c8bdb1
   isolated_hosts = ubuntu
   restrict_isolated_hosts_to_isolated_images = true

2. Have two images, one isolated and one not:

   $ openstack image list

     8d0581a5-ed9d-4b98-a766-a41efbc99929 | centos | active
     41d3e5ca-14cf-436c-9413-4826b5c8bdb1 | cirros-0.3.5-x86_64-disk | active

     cirros is the isolated one

3. Have only one hypervisor (the isolated one):

   $ openstack hypervisor list

     ubuntu | QEMU | 192.168.100.194 | up

4. To confirm, boot a centos (non-isolated) image,
   expecting it to be refused by the scheduler:

   $ openstack server create \
     --image 8d0581a5-ed9d-4b98-a766-a41efbc99929 \
     --flavor \
     m1.nano centos-test-expect-fail

   $ openstack server list

     centos-test-expect-fail | ERROR | | centos | m1.nano

5. Boot a cirros (isolated) image:

   $ openstack server create \
     --image 41d3e5ca-14cf-436c-9413-4826b5c8bdb1 \
     --flavor m1.nano \
     cirros-test-expect-success

   $ openstack server list

     cirros-test-expect-success | ACTIVE | [...] | cirros-0.3.5-x86_64-disk | m1.nano

6. Rebuild the cirros instance with centos:

   $ nova --debug rebuild cirros-test-expect-success centos

     DEBUG (session:722) POST call to compute for
     http://192.168.100.194/compute/v2.1/servers/d9d98bf7-623e-4587-b82c-06f36abf59cb/action
     used request id req-c234346a-6e05-47cf-a0cd-45f89d11e15d

7. Observe the rebuild being refused in the conductor:

   WARNING nova.conductor.manager
   [None req-c234346a-6e05-47cf-a0cd-45f89d11e15d demo admin]
   [instance: d9d98bf7-623e-4587-b82c-06f36abf59cb]
   No valid host found for rebuild: NoValidHost_Remote:
   No valid host was found. There are not enough hosts available.

8. Observe the instance going to ERROR,
   but still showing the new centos image :

   $ nova show cirros-test-expect-success

     [...]
     status | ERROR
     image  | centos (8d0581a5-ed9d-4b98-a766-a41efbc99929)
     [...]

9. Rebuild again with the same centos image:

   $ nova rebuild cirros-test-expect-success centos

10. The rebuild goes through.

Expected results
================

The rebuild keeps being refused.

Additional info
===============

We only set the scheduler hint if the new imageref is different from the old imageref [1], this is a legit bug.

[1] https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3055-L3073

Comment 1 Matt Riedemann 2018-01-29 16:10:41 UTC

Here is the fix:

https://review.openstack.org/#/c/538961/

There is a patch with a recreate functional test underneath that in the series.

Comment 2 Matthew Booth 2018-05-15 13:49:19 UTC

Looks like the stable backport merged.

Comment 11 errata-xmlrpc 2018-08-20 12:55:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2332