Bug 1102758
| Summary: | Glance's filesystem store should report when the same dir has been configured twice | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Tzach Shefi <tshefi> | ||||
| Component: | python-glance-store | Assignee: | Gorka Eguileor <geguileo> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Prasanth Anbalagan <panbalag> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 5.0 (RHEL 7) | CC: | apevec, dnavale, eglynn, fpercoco, hguemar, jruzicka, lhh, mlopes, panbalag, scohen, sgotliv, yeylon | ||||
| Target Milestone: | z2 | Keywords: | ZStream | ||||
| Target Release: | 7.0 (Kilo) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | python-glance-store-0.4.0-2.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
Previously, 'glance-store' exceptions were not being passed to the library user (Glance) on startup and were only being logged. Consequently, there was a possibility that configuration errors could be missed in the logs and unexpected behavior could occur (for example, being unable to add images).
With this update, exceptions are reported back to Image Service (glance), and startup will fail if there is an exception. As a result, if the same path-priority is used multiple times in a multistore NFS backend, you will receive a warning and it will continue as if it were only defined once. However, if the same path is defined multiple times with different priorities, then startup will fail.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-10-08 12:06:39 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Tzach, I don't think this is something we want to disclose to clients. It's a deployment error and it should be reported in Glance's logs instead of HTTP errors. Moving this bug to glance and updating the title. As this patch https://review.openstack.org/#/c/137416/ was merged, moving it to ON_QA for testing in rhos-7.0, Sean The patch we want is this one here: https://review.openstack.org/#/c/157013/ I'll move it to POST and re-assign to Gorka. When I add another line for filesystem_store_datadirs or remove the hash i get this error message. # Directory that the Filesystem backend store # writes image data to #filesystem_store_datadir=/var/lib/glance/images/ # A list of directories where image data can be stored. # This option may be specified multiple times for specifying multiple store # directories. Either one of filesystem_store_datadirs or # filesystem_store_datadir option is required. A priority number may be given # after each directory entry, separated by a ":". # When adding an image, the highest priority directory will be selected, unless # there is not enough space available in cases where the image size is already # known. If no priority is given, it is assumed to be zero and the directory # will be considered for selection last. If multiple directories have the same # priority, then the one with the most free space available is selected. # If same store is specified multiple times then BadStoreConfiguration # exception will be raised. filesystem_store_datadirs=/var/lib/glance/images/:1 filesystem_store_datadirs=/var/lib/glance/images/:1 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils Traceback (most recent call last): 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils context=req.context) 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 339, in store_add_to_backend 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils context=context) 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 224, in op_checker 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils raise op_exec_map[op](**kwargs) 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils StoreAddDisabled: Configuration for store failed. Adding images to this store is disabled. 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils That's because the patch from https://review.openstack.org/#/c/157013/ is not in place. I see that our glance_store's last patch is from 03-17-2015 and my patch didn't merge until 04-14-2015 so I guess it will be in on the next synchronization. (In reply to nlevinki from comment #6) > When I add another line for filesystem_store_datadirs or remove the hash i > get this error message. > > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils Traceback > (most recent call last): > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils File > "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 113, > in upload_data_to_store > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils > context=req.context) > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils File > "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 339, in > store_add_to_backend > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils > context=context) > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils File > "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 224, > in op_checker > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils raise > op_exec_map[op](**kwargs) > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils > StoreAddDisabled: Configuration for store failed. Adding images to this > store is disabled. > 2015-07-21 16:35:55.429 17576 TRACE glance.api.v1.upload_utils This patch needs the stable/kilo backport. Upstream backport still pending for approval Backport landed, this is ready to be packaged! Verified as follows - confguring same directory twice does display warning message in glance/api logs when same priority is used and error when different priority is used. Refer logs below for detailed steps. Version ======== [root@rhos-compute-node-02 glance]# yum list installed | grep openstack-glance openstack-glance.noarch 2015.1.1-1.el7ost @rhelosp-7.0-puddle [root@rhos-compute-node-02 glance]# yum list installed | grep python-glance python-glance.noarch 2015.1.1-1.el7ost @rhelosp-7.0-puddle python-glance-store.noarch 0.4.0-2.el7ost @rhelosp-7.0-puddle python-glanceclient.noarch 1:0.17.0-2.el7ost @rhelosp-7.0-puddle Logs ===== ******************************************** SAME DIRECTORY SAME PRIORITY ******************************************** [root@rhos-compute-node-02 ~(keystone_admin)]# grep "filesystem_store_datadirs" /etc/glance/glance-api.conf # directories. Either one of filesystem_store_datadirs or filesystem_store_datadirs=/etc/glance/nfsimages/:1 filesystem_store_datadirs=/etc/glance/nfsimages/:1 [root@rhos-compute-node-02 ~(keystone_admin)]# /var/log/glance/api.conf 2015-09-15 18:02:49.118 9049 WARNING glance_store._drivers.filesystem [-] Directory /etc/glance/nfsimages/ specified multiple times in filesystem_store_datadirs option of filesystem configuration 2015-09-15 18:02:49.488 9049 INFO glance.common.wsgi [-] Starting 12 workers 2015-09-15 18:02:49.491 9049 INFO glance.common.wsgi [-] Started child 9077 2015-09-15 18:02:49.491 9077 INFO eventlet.wsgi.server [-] (9077) wsgi starting up on http://0.0.0.0:9292/ [root@rhos-compute-node-02 ~(keystone_admin)]# [root@rhos-compute-node-02 ~(keystone_admin)]# glance image-create --name image2 --disk-format qcow2 --container-format bare --file fedora.qcow2 --is-public true +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | d41d8cd98f00b204e9800998ecf8427e | | container_format | bare | | created_at | 2015-09-15T15:04:18.000000 | | deleted | False | | deleted_at | None | | disk_format | qcow2 | | id | 87d52184-1136-4484-b45d-7710cf8c1103 | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | image2 | | owner | 9c7a0ee8f8b545bdb19191ff8584f58c | | protected | False | | size | 0 | | status | active | | updated_at | 2015-09-15T15:04:20.000000 | | virtual_size | None | +------------------+--------------------------------------+ [root@rhos-compute-node-02 ~(keystone_admin)]# glance image-list +--------------------------------------+--------+-------------+------------------+----------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+--------+-------------+------------------+----------+--------+ | e7ada0ba-49be-4bfb-9271-81173c2b5c8c | cirros | qcow2 | bare | 13200896 | active | | 004d4f4c-5723-49d8-ae77-1768c6f7139b | image1 | qcow2 | bare | | active | | 87d52184-1136-4484-b45d-7710cf8c1103 | image2 | qcow2 | bare | | active | +--------------------------------------+--------+-------------+------------------+----------+--------+ [root@rhos-compute-node-02 ~(keystone_admin)]# ********************************** SAME DIRECTORY DIFFERENT PRIORITY ********************************** [root@rhos-compute-node-02 ~(keystone_admin)]# [root@rhos-compute-node-02 ~(keystone_admin)]# grep "filesystem_store_datadirs" /etc/glance/glance-api.conf # directories. Either one of filesystem_store_datadirs or filesystem_store_datadirs=/etc/glance/nfsimages/:1 filesystem_store_datadirs=/etc/glance/nfsimages/:2 [root@rhos-compute-node-02 ~(keystone_admin)]# 2015-09-15 18:08:48.190 9704 INFO glance.common.wsgi [-] Child 9704 exiting normally 2015-09-15 18:08:49.360 9711 ERROR glance_store._drivers.filesystem [-] Directory /etc/glance/nfsimages/ specified multiple times in filesystem_store_datadirs option of filesystem configuration 2015-09-15 18:08:49.360 9711 TRACE glance_store._drivers.filesystem None 2015-09-15 18:08:49.360 9711 TRACE glance_store._drivers.filesystem 2015-09-15 18:08:49.361 9711 WARNING glance_store.driver [-] Failed to configure store correctly: Store filesystem could not be configured correctly. Reason: Directory /etc/glance/nfsimages/ specified multiple times in filesystem_store_datadirs option of filesystem configuration Disabling add method. 2015-09-15 18:08:49.361 9711 INFO glance.common.wsgi [-] Starting 12 workers [root@rhos-compute-node-02 ~(keystone_admin)]# [root@rhos-compute-node-02 ~(keystone_admin)]# glance image-create --name image3 --disk-format qcow2 --container-format bare --file fedora.qcow2 --is-public true 410 Gone: Error in store configuration. Adding images to store is disabled. (HTTP N/A) [root@rhos-compute-node-02 ~(keystone_admin)]# 2015-09-15 18:11:13.475 9745 ERROR glance.api.v1.upload_utils [req-641b5c43-b12b-4595-90fa-6470de06d6cb 825d4439c2db446b9149ae0fdd4ffac6 9c7a0ee8f8b545bdb19191ff8584f58c - - -] Error in store configuration. Adding images to store is disabled. 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils Traceback (most recent call last): 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils context=req.context) 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 339, in store_add_to_backend 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils context=context) 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 224, in op_checker 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils raise op_exec_map[op](**kwargs) 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils StoreAddDisabled: Configuration for store failed. Adding images to this store is disabled. 2015-09-15 18:11:13.475 9745 TRACE glance.api.v1.upload_utils Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1869 |
Created attachment 900355 [details] Glance api.log Description of problem: When configuring Glance for multiple store NFS backend, if "by mistake" two same folders are used, image upload fails as it should, but cli error is none informative. glance-api.conf -> filesystem_store_datadirs=/mnt/nfs1/images/:1 filesystem_store_datadirs=/mnt/nfs1/images/:1 [root@dhcp160-213 ~(keystone_admin)]# glance image-create --name flotz --disk-format qcow2 --container-format bare --file c6-x86_64-20130910-1.qcow2 Request returned failure status. 500 Internal Server Error Failed to upload image 75a41998-b0ce-4be1-b3c8-d1618362f4e8 (HTTP 500) Version-Release number of selected component (if applicable): RHEL7 python-glanceclient-0.12.0-1.el7ost.noarch openstack-glance-2014.1-2.el7ost.noarch python-glance-2014.1-2.el7ost.noarch How reproducible: Every time Steps to Reproduce: 1. Configure Glance to use multi store NFS back-end, give same NFS shares twice. 2. Try to upload image Actual results: Request returned failure status. 500 Internal Server Error Failed to upload image 75a41998-b0ce-4be1-b3c8-d1618362f4e8 (HTTP 500) Expected results: A more detailed informative error on CLI would be helpful, maybe similar to api.log "Reason: Directory /mnt/nfs1/images/ specified multiple times in filesystem_store_datadirs option of filesystem configuration Disabling add method."