| Summary: | error getting overcloud and discovery images | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Steve Reichard <sreichar> | ||||
| Component: | openstack-glance | Assignee: | Pranali Deore <pdeore> | ||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Mike Abrams <mabrams> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 9.0 (Mitaka) | CC: | cschwede, cyril, dprince, ebarrera, egafford, eglynn, emacchi, fpercoco, gbarros, gdrapeau, ikedajnk, jschluet, kawamurayus, pdeore, pgrist, scohen, srevivo, ssigwald, tshefi | ||||
| Target Milestone: | --- | Keywords: | Triaged, ZStream | ||||
| Target Release: | 9.0 (Mitaka) | Flags: | tshefi:
automate_bug-
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-08-08 13:53:40 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
|
Description
Steve Reichard
2016-08-30 22:44:58 UTC
Wonder if this is a race condition between the image creation and the boot attempt. Just a guess for now. I'll ping some more knowledgeable folks on tripleo to help with this. I never saw this problem before but indeed it might be a race somewhere in the deployment. I'm also wondering which tenant do you use to list the images. Maybe the tripleo tenant can't see it? Is the image public? I did not even try booting yet, since there are no master_images. Why I try "opestack baremetal configure host" I see glance errors, when I do it using glance cli, I have errors. Doesn't sound like a race condition involving boot. Again I have this repeatable in an internal RH conifg, let me know if you want access. My intrepetation of what I see in the logs is the failed attempt assume the name, while the successfull cli does a search to get the name: [root@ks-inst2 glance]$ source ~/stackrc [root@ks-inst2 glance]$ glance image-list +--------------------------------------+------------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+------------------------+-------------+------------------+------------+--------+ | 61b09236-be9f-4626-9040-6093a1bf53ed | bm-deploy-kernel | aki | aki | 5157696 | active | | a6ce1373-91e5-4c8e-8ff2-65adf8f10be7 | bm-deploy-ramdisk | ari | ari | 447551200 | active | | e5d49231-5d40-4090-b14c-f1c4d5220248 | overcloud-full | qcow2 | bare | 1239089152 | active | | a3cc5a73-e510-4201-87eb-77e1b214386e | overcloud-full-initrd | ari | ari | 42613432 | active | | 5519f6a6-d7d6-4b8d-a272-cca82ea5aca6 | overcloud-full-vmlinuz | aki | aki | 5157696 | active | +--------------------------------------+------------------------+-------------+------------------+------------+--------+ [root@ks-inst2 glance]$ glance image-show bm-deploy-ramdisk +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | f524e098d6f829ac785246a2f314e5ed | | container_format | ari | | created_at | 2016-08-30T22:10:28.000000 | | deleted | False | | disk_format | ari | | id | a6ce1373-91e5-4c8e-8ff2-65adf8f10be7 | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | bm-deploy-ramdisk | | owner | d66824c3fa244369900a6063fdb0e4ba | | protected | False | | size | 447551200 | | status | active | | updated_at | 2016-08-30T22:10:38.000000 | +------------------+--------------------------------------+ [root@ks-inst2 glance]$ grep -e f524e098d6f829ac785246a2f314e5ed -e bm-deploy-ramdisk api.log 2016-08-30 18:10:28.500 550 DEBUG glance.api.middleware.version_negotiation [req-d7a16caf-65df-493f-a5e5-2766748df6de 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Determining version of request: HEAD /v1/images/bm-deploy-ramdisk Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:46 2016-08-30 18:10:28.503 550 DEBUG glance.api.middleware.version_negotiation [req-d7a16caf-65df-493f-a5e5-2766748df6de 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] new path /v1/images/bm-deploy-ramdisk process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:71 2016-08-30 18:10:28.511 550 DEBUG glance.common.client [req-ff1b6789-2d66-42f7-965e-a0a219e4489f 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Constructed URL: http://192.168.200.1:9191/images/bm-deploy-ramdisk _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398 2016-08-30 18:10:28.563 550 ERROR glance.registry.client.v1.client [req-ff1b6789-2d66-42f7-965e-a0a219e4489f 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Registry client request GET /images/bm-deploy-ramdisk raised NotFound 2016-08-30 18:10:28.566 550 DEBUG glance.api.v1.controller [req-ff1b6789-2d66-42f7-965e-a0a219e4489f 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Image with identifier bm-deploy-ramdisk not found get_image_meta_or_404 /usr/lib/python2.7/site-packages/glance/api/v1/controller.py:43 2016-08-30 18:10:28.569 550 INFO eventlet.wsgi.server [req-ff1b6789-2d66-42f7-965e-a0a219e4489f 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] 192.168.200.1 - - [30/Aug/2016 18:10:28] "HEAD /v1/images/bm-deploy-ramdisk HTTP/1.1" 404 213 0.070071 2016-08-30 18:10:38.364 550 DEBUG swiftclient [req-d28d40a5-9e4b-47b1-a551-f7b2a1f480fc 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] RESP HEADERS: {u'Content-Length': u'0', u'Last-Modified': u'Tue, 30 Aug 2016 22:10:30 GMT', u'Etag': u'f524e098d6f829ac785246a2f314e5ed', u'X-Trans-Id': u'tx20a893afd8bb4949b14c6-0057c60455', u'Date': u'Tue, 30 Aug 2016 22:10:38 GMT', u'Content-Type': u'text/html; charset=UTF-8'} http_log /usr/lib/python2.7/site-packages/swiftclient/client.py:166 2016-08-30 18:10:38.366 550 DEBUG glance.api.v1.upload_utils [req-d28d40a5-9e4b-47b1-a551-f7b2a1f480fc 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Updating image a6ce1373-91e5-4c8e-8ff2-65adf8f10be7 data. Checksum set to f524e098d6f829ac785246a2f314e5ed, size set to 447551200 upload_data_to_store /usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py:155 2016-08-30 18:11:09.324 550 DEBUG glance.api.middleware.version_negotiation [req-6f12bf64-1a7d-4532-8482-9bfef1e0293a 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Determining version of request: HEAD /v1/images/bm-deploy-ramdisk Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:46 2016-08-30 18:11:09.327 550 DEBUG glance.api.middleware.version_negotiation [req-6f12bf64-1a7d-4532-8482-9bfef1e0293a 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] new path /v1/images/bm-deploy-ramdisk process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:71 2016-08-30 18:11:09.333 550 DEBUG glance.common.client [req-e9aa44e8-1520-47fc-8bbe-30f4465684de 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Constructed URL: http://192.168.200.1:9191/images/bm-deploy-ramdisk _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398 2016-08-30 18:11:09.372 550 ERROR glance.registry.client.v1.client [req-e9aa44e8-1520-47fc-8bbe-30f4465684de 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Registry client request GET /images/bm-deploy-ramdisk raised NotFound 2016-08-30 18:11:09.375 550 DEBUG glance.api.v1.controller [req-e9aa44e8-1520-47fc-8bbe-30f4465684de 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Image with identifier bm-deploy-ramdisk not found get_image_meta_or_404 /usr/lib/python2.7/site-packages/glance/api/v1/controller.py:43 2016-08-30 18:11:09.377 550 INFO eventlet.wsgi.server [req-e9aa44e8-1520-47fc-8bbe-30f4465684de 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] 192.168.200.1 - - [30/Aug/2016 18:11:09] "HEAD /v1/images/bm-deploy-ramdisk HTTP/1.1" 404 213 0.053548 2016-08-30 18:11:09.486 550 DEBUG glance.api.middleware.version_negotiation [req-d9984d2e-ec96-474c-95e4-616f01329491 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Determining version of request: HEAD /v1/images/bm-deploy-ramdisk Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:46 2016-08-30 18:11:09.489 550 DEBUG glance.api.middleware.version_negotiation [req-d9984d2e-ec96-474c-95e4-616f01329491 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] new path /v1/images/bm-deploy-ramdisk process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:71 2016-08-30 18:11:09.494 550 DEBUG glance.common.client [req-64b691d4-678c-493c-8de7-114f76cfbd84 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Constructed URL: http://192.168.200.1:9191/images/bm-deploy-ramdisk _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398 2016-08-30 18:11:09.530 550 ERROR glance.registry.client.v1.client [req-64b691d4-678c-493c-8de7-114f76cfbd84 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Registry client request GET /images/bm-deploy-ramdisk raised NotFound 2016-08-30 18:11:09.533 550 DEBUG glance.api.v1.controller [req-64b691d4-678c-493c-8de7-114f76cfbd84 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Image with identifier bm-deploy-ramdisk not found get_image_meta_or_404 /usr/lib/python2.7/site-packages/glance/api/v1/controller.py:43 2016-08-30 18:11:09.535 550 INFO eventlet.wsgi.server [req-64b691d4-678c-493c-8de7-114f76cfbd84 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] 192.168.200.1 - - [30/Aug/2016 18:11:09] "HEAD /v1/images/bm-deploy-ramdisk HTTP/1.1" 404 213 0.049765 2016-08-30 18:12:29.511 549 DEBUG glance.api.middleware.version_negotiation [req-14c01a17-8f92-4249-ab58-b3e270c1cba4 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Determining version of request: HEAD /v1/images/bm-deploy-ramdisk Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:46 2016-08-30 18:12:29.514 549 DEBUG glance.api.middleware.version_negotiation [req-14c01a17-8f92-4249-ab58-b3e270c1cba4 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] new path /v1/images/bm-deploy-ramdisk process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:71 2016-08-30 18:12:29.520 549 DEBUG glance.common.client [req-de742feb-7166-4c03-a75c-d935fa749071 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Constructed URL: http://192.168.200.1:9191/images/bm-deploy-ramdisk _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398 2016-08-30 18:12:29.582 549 ERROR glance.registry.client.v1.client [req-de742feb-7166-4c03-a75c-d935fa749071 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Registry client request GET /images/bm-deploy-ramdisk raised NotFound 2016-08-30 18:12:29.585 549 DEBUG glance.api.v1.controller [req-de742feb-7166-4c03-a75c-d935fa749071 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Image with identifier bm-deploy-ramdisk not found get_image_meta_or_404 /usr/lib/python2.7/site-packages/glance/api/v1/controller.py:43 2016-08-30 18:12:29.588 549 INFO eventlet.wsgi.server [req-de742feb-7166-4c03-a75c-d935fa749071 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] 192.168.200.1 - - [30/Aug/2016 18:12:29] "HEAD /v1/images/bm-deploy-ramdisk HTTP/1.1" 404 213 0.077689 2016-08-30 18:41:19.916 549 DEBUG glance.common.client [req-d8646d04-58fc-437e-b734-d8695c295271 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Constructed URL: http://192.168.200.1:9191/images/detail?limit=20&name=bm-deploy-ramdisk _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398 2016-08-30 18:41:20.066 549 INFO eventlet.wsgi.server [req-d8646d04-58fc-437e-b734-d8695c295271 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] 192.168.200.1 - - [30/Aug/2016 18:41:20] "GET /v1/images/detail?limit=20&name=bm-deploy-ramdisk HTTP/1.1" 200 718 0.246609 2016-08-30 18:41:22.937 549 DEBUG swiftclient [req-f98ed898-5751-48dd-b73e-02d475a20085 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] RESP HEADERS: {u'Content-Length': u'447551200', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Tue, 30 Aug 2016 22:10:30 GMT', u'Etag': u'f524e098d6f829ac785246a2f314e5ed', u'X-Timestamp': u'1472595029.50790', u'X-Trans-Id': u'tx236dd4b53e0346579a4c1-0057c60b90', u'Date': u'Tue, 30 Aug 2016 22:41:22 GMT', u'Content-Type': u'application/octet-stream'} http_log /usr/lib/python2.7/site-packages/swiftclient/client.py:166 2016-09-01 08:40:11.192 552 DEBUG glance.common.client [req-ff0c6ead-fe35-4c51-a856-373dbe1d2ce6 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] Constructed URL: http://192.168.200.1:9191/images/detail?limit=20&name=bm-deploy-ramdisk _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:398 2016-09-01 08:40:11.475 552 INFO eventlet.wsgi.server [req-ff0c6ead-fe35-4c51-a856-373dbe1d2ce6 138a9c0618b94c8491360b9c45c84e41 d66824c3fa244369900a6063fdb0e4ba - - -] 192.168.200.1 - - [01/Sep/2016 08:40:11] "GET /v1/images/detail?limit=20&name=bm-deploy-ramdisk HTTP/1.1" 200 718 0.481127 [root@ks-inst2 glance]$ Created attachment 1197216 [details]
console capture
I not am seeing this on another configuration that I just attempted to install with OSP9. If I ignore the glance error and attempt to introspect I see the errors as seen in the attached console capture. I re-installed OSP8 on the second reproducer cluster, just to verify the config. No problem. I then switched back to OSP9 and am not have the PXE provisioning problem. I will retry the first reproducer also, which means it state is going to change. I think with OSP9 python-tripleoclient switched to use /httpboot instead of /tftpboot/ for the introspection setup on the actual undercloud. I believe it is not the 'openstack overcloud image upload' command which actually now configures this root directory. Could it be the issue here was that 'openstack overcloud image upload' had not been executed yet? And as such the /httpboot directory wasn't created yet? This really looks like a triple-o issue rather than an openstack-glance one. Maybe this should be retargeted. @Eduard: what do you think? @Pranali: can you please have a look at this one and check if we can fix this in OOO? Thx! Closed as insufficient data, nothing for QE to test/automate at the moment. Also clearing out old needinfo to tidy up things. |