Bug 1245572 - overcloud: HA deployment: Glance api.log flooded by warning messages.
Summary: overcloud: HA deployment: Glance api.log flooded by warning messages.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: Director
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 7.0 (Kilo)
Assignee: Giulio Fidente
QA Contact: Leonid Natapov
URL:
Whiteboard:
: 1245573 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-22 10:37 UTC by Leonid Natapov
Modified: 2023-02-22 23:02 UTC (History)
10 users (show)

Fixed In Version: openstack-puppet-modules-2015.1.8-14.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the HAProxy configuration contained the "option httpchk GET /" directive. Consequently, a "GET /" request was sent every two seconds, and the following message was repeated indefinitely in the /var/log/glance/api.log file: WARNING glance.api.middleware.version_negotiation [req-...] Unknown version. Returning version choices. This update removes the aforementioned directive. As a result, the warning message is no longer logged.
Clone Of:
Environment:
Last Closed: 2015-10-08 12:23:37 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 197904 0 None None None Never
Red Hat Product Errata RHBA-2015:1872 0 normal SHIPPED_LIVE openstack-puppet-modules bug fix and enhancement update 2015-10-08 16:14:03 UTC

Description Leonid Natapov 2015-07-22 10:37:20 UTC 
overcloud: HA deployment: Glance api.log flooded by warning messages.

2015-07-22 06:27:01.392 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:02.706 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:02.772 7643 WARNING glance.api.middleware.version_negotiation [req-96ae92f2-7761-4ae8-9b01-58a758fd3c94 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:03.395 7645 WARNING glance.api.middleware.version_negotiation [req-1b5817d5-7cb5-4bf2-9332-f1478bb814f1 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:04.710 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:04.776 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:05.400 7639 WARNING glance.api.middleware.version_negotiation [req-334037a8-867d-4618-9087-fa2a7a7adbfb c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:06.713 7645 WARNING glance.api.middleware.version_negotiation [req-1b5817d5-7cb5-4bf2-9332-f1478bb814f1 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:06.779 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:07.404 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:08.717 7646 WARNING glance.api.middleware.version_negotiation [req-a3f4fa4f-6109-4eeb-96e1-0edb7792793f 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:08.781 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:09.408 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:10.720 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:10.783 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:11.412 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:12.723 7648 WARNING glance.api.middleware.version_negotiation [req-dddb698b-d53f-40df-b566-a3f38cfc1a36 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:12.786 7639 WARNING glance.api.middleware.version_negotiation [req-334037a8-867d-4618-9087-fa2a7a7adbfb c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:13.415 7648 WARNING glance.api.middleware.version_negotiation [req-dddb698b-d53f-40df-b566-a3f38cfc1a36 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:14.727 7639 WARNING glance.api.middleware.version_negotiation [req-334037a8-867d-4618-9087-fa2a7a7adbfb c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:14.789 7639 WARNING glance.api.middleware.version_negotiation [req-334037a8-867d-4618-9087-fa2a7a7adbfb c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:15.419 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:16.728 7646 WARNING glance.api.middleware.version_negotiation [req-a3f4fa4f-6109-4eeb-96e1-0edb7792793f 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:16.793 7638 WARNING glance.api.middleware.version_negotiation [req-2ac03a68-b281-402f-813b-43ec85a3b659 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:17.422 7642 WARNING glance.api.middleware.version_negotiation [req-43cae65f-69ce-49cd-8650-18a72907ca91 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:18.732 7643 WARNING glance.api.middleware.version_negotiation [req-96ae92f2-7761-4ae8-9b01-58a758fd3c94 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:18.794 7646 WARNING glance.api.middleware.version_negotiation [req-a3f4fa4f-6109-4eeb-96e1-0edb7792793f 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:19.426 7650 WARNING glance.api.middleware.version_negotiation [req-51253d35-db19-41a4-b692-31c47fc74ae6 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:20.736 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:20.796 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:21.430 7638 WARNING glance.api.middleware.version_negotiation [req-2ac03a68-b281-402f-813b-43ec85a3b659 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:22.740 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:22.800 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:23.434 7644 WARNING glance.api.middleware.version_negotiation [req-d78011dc-144e-4280-b2ce-4a078d917e3a 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:24.744 7640 WARNING glance.api.middleware.version_negotiation [req-20755022-ed6c-4b36-b8bf-d1cd835d2ef4 adb88a8281ac4b489d87d48c2f2945df e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:24.802 7638 WARNING glance.api.middleware.version_negotiation [req-2ac03a68-b281-402f-813b-43ec85a3b659 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:25.438 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:26.748 7640 WARNING glance.api.middleware.version_negotiation [req-20755022-ed6c-4b36-b8bf-d1cd835d2ef4 adb88a8281ac4b489d87d48c2f2945df e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:26.806 7640 WARNING glance.api.middleware.version_negotiation [req-20755022-ed6c-4b36-b8bf-d1cd835d2ef4 adb88a8281ac4b489d87d48c2f2945df e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:27.442 7638 WARNING glance.api.middleware.version_negotiation [req-2ac03a68-b281-402f-813b-43ec85a3b659 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:28.752 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:28.807 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:29.446 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:30.756 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:30.810 7646 WARNING glance.api.middleware.version_negotiation [req-a3f4fa4f-6109-4eeb-96e1-0edb7792793f 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:31.450 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:32.760 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:32.814 7648 WARNING glance.api.middleware.version_negotiation [req-dddb698b-d53f-40df-b566-a3f38cfc1a36 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:33.454 7642 WARNING glance.api.middleware.version_negotiation [req-43cae65f-69ce-49cd-8650-18a72907ca91 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:34.764 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:34.817 7640 WARNING glance.api.middleware.version_negotiation [req-20755022-ed6c-4b36-b8bf-d1cd835d2ef4 adb88a8281ac4b489d87d48c2f2945df e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:35.458 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:36.768 7648 WARNING glance.api.middleware.version_negotiation [req-dddb698b-d53f-40df-b566-a3f38cfc1a36 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:36.819 7645 WARNING glance.api.middleware.version_negotiation [req-1b5817d5-7cb5-4bf2-9332-f1478bb814f1 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:37.462 7647 WARNING glance.api.middleware.version_negotiation [req-afe5cabe-f69d-468c-bf64-cac7a6d88ddd 4786398be33d4f39829c2247b6907ff7 e3506ea62f914156ad0dea576d672f46 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:38.823 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:40.827 7641 WARNING glance.api.middleware.version_negotiation [req-1ad11245-f640-4aa3-a557-0f5776e231ab c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:42.831 7648 WARNING glance.api.middleware.version_negotiation [req-dddb698b-d53f-40df-b566-a3f38cfc1a36 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
2015-07-22 06:27:44.835 7643 WARNING glance.api.middleware.version_negotiation [req-96ae92f2-7761-4ae8-9b01-58a758fd3c94 c725eb9cb9f247129d1242864100ec1a 6aaa5c89ac4b430d9c1a8ad33fbac932 - - -] Unknown version. Returning version choices.
(END)
Comment 3 Mike Burns 2015-07-22 10:56:53 UTC 
*** Bug 1245573 has been marked as a duplicate of this bug. ***
Comment 4 Flavio Percoco 2015-07-25 08:59:27 UTC 
Leonid,

Would you mind sharing what you did to get all these messages?

The message is logged when someone tries to call a glance endpoint with a non-valid version number. I don't see anything wrong with this log.

Please, provide - always - as much information as possible to understand the issue. This report, as it is, is not useful at all.

Who is calling Glance? Why is it being called with the wrong version number? How was glance configured? etc.
Comment 5 Leonid Natapov 2015-07-25 14:35:35 UTC 
glance configured with iscsi backend.
I have downloaded cirros images and created several VMs. That's all I did. I don't know why it's being called with the wrong number. Things seems to be working ok. Image was downloaded and I created several instances using that image. I just wanted to check something in glance log and when I opened it I saw this flood of messages.
Comment 6 Leonid Natapov 2015-07-26 17:14:34 UTC 
Happen with ceph as well. Flavio,I have an environment if you want to look at the logs. I will send you an email with the environment details.
Comment 7 Flavio Percoco 2015-07-31 04:27:27 UTC 
I looked into the box. Apparently there's a process doing periodic request's to Glance and that process is not passing the version as part of the URL.

What are you using for HA? Is any of these tools continuously pinging Glance?
Comment 8 Leonid Natapov 2015-08-02 04:14:23 UTC 
What do you mean by "what are you using for HA" ?. 
Pls,let's talk on IRC in real time. My nickname is lesik.
Comment 9 Flavio Percoco 2015-08-02 22:36:03 UTC 
I dug into this myself. You've haproxy installed and it's configured to run `GET` requests on glance-api to know if it's alive. The requests don't contain a version in hte URL, which is causing all these log lines. This is your haproxy config:

listen glance_api
  bind 192.0.2.2:9292 
  bind 192.0.2.3:9292 
  option httpchk GET /
  server 192.0.2.1 192.0.2.1:9292 check fall 5 inter 2000 rise 2

listen glance_registry
  bind 192.0.2.3:9191 
  server 192.0.2.1 192.0.2.1:9191 check fall 5 inter 2000 rise 2

I'd recommend using `/versions` instead of `/`
Comment 10 Flavio Percoco 2015-08-02 22:37:36 UTC 
Fabio, 

do you know who to ping to change `haproxy`'s official configuration and docs?
Comment 11 Fabio Massimo Di Nitto 2015-08-03 03:10:36 UTC 
(In reply to Flavio Percoco from comment #10)
> Fabio, 
> 
> do you know who to ping to change `haproxy`'s official configuration and
> docs?

I would talk to Ryan O´hara for the configuration (to check what is the best config) and Giulio Fidente to make the change in OSPd. For the doc, maybe Summer Long, but I am not entirely sure.
Comment 12 Flavio Percoco 2015-08-03 04:38:38 UTC 
Ryan, Giulio

Mind taking a look at this? Feel free to move the bug respectively.
Comment 13 Giulio Fidente 2015-08-03 09:49:48 UTC 
Probably due to the haproxy httpchk option currently configured to send a 'GET /' every 2 secs.

Should go away with: https://review.openstack.org/#/c/197904
Comment 14 Ryan O'Hara 2015-08-03 13:14:29 UTC 
(In reply to Flavio Percoco from comment #12)
> Ryan, Giulio
> 
> Mind taking a look at this? Feel free to move the bug respectively.

I don't think you want httpchk option, but that may have changed. I know there was an effort upstream to get httpchk working with the various API services, but I do not know the current state of this effort. It looks like Guilio is correct in comment #13.
Comment 18 Leonid Natapov 2015-09-21 17:11:39 UTC 
No flood in the api.log
openstack-puppet-modules-2015.1.8-17.el7ost.noarch
Comment 21 errata-xmlrpc 2015-10-08 12:23:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1872
Note You need to log in before you can comment on or make changes to this bug.