Bug 1665239 - [rhos-prio] StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched when creating a lot of security group rules through heat
Summary: [rhos-prio] StaleDataError: UPDATE statement on table 'standardattributes' ex...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: async
: 10.0 (Newton)
Assignee: Brian Haley
QA Contact: Federico Ressi
URL:
Whiteboard:
Depends On: 1671406
Blocks: 1667257 1776950
TreeView+ depends on / blocked
 
Reported: 2019-01-10 18:37 UTC by David Vallee Delisle
Modified: 2023-10-06 18:04 UTC (History)
11 users (show)

Fixed In Version: openstack-neutron-9.4.1-37.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1667257 1776950 (view as bug list)
Environment:
Last Closed: 2019-04-30 16:58:15 UTC
Target Upstream Version:
Embargoed:
bhaley: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 635671 0 'None' MERGED Run revision bump operations en masse 2020-11-10 10:36:16 UTC
Red Hat Issue Tracker OSP-17409 0 None None None 2022-07-09 12:59:03 UTC
Red Hat Product Errata RHSA-2019:0916 0 None None None 2019-04-30 16:58:23 UTC

Comment 1 David Vallee Delisle 2019-01-10 18:41:33 UTC 
Created attachment 1519890 [details]
Following the request id in the logs with debug log level enabled in my lab
Comment 2 David Vallee Delisle 2019-01-10 19:03:14 UTC 
Affected components:
openstack-neutron-ml2-9.1.0-8.el7ost.noarch
openstack-neutron-openvswitch-9.1.0-8.el7ost.noarch
openstack-neutron-common-9.1.0-8.el7ost.noarch
puppet-neutron-9.4.2-1.el7ost.noarch
python-neutronclient-6.0.0-2.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
openstack-neutron-9.1.0-8.el7ost.noarch
python-neutron-9.1.0-8.el7ost.noarch
Comment 3 Brian Haley 2019-01-11 21:38:10 UTC 
I've been trying to reproduce this in my OSP 10 setup to get more info on the timing aspects, but have not so far.  Maybe I can get access to your environment or maybe you can answer a question.

Do you see each API call taking a longer and longer time?  The POST and error lines in the sosreport server.log don't show much of a difference, but I'm figuring there must be since retries were triggered.

You also mentioned tweaking a few timeouts.  Can you tell me what you changed and share any scripts that might help?  Thanks.

I'm trying to tell if this is related to https://bugs.launchpad.net/neutron/+bug/1810563 and getting some more data would be helpful.
Comment 4 David Vallee Delisle 2019-01-11 23:22:01 UTC 
Brian,

This is interesting. My lab is unfortunately in my lan and not publicly accessible.

The timeout I tweaked was haproxy client and server. I brought it up from 2m to 5m.

We do see a buildup in the POST [1] in the customer's logs. When we reach the 54 second mark, we start seeing the 500s.

[2] I have a similar pattern in my lab, except that the mark is at ~140s. I'm not 100% sure that I had the exact same problem as the customer because I had DbDeadLock as well as the StaleData, and I couldn't find any dead locks in the customer's logs. 

I've attached to the case a tarball of /var/log/neutron from my lab if it helps. Let me know if you need any other kind of logs.

[1]
~~~
2019-01-09 18:11:08.461 659082 INFO neutron.wsgi [req-db9092f4-3660-429d-a44a-3836c5e3dd83 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:11:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 692 0.211599
2019-01-09 18:12:39.416 659053 INFO neutron.wsgi [req-570d7562-708c-44ef-b6c9-a5d24b0dfa50 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:39] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 0.221999
2019-01-09 18:12:40.044 659067 INFO neutron.wsgi [req-3b497e7c-58a6-4458-861b-3fe9fb29c37c 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 0.208155
2019-01-09 18:12:40.485 659078 INFO neutron.wsgi [req-32a05ffc-244e-4689-a697-721a5f5f3e49 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 0.243344
2019-01-09 18:12:40.760 659064 INFO neutron.wsgi [req-9ce264d5-60e8-43e5-8221-e0e1236233b8 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.622608
2019-01-09 18:12:40.859 659052 INFO neutron.wsgi [req-c1810ef6-f6b0-47a7-97a8-a4701c9c851c 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.256596
2019-01-09 18:12:40.923 659034 INFO neutron.wsgi [req-ede244da-7239-474a-906e-31539002f5da 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 0.275233
2019-01-09 18:12:41.122 659044 INFO neutron.wsgi [req-8cea63d5-13eb-4878-b3f8-0d5ccf14ba22 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:41] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.283887
2019-01-09 18:12:41.196 659050 INFO neutron.wsgi [req-256d3c97-a447-42cd-8360-68b9199cf79b 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:41] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 0.477232
2019-01-09 18:12:48.210 659075 INFO neutron.wsgi [req-2cc1299c-0e44-45d0-b6c8-aa6126b26518 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:48] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 6.822488
2019-01-09 18:12:52.426 659086 INFO neutron.wsgi [req-5efedcf5-e5b5-4978-83f4-c8b298dd7d93 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:52] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 8.959367
2019-01-09 18:12:55.188 659086 INFO neutron.wsgi [req-37089c6a-7941-4c4b-80ab-bbf288b1c359 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:55] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 12.974644
2019-01-09 18:12:56.378 659060 INFO neutron.wsgi [req-c223290d-77f2-4544-8dd9-454b07a441be 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:56] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 13.237137
2019-01-09 18:12:56.784 659072 INFO neutron.wsgi [req-d458eda0-775e-484b-a058-357f41ba542e 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:56] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 15.977544
2019-01-09 18:12:56.935 659060 INFO neutron.wsgi [req-3de75d55-d89b-4a59-b6e1-2c9a644dc28f 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:56] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 15.026253
2019-01-09 18:12:57.096 659035 INFO neutron.wsgi [req-373bd6d8-5a8f-468f-b9f4-694476b49477 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:57] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 15.335712
2019-01-09 18:12:57.504 659061 INFO neutron.wsgi [req-f8d1abc9-f4c7-49e3-9c69-9a69c40f0100 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:57] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 15.291020
2019-01-09 18:12:58.237 659044 INFO neutron.wsgi [req-c00753f2-07f1-44eb-aca0-79864b908d5e 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:58] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 14.575042
2019-01-09 18:12:58.587 659053 INFO neutron.wsgi [req-7ade5a5b-5c10-4f2f-a760-e47cd727b164 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:58] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 17.350373
2019-01-09 18:12:59.171 659076 INFO neutron.wsgi [req-8ac28776-2d05-4882-85a1-a0a6d6330c39 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:59] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 17.904911
2019-01-09 18:12:59.931 659076 INFO neutron.wsgi [req-089bfa16-a3f2-44e4-927f-d946a016d9ab 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:12:59] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 18.240242
2019-01-09 18:13:00.553 659053 INFO neutron.wsgi [req-e86395b0-fa17-434a-8f22-06a03651990c 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:00] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 18.606396
2019-01-09 18:13:00.614 659076 INFO neutron.wsgi [req-ceb7791b-a3fc-4ace-82f5-7c0f9c911fa1 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:00] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 18.320560
2019-01-09 18:13:02.238 659064 INFO neutron.wsgi [req-2e62fafa-4575-4027-b5e9-1acf519b2a65 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:02] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 19.036767
2019-01-09 18:13:02.704 659074 INFO neutron.wsgi [req-f727760e-578d-4b5a-bc53-6a3322a67e81 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:02] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 19.415940
2019-01-09 18:13:02.918 659042 INFO neutron.wsgi [req-8b76a08f-c968-4b42-a027-67886e82a13e 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:02] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 22.411786
2019-01-09 18:13:03.099 659064 INFO neutron.wsgi [req-c2c4a2a5-eb3a-43a5-ac28-f78859f9dfaa 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:03] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 22.282610
2019-01-09 18:13:05.242 659086 INFO neutron.wsgi [req-7e39e234-c69b-4397-8719-f49ce0950f84 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:05] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 24.062714
2019-01-09 18:13:06.593 659047 INFO neutron.wsgi [req-2885085c-3235-4442-8111-6902d0f99187 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:06] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 25.191591
2019-01-09 18:13:07.260 659064 INFO neutron.wsgi [req-5a88ff18-cf34-4245-ae63-5d75ba5322e9 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:07] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 25.044951
2019-01-09 18:13:08.180 659033 INFO neutron.wsgi [req-a2f1fe14-0099-4dcb-9ddf-72ada5652cf1 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 26.000256
2019-01-09 18:13:08.315 659050 INFO neutron.wsgi [req-07a0379e-ad86-4145-926c-4ae500fdbac1 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 25.536324
2019-01-09 18:13:08.378 659044 INFO neutron.wsgi [req-c813269a-c727-4269-b0eb-5c0bb82f887b 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 25.599472
2019-01-09 18:13:08.803 659038 INFO neutron.wsgi [req-d375f25d-2cc3-4956-bb33-f2735a39ed51 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 25.858612
2019-01-09 18:13:08.861 659040 INFO neutron.wsgi [req-50ee7596-ede6-4315-b26d-5275b25fcbd7 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 25.988644
2019-01-09 18:13:09.048 659037 INFO neutron.wsgi [req-e18a1b2f-cec4-4dc7-95c7-4e97b54741a9 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:09] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 25.845644
2019-01-09 18:13:11.075 659088 INFO neutron.wsgi [req-87f5ab31-f789-4501-a52d-cace515a4cd3 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:11] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 30.743967
2019-01-09 18:13:15.268 659070 INFO neutron.wsgi [req-a07e6c15-8aca-4025-bebc-91aa4b295b2d 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:15] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 34.232041
2019-01-09 18:13:15.422 659083 INFO neutron.wsgi [req-92f7d741-0730-489c-8134-0a2d1860bb0f 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:15] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 34.336818
2019-01-09 18:13:16.968 659074 INFO neutron.wsgi [req-833749f9-eeb9-449a-803b-a949605b8b94 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:16] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 34.788751
2019-01-09 18:13:17.557 659041 INFO neutron.wsgi [req-d5146ffc-1ce3-485f-888a-d4f07467e86e 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:17] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 35.646153
2019-01-09 18:13:17.690 659039 INFO neutron.wsgi [req-9d562a21-5fd7-4201-9542-4ff15cae867b 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:17] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 34.948522
2019-01-09 18:13:17.766 659041 INFO neutron.wsgi [req-779a96ea-74df-4c15-9651-5e6736d30fc5 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:17] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 35.848429
2019-01-09 18:13:18.255 659046 INFO neutron.wsgi [req-9fd3ed85-0295-479f-8c24-9ab77b8d04d0 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:18] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 35.622189
2019-01-09 18:13:18.559 659057 INFO neutron.wsgi [req-9ba95f80-3f27-4ac2-b372-982155b68854 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:18] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 35.273351
2019-01-09 18:13:18.627 659083 INFO neutron.wsgi [req-92ce19c1-43e3-4b6a-ae34-4363ece3991f 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:18] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 36.227857
2019-01-09 18:13:19.014 659040 INFO neutron.wsgi [req-1658062c-e3fb-45a7-9fd1-c59c892a73ca 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:19] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 36.534461
2019-01-09 18:13:19.109 659060 INFO neutron.wsgi [req-c1cc5ea6-1184-4eb3-9792-1dc632d6ca68 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:19] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 35.951095
2019-01-09 18:13:20.240 659063 INFO neutron.wsgi [req-5755b1db-5d88-4a1e-b191-7e0f74ee2370 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:20] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 37.459986
2019-01-09 18:13:20.386 659036 INFO neutron.wsgi [req-f65c5920-2d3b-4dea-b5ea-18079d0bb528 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:20] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 38.422515
2019-01-09 18:13:24.270 659042 INFO neutron.wsgi [req-55ebbb23-3fcb-4c53-863d-91f3f629f003 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:24] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 43.431919
2019-01-09 18:13:27.821 659070 INFO neutron.wsgi [req-e05d88bb-f58a-4776-8abc-0af2fd9829ba 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 45.568455
2019-01-09 18:13:28.150 659035 INFO neutron.wsgi [req-f6ece33f-1b61-462f-b603-a0ea06a429f1 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 45.939718
2019-01-09 18:13:28.220 659044 INFO neutron.wsgi [req-a816061f-0767-4c24-be52-b01bd15baf07 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 45.721643
2019-01-09 18:13:28.850 659033 INFO neutron.wsgi [req-f9f26ca4-fbb4-4d5d-97f7-bc0102ade9b0 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 45.507014
2019-01-09 18:13:29.244 659079 INFO neutron.wsgi [req-8c524ec1-d326-494a-a1aa-86c0c527e60d 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:29] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 45.749202
2019-01-09 18:13:29.451 659038 INFO neutron.wsgi [req-46967491-bc69-4846-a3b9-ee776c17d900 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:29] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 46.302063
2019-01-09 18:13:29.532 659033 INFO neutron.wsgi [req-40909a32-e505-4e9b-a1a9-403b4939f332 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:29] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 46.166459
2019-01-09 18:13:29.722 659072 INFO neutron.wsgi [req-92410ea4-b218-433f-baa5-7e23d6321050 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:29] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 46.527937
2019-01-09 18:13:30.761 659078 INFO neutron.wsgi [req-98bf1e81-d739-48a4-b871-434de7ab40be 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:30] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 49.156775
2019-01-09 18:13:30.883 659062 INFO neutron.wsgi [req-379bd1db-e753-4c46-992b-8f1d18315b88 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:30] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 47.807229
2019-01-09 18:13:32.690 659054 INFO neutron.wsgi [req-6bf0f52c-1e34-4adf-acec-f163068e2247 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:32] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 50.738091
2019-01-09 18:13:32.705 659054 INFO neutron.wsgi [req-3b83c9fb-f7d7-436b-91b4-e15de5437248 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:32] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 49.249812
2019-01-09 18:13:34.935 659087 INFO neutron.wsgi [req-d5b4a357-d302-46e4-a0a6-a9560d774da0 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:34] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 53.998600
2019-01-09 18:13:35.949 659084 INFO neutron.wsgi [req-3ada2fd3-a8aa-46ab-a21f-b60b5c246a07 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:35] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 54.924183
2019-01-09 18:13:37.577 659080 INFO neutron.wsgi [req-9b92c55a-f6be-4e56-a21a-4157bcbf6eb0 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.000273
2019-01-09 18:13:37.610 659034 INFO neutron.wsgi [req-23311b42-2472-4063-bc65-ac72086f437b 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.058743
2019-01-09 18:13:37.647 659045 INFO neutron.wsgi [req-b9d7a452-7c6c-4c84-8a13-20c9836bc0cd 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 55.728784
2019-01-09 18:13:37.686 659083 INFO neutron.wsgi [req-3488bc1d-be09-471a-908d-1d8e7b7dc806 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.177863
2019-01-09 18:13:37.704 659064 INFO neutron.wsgi [req-7567311a-3f9b-4c91-9884-9eda679d10f7 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.081056
2019-01-09 18:13:37.855 659085 INFO neutron.wsgi [req-22a4adb3-551f-4bf9-9652-30c2e25c7c2a 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 55.953447
2019-01-09 18:13:38.029 659078 INFO neutron.wsgi [req-752dd4b9-92dc-444c-b4d5-7e8b8ced98ff 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:38] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.369914
2019-01-09 18:13:38.045 659039 INFO neutron.wsgi [req-e67ebcf4-3b2f-41c8-8f02-ef85b443f3c0 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:38] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.332945
2019-01-09 18:13:38.122 659062 INFO neutron.wsgi [req-294e6202-1c59-440a-8956-168ba95d55ae 7a491879c2dc226e8ae7579b645fe5a1dce331580c82cf041e9af04da852c261 d41457ddbeac42fc9f4f2ee5d014d984 - - -] 172.16.64.69 - - [09/Jan/2019 18:13:38] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 56.148020
~~~

[2]
~~~
2019-01-10 17:40:12.572 12318 INFO neutron.wsgi [req-78b8350c-8668-4ea4-a649-01705aa6e45a d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:40:12] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 10.772870
2019-01-10 17:40:33.451 12318 INFO neutron.wsgi [req-ec27e91a-aabb-43e3-a42f-4d7ca0c47762 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:40:33] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 55.123923
2019-01-10 17:40:35.992 12318 INFO neutron.wsgi [req-77304e97-c4e2-48b7-8f65-3fd9549ed0fa d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:40:35] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 16.061306
2019-01-10 17:40:43.854 12318 INFO neutron.wsgi [req-e5d2c61e-0af6-45e9-a336-73d841f2cfc6 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:40:43] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 21.833349
2019-01-10 17:40:52.226 12318 INFO neutron.wsgi [req-6a62ff59-6328-4d64-8cd4-9f5aefa0a484 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:40:52] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 30.500384
2019-01-10 17:40:52.480 12318 INFO neutron.wsgi [req-cfeb87db-f31c-4c9e-887a-38478b26c08b d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:40:52] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 12.734167
2019-01-10 17:41:01.198 12318 INFO neutron.wsgi [req-445d3bf9-34a9-4b20-9002-19716491f4ac d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:01] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 14.581553
2019-01-10 17:41:06.895 12318 INFO neutron.wsgi [req-868114db-07cc-48e4-a7f7-e98446a0df23 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:06] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 78.888712
2019-01-10 17:41:07.008 12318 INFO neutron.wsgi [req-9ec40a42-8737-4fbb-a85c-9ade3ee6b071 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:07] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 16.365100
2019-01-10 17:41:07.181 12318 INFO neutron.wsgi [req-1ae10178-f061-438d-b9e0-d671ce73eaf1 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:07] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 11.913032
2019-01-10 17:41:07.388 12318 INFO neutron.wsgi [req-1d3f2cbd-879c-49d8-8ac0-494a17155cc9 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:07] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 10.738028
2019-01-10 17:41:37.127 12318 INFO neutron.wsgi [req-347df899-1b20-4084-a1c5-8cadd80f1394 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 45.404808
2019-01-10 17:41:38.642 12318 INFO neutron.wsgi [req-37d566e7-0789-4900-af22-e11f71a09a18 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:38] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 26.705157
2019-01-10 17:41:41.445 12318 INFO neutron.wsgi [req-35ac409f-79c9-4462-9303-45fa05cccf68 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:41] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 27.064104
2019-01-10 17:41:43.910 12318 INFO neutron.wsgi [req-88ed83ac-a1c7-485d-bda9-4a972d78e546 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:43] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 31.888437
2019-01-10 17:41:56.683 12318 INFO neutron.wsgi [req-5f037366-dbad-4d1e-ab9b-bc6a558a8d95 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:56] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 40.519821
2019-01-10 17:41:59.140 12318 INFO neutron.wsgi [req-0c520c0b-d318-4e68-b117-8dd8a5a85f7d d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:41:59] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 24.979791
2019-01-10 17:42:07.494 12318 INFO neutron.wsgi [req-80ec1708-3ed8-49d1-a65b-ccd7d0e38029 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:42:07] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 48.835000
2019-01-10 17:42:41.039 12318 INFO neutron.wsgi [req-ca94ce33-f26b-44af-8368-7a2cd0fc6c01 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:42:41] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 15.085896
2019-01-10 17:42:53.380 12318 INFO neutron.wsgi [req-53288ba1-0eab-435b-ad90-d18867dddd81 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:42:53] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 7.280197
2019-01-10 17:42:56.450 12318 INFO neutron.wsgi [req-864d8485-d86a-4bce-b18f-d2a02f6b4c8c d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:42:56] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 7.418859
2019-01-10 17:42:58.501 12318 INFO neutron.wsgi [req-903abc8b-7d31-46f9-b558-d1051313925f d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:42:58] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 9.464953
2019-01-10 17:43:22.511 12318 INFO neutron.wsgi [req-a94bdfd3-6798-49c8-953e-e3085ed03dcf d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:22] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 13.797552
2019-01-10 17:43:23.589 12318 INFO neutron.wsgi [req-23abbbd4-9404-410c-a37b-290654a59660 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:23] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 17.315682
2019-01-10 17:43:23.794 12318 INFO neutron.wsgi [req-324902d6-88fb-4171-be75-16cc0ab006b0 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:23] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 13.292059
2019-01-10 17:43:27.449 12318 INFO neutron.wsgi [req-159222cf-318a-4ef1-93a5-01b83280bd08 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 16.969784
2019-01-10 17:43:43.514 12318 INFO neutron.wsgi [req-caac8ea9-6d23-48a1-b06f-0809fd3523ad d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:43] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 27.242209
2019-01-10 17:43:44.698 12318 INFO neutron.wsgi [req-570232ba-f89d-4990-afb2-161e589b8931 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:44] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 21.775630
2019-01-10 17:43:45.529 12318 INFO neutron.wsgi [req-401b1001-2415-447f-a006-940226c23e75 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:45] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 10.987276
2019-01-10 17:43:45.677 12318 INFO neutron.wsgi [req-d8f7344b-32ed-46b2-93dc-8675363459b8 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:45] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 9.080061
2019-01-10 17:43:45.758 12318 INFO neutron.wsgi [req-6701b2ad-6181-404b-8f04-e2f8fc6e14dc d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:45] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 7.619617
2019-01-10 17:43:51.813 12318 INFO neutron.wsgi [req-fec67e6f-a05a-4c77-81cc-878130f9a95d d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:43:51] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 3.688603
2019-01-10 17:44:00.713 12318 INFO neutron.wsgi [req-6b4ea69c-8a81-4059-be19-0054b94607f6 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:00] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 5.488108
2019-01-10 17:44:07.891 12318 INFO neutron.wsgi [req-ebdb9510-ee74-40c8-bbe0-b43db2f9577e d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:07] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 8.145477
2019-01-10 17:44:09.654 12318 INFO neutron.wsgi [req-6e5f4a7d-6202-463b-b8e0-55e690d87e96 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:09] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 3.432559
2019-01-10 17:44:28.452 12318 INFO neutron.wsgi [req-b7ee6730-f46f-44b3-bec6-c8bf044ecd4d d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 12.918780
2019-01-10 17:44:28.542 12318 INFO neutron.wsgi [req-c34ec39c-ca3d-4ef6-af8d-89e4bb6ee020 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 4.747629
2019-01-10 17:44:28.611 12318 INFO neutron.wsgi [req-7d383208-d67b-46b4-ba93-099ff1404d3a d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 18.433398
2019-01-10 17:44:43.673 12318 INFO neutron.wsgi [req-27b12bde-9576-4a53-b3a1-9f5ae8ce24ed d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:43] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 15.513576
2019-01-10 17:44:48.959 12318 INFO neutron.wsgi [req-6cb3d363-4dd0-4e20-ab19-ee7f3f5f6ef0 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:48] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 10.607324
2019-01-10 17:44:57.140 12318 INFO neutron.wsgi [req-b94a6f1f-cda5-4ed8-a82c-9e67a8aa692c d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:44:57] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 5.508763
2019-01-10 17:45:01.668 12318 INFO neutron.wsgi [req-762457bb-4932-4b86-8a8e-22677512586e d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:45:01] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 5.171662
2019-01-10 17:45:03.230 12318 INFO neutron.wsgi [req-7eb7ed74-f9e9-4ea1-88f3-6ba8ae2a69bb d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:45:03] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 31.429428
2019-01-10 17:45:17.296 12318 INFO neutron.wsgi [req-6e930aa1-5bbe-4191-a169-d35d1a4208c2 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:45:17] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 4.814631
2019-01-10 17:45:59.659 12318 INFO neutron.wsgi [req-3f970ded-5e9d-4b5b-b65b-7f90b13becb5 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:45:59] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 43.267099
2019-01-10 17:46:01.244 12318 INFO neutron.wsgi [req-4b0ac169-0eef-47eb-8998-0b0781d60098 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:46:01] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 30.616923
2019-01-10 17:46:14.203 12318 INFO neutron.wsgi [req-d70ba25c-78f4-4a88-be9f-2c4162c659e3 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:46:14] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 43.800272
2019-01-10 17:46:16.711 12318 INFO neutron.wsgi [req-0f5189ca-03ae-4316-90b2-9dca425fecd7 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:46:16] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 60.481034
2019-01-10 17:47:23.305 12318 INFO neutron.wsgi [req-8764cea4-d22a-48b2-add2-109127a30ca1 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:47:23] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 58.530555
2019-01-10 17:47:26.421 12318 INFO neutron.wsgi [req-60ea38c0-f471-4933-b28b-3c3f65e103af d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:47:26] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 89.963219
2019-01-10 17:47:34.382 12318 INFO neutron.wsgi [req-a7630f4c-0f96-4894-aef1-fbed1d64bcab d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:47:34] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 140.461731
2019-01-10 17:48:13.079 12318 INFO neutron.wsgi [req-5db4ad3c-26e9-46c0-998e-9d231a355afe d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:48:13] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 167.738372
2019-01-10 17:48:13.457 12318 INFO neutron.wsgi [req-3f1bcb21-fa74-4035-9ad5-0a941a7e1f4f d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:48:13] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 157.889871
2019-01-10 17:50:21.844 12318 INFO neutron.wsgi [req-3b5b4fc1-e5a5-4ad8-8722-5854fbb7de3d d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:50:21] "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 278.761917
2019-01-10 17:51:11.882 12318 INFO neutron.wsgi [req-e7e5b68b-27ed-4442-befe-02cf9ff5c18d d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:11] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 182.949600
2019-01-10 17:51:20.993 12318 INFO neutron.wsgi [req-3edd678f-199c-4ccd-bed9-4ae42196e33a d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:20] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 347 268.617445
2019-01-10 17:51:21.255 12318 INFO neutron.wsgi [req-5029476a-a9ae-4cb8-8d76-36e7cda627f1 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:21] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 347 280.915733
2019-01-10 17:51:21.595 12318 INFO neutron.wsgi [req-2a3f7c3d-b289-4c98-b988-b768c61644b2 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:21] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 0 317.639249
2019-01-10 17:51:24.655 12318 INFO neutron.wsgi [req-0c89b68d-4b9c-4dd6-b50e-30f7bc82c291 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:24] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 148.081944
2019-01-10 17:51:26.127 12318 INFO neutron.wsgi [req-b9f0effc-a8c1-4a73-914b-b15d465298a0 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:26] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 347 253.688642
2019-01-10 17:51:26.133 12318 INFO neutron.wsgi [req-da28b0c6-6e50-4cd1-ae8a-eaa9e6a4d500 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:26] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 347 289.802585
2019-01-10 17:51:26.170 12318 INFO neutron.wsgi [req-525f0a3c-9b91-4cfd-be02-bedaf01571e8 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:26] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 209.765501
2019-01-10 17:51:27.918 12318 INFO neutron.wsgi [req-58959530-25d8-4253-ae49-b9f92cead69a d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 347 293.244829
2019-01-10 17:51:27.953 12318 INFO neutron.wsgi [req-df40a875-0e52-48ef-87f7-8b0819c20ada d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 157.116738
2019-01-10 17:51:28.451 12318 INFO neutron.wsgi [req-a57e02b3-5ef0-4d37-9eba-b37fea609b90 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:28] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 153.255538
2019-01-10 17:51:31.404 12318 INFO neutron.wsgi [req-0162940f-264e-4631-b071-49fdb91521fc d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:31] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 0 315.656436
2019-01-10 17:51:31.406 12318 INFO neutron.wsgi [req-78e33ba9-6716-47d7-ae54-c24a555b3403 d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:31] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 0 346.346748
2019-01-10 17:51:31.573 12318 INFO neutron.wsgi [req-daa95ec1-c92b-43ba-be10-544122585a8f d916eec19bd644a8821cd361a388db5b 24059ad8493444fa8d45793a2e463e62 - - -] 172.17.1.11 - - [10/Jan/2019 17:51:31] "POST /v2.0/security-group-rules.json HTTP/1.1" 409 347 291.111367
~~~
Comment 5 Brian Haley 2019-01-14 20:47:55 UTC 
I send David a patch to try in his environment, just to get some feedback.  My OSP environment is installing at the moment.
Comment 8 David Vallee Delisle 2019-01-15 22:39:10 UTC 
(for posterity, I'll paste this from the email thread)

Unfortunately, the patch didn't seem to work for me

As a workaround, I realized that the stack was creating successfully on my undercloud instead of my overcloud. So I tried to stop neutron-server on 2 out of the 3 controllers and I increased haproxy's timeout, as well as neutron's sqlalchemy settings.

It's still failing at some point and there's clearly a buildup in the duration of the POSTs [1].

Also, it looks like like heat-engine is flooding the neutron API with GETs to /v2.0/extensions.json. Even though the number of these calls is enormous, These are pretty lightweight but the heat stack fails to create when one of these calls is not returning fast enough. [2] contains the number of these calls / minute on the neutron api side, and [3] contains the number of these calls / minute from one of the heat engine's side. 

Here's the stack trace [4]

At this point, I'm wondering if we don't have 2 issues. Neutron getting slower and slower, and heat being a bit too agressive with its "service availability validations".

Also, I'd like to point out that after a stack failed, it's impossible to delete it unless I manually loop through the service group rules and delete them with API calls. I can't even list the rules of a service group, I get a timeout. 

I'll try the patched version on one controller for now, but that would be great to have more feedback from engineering. That would be great to know if 50+ calls/second to /v2.0/extensions.json can be impacting at some point, and if it's really necessary.


DVD

[1]
~~~
2019-01-15 19:47:19.580 988415 INFO neutron.wsgi [req-a08cf5a3-f78c-4db7-a695-12fa6c633178 <SNIP> "POST /v2.0/security-groups.json HTTP/1.1" 201 1585 0.177781
2019-01-15 19:47:19.815 988415 INFO neutron.wsgi [req-ffcf0c33-92f7-433f-956f-83447192e780 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 692 0.209263
2019-01-15 19:47:20.317 988415 INFO neutron.wsgi [req-9ba6afe2-da48-4383-83d7-bfb234c319e5 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 691 0.204312
2019-01-15 19:51:32.691 988415 INFO neutron.wsgi [req-7803947e-50ad-44c9-b11b-117bf7c6f85f <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 16.791251
2019-01-15 19:51:36.343 988415 INFO neutron.wsgi [req-b9d4092a-26c6-4480-a00d-d3f0ec4ac9c8 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 14.068940
2019-01-15 19:51:36.698 988415 INFO neutron.wsgi [req-7afc5146-c460-40dc-a087-d1f276271427 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 1.517953
2019-01-15 19:51:49.328 988415 INFO neutron.wsgi [req-d4de311b-71d4-43c6-bd6c-0361aba957ae <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 9.610927
2019-01-15 19:51:50.892 988415 INFO neutron.wsgi [req-25eb98b2-f7aa-45f0-bf9c-bcb754dac184 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 14.876030
2019-01-15 19:51:58.667 988415 INFO neutron.wsgi [req-d538e36f-db02-4f2c-b92c-2d522804aba5 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 14.241387
2019-01-15 19:52:01.714 988415 INFO neutron.wsgi [req-0db87c71-4d02-42d1-9921-e31c5ff1a05f <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 14.407799
2019-01-15 19:52:10.841 988415 INFO neutron.wsgi [req-da9f14f9-9c45-42a4-8ac1-3330440648f2 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 26.159587
2019-01-15 19:52:16.303 988415 INFO neutron.wsgi [req-96e571bf-6b34-425b-8f7e-5e036eb1f820 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 15.903058
2019-01-15 19:52:16.654 988415 INFO neutron.wsgi [req-e6a94f99-52cc-4209-9407-4c79f6ded4f4 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 13.617583
2019-01-15 19:52:29.603 988415 INFO neutron.wsgi [req-767ff22e-1501-429f-8456-f79136741563 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 16.726857
2019-01-15 19:52:36.672 988415 INFO neutron.wsgi [req-277c428a-c2b4-4306-9f8a-93ec6c330f2d <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 26.011657
2019-01-15 19:52:39.039 988415 INFO neutron.wsgi [req-d046d305-a858-4197-9aff-d1aa92cc5013 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 18.620860
2019-01-15 19:52:39.129 988415 INFO neutron.wsgi [req-1387dd6d-4b7d-4ea1-a2d3-a8025691339d <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 21.609913
2019-01-15 19:52:45.177 988415 INFO neutron.wsgi [req-e423bac0-d3a3-4bd5-80a7-26ecc5acf992 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 20.025870
2019-01-15 19:52:46.028 988415 INFO neutron.wsgi [req-1627776b-69cc-450b-af0f-6a1cd46b3156 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 22.658502
2019-01-15 19:52:51.165 988415 INFO neutron.wsgi [req-2302153e-7e74-453a-92a5-977d5fe8241c <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 22.737595
2019-01-15 19:52:51.180 988415 INFO neutron.wsgi [req-39c624e5-7561-4991-9b52-b1158c35e36d <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 18.844666
2019-01-15 19:52:54.547 988415 INFO neutron.wsgi [req-cfac5a7e-2f77-41af-9b4e-2a9eaf5834ca <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 24.258442
2019-01-15 19:52:54.662 988415 INFO neutron.wsgi [req-5f3307a8-c245-4072-8d37-8515602819c9 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 15.109443
2019-01-15 19:52:59.831 988415 INFO neutron.wsgi [req-b0a3583f-87bc-4fd8-8b50-2b5f85733b82 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 15.903552
2019-01-15 19:53:06.112 988415 INFO neutron.wsgi [req-434d319b-d064-4867-b649-7ce8476472d7 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 35.665496
2019-01-15 19:53:06.530 988415 INFO neutron.wsgi [req-b5698a80-8bd1-40d0-aa65-9aa7b036d60d <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 17.874464
2019-01-15 19:53:06.886 988415 INFO neutron.wsgi [req-d1b19d79-3162-4c22-879b-e9f6a0d0be48 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 14.455133
2019-01-15 19:53:15.236 988415 INFO neutron.wsgi [req-b58bd74a-6b79-4509-b248-c0bfd4495851 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 16.102678
2019-01-15 19:53:18.277 988415 INFO neutron.wsgi [req-e4cc92f1-1a9d-48e2-885c-635a6b229150 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 45.369257
2019-01-15 19:53:20.133 988415 INFO neutron.wsgi [req-0050eec3-436e-402f-9329-7c6e17646f05 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 14.844351
2019-01-15 19:53:20.989 988415 INFO neutron.wsgi [req-27b1329d-be50-4d50-9388-1a3e0625ba40 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 44.393658
2019-01-15 19:53:27.791 988415 INFO neutron.wsgi [req-8e27c892-1cff-4612-b049-5bd7a7f89d28 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 21.656269
2019-01-15 19:53:32.453 988415 INFO neutron.wsgi [req-650d4517-e9d1-41a2-b6ad-5608f9c700c0 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 20.101678
2019-01-15 19:53:34.202 988415 INFO neutron.wsgi [req-6c6ba3fb-ce0c-4b97-999c-5c830821d6ec <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 15.787255
2019-01-15 19:53:42.130 988415 INFO neutron.wsgi [req-b671cc75-6ea5-445b-bbd6-5ada6fea7be3 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 54.972306
2019-01-15 19:53:45.845 988415 INFO neutron.wsgi [req-e08068d7-db37-4400-b306-74f0f1172125 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 23.388320
2019-01-15 19:53:58.979 988415 INFO neutron.wsgi [req-086f5e3d-1b75-4fd5-a06c-fc14471fcb2c <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 30.182090
2019-01-15 19:54:01.465 988415 INFO neutron.wsgi [req-d38479ce-841d-48be-afe2-d5a2ffa3c606 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 54.739885
2019-01-15 19:54:10.876 988415 INFO neutron.wsgi [req-30837d47-30ec-48d7-af7a-a395406c6445 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 39.601803
2019-01-15 19:54:25.617 988415 INFO neutron.wsgi [req-ecf947f2-398b-490f-a5d0-cc57215cb78e <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 44.332826
2019-01-15 19:54:40.077 988415 INFO neutron.wsgi [req-3921fd53-45a1-441d-ade4-dd608dc151d3 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 47.553737
2019-01-15 19:54:52.544 988415 INFO neutron.wsgi [req-7c719f4f-72dd-40ec-bc77-ddc839183408 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 154.134486
2019-01-15 19:54:53.516 988415 INFO neutron.wsgi [req-e9707c8b-d868-4c88-89ed-6930b62958cc <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 28.462749
2019-01-15 19:55:14.696 988415 INFO neutron.wsgi [req-a5c3b87b-bec0-411c-a32b-d91f63daa7ae <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 118.773054
2019-01-15 19:55:30.893 988415 INFO neutron.wsgi [req-d3a9210a-510a-4d5b-b795-182d4800ab32 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 102.303439
2019-01-15 19:55:47.641 988415 INFO neutron.wsgi [req-b5bdf2f7-2521-40af-8618-f4b11c892d6a <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 106.655262
2019-01-15 19:55:56.359 988415 INFO neutron.wsgi [req-2530ec41-7ebb-465e-9f29-5d33728c1095 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 188.876560
2019-01-15 19:56:02.326 988415 INFO neutron.wsgi [req-7c456e85-9c7d-4098-aab3-f41196a6b136 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 107.206438
2019-01-15 19:56:16.989 988415 INFO neutron.wsgi [req-80a1b4e1-8f5e-4a43-8a8f-d8f323169fc1 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 143.618252
2019-01-15 19:56:18.348 988415 INFO neutron.wsgi [req-bb2b5c17-7e00-4a2f-8319-59054a91cb65 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 199.180581
2019-01-15 19:56:28.786 988415 INFO neutron.wsgi [req-bb911917-cde4-4dbc-b84c-dc7f0d15d990 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 183.594454
2019-01-15 19:56:34.992 988415 INFO neutron.wsgi [req-700ea889-3c41-427e-894e-c5ead319dda3 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 207.171425
2019-01-15 19:56:41.027 988415 INFO neutron.wsgi [req-ddb79694-a729-4c5e-872f-6d41fc69bfa4 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 193.279332
2019-01-15 19:56:47.757 988415 INFO neutron.wsgi [req-330def83-1180-4a03-af6e-61c6666e314b <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 215.920990
2019-01-15 19:56:49.348 988415 INFO neutron.wsgi [req-472aa58f-d6af-44c5-bc9f-c0e870b252c8 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 213.843158
2019-01-15 19:56:50.166 988415 INFO neutron.wsgi [req-330597d3-1e03-4718-acc7-7816edab26a5 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 137.999861
2019-01-15 19:56:50.194 988415 INFO neutron.wsgi [req-76d9497e-64be-46f8-9817-e372be743f9b <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 211.362064
2019-01-15 19:56:54.332 988415 INFO neutron.wsgi [req-a65f78ed-0043-458e-b914-1cf809416190 <SNIP> "POST /v2.0/security-group-rules.json HTTP/1.1" 500 344 215.126256
~~~

[2]
~~~
   1081 2019-01-15 19:46
   5938 2019-01-15 19:47
  11942 2019-01-15 19:48
  12252 2019-01-15 19:49
  11916 2019-01-15 19:50
  11469 2019-01-15 19:51
  10505 2019-01-15 19:52
   7595 2019-01-15 19:53
   5817 2019-01-15 19:54
   6669 2019-01-15 19:55
   7252 2019-01-15 19:56
   7546 2019-01-15 19:57
   9235 2019-01-15 19:58
  10187 2019-01-15 19:59
  10296 2019-01-15 20:00
   9974 2019-01-15 20:01
  10241 2019-01-15 20:02
   6472 2019-01-15 20:03
      4 2019-01-15 20:08
~~~

[3]
~~~
   1846 2019-01-15 19:47
   4071 2019-01-15 19:48
   4181 2019-01-15 19:49
   3991 2019-01-15 19:50
   3871 2019-01-15 19:51
   3404 2019-01-15 19:52
   2318 2019-01-15 19:53
   1913 2019-01-15 19:54
   2630 2019-01-15 19:55
   2879 2019-01-15 19:56
   2810 2019-01-15 19:57
   3337 2019-01-15 19:58
   3452 2019-01-15 19:59
   3450 2019-01-15 20:00
   3249 2019-01-15 20:01
   3466 2019-01-15 20:02
   2163 2019-01-15 20:03
~~~

[4]
~~~
2019-01-15 19:53:53.353 107421 DEBUG dogpile.lock [req-599d00b6-5d36-4d4f-be5c-2dbc1ff23792 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:144
2019-01-15 19:53:53.385 107421 DEBUG neutronclient.v2_0.client [req-599d00b6-5d36-4d4f-be5c-2dbc1ff23792 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] Error message: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
 _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:265
2019-01-15 19:53:53.387 107421 DEBUG neutronclient.v2_0.client [req-599d00b6-5d36-4d4f-be5c-2dbc1ff23792 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] GET call to neutron for https://10.0.0.101:13696/v2.0/extensions.json used request id None _append_request_id /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:127
2019-01-15 19:53:53.388 107421 DEBUG dogpile.lock [req-599d00b6-5d36-4d4f-be5c-2dbc1ff23792 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:150
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource [req-599d00b6-5d36-4d4f-be5c-2dbc1ff23792 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] Resource type OS::Neutron::SecurityGroupRule unavailable
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource Traceback (most recent call last):
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 699, in is_service_available
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     req_extension))
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/neutron/__init__.py", line 88, in has_extension
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     return alias in self._list_extensions()
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1220, in decorate
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     should_cache_fn)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 825, in get_or_create
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     async_creator) as value:
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 154, in __enter__
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     return self._enter()
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 94, in _enter
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/lock.py", line 145, in _enter_create
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     created = self.creator()
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 792, in gen_value
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     created_value = creator()
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1216, in creator
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     return fn(*arg, **kw)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/neutron/__init__.py", line 83, in _list_extensions
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 360, in get
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     headers=headers, params=params)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     headers=headers, params=params)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource     request_ids=request_ids)
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource NeutronClientException: <html><body><h1>504 Gateway Time-out</h1>
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource The server didn't respond in time.
2019-01-15 19:53:53.388 107421 ERROR heat.engine.resource </body></html>
~~~
Comment 9 David Vallee Delisle 2019-01-16 14:48:39 UTC 
I tried to stop heat-engine on 2/3 controllers this morning and so far there's no buildup [1] when creating the stack. I'll let this round finish, and I'll try to re-enable neutron-server on all 3 controllers, and leave heat-engine stopped on 2/3 and try again.



[1]
~~~
2019-01-16 14:40:06.195 210498 INFO neutron.wsgi [req-c03be612-8dab-415e-a437-72adc1575bd4 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:06] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.626889
2019-01-16 14:40:16.527 210498 INFO neutron.wsgi [req-0321be5a-d074-475a-9f3c-11fd9a51c994 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:16] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 1.092957
2019-01-16 14:40:23.652 210498 INFO neutron.wsgi [req-a3235f17-cd03-4ab5-a447-912e23d484f1 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:23] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 0.601209
2019-01-16 14:40:31.903 210498 INFO neutron.wsgi [req-61c10885-c2f4-4112-9bb4-207b16ba315d 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:31] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.756505
2019-01-16 14:40:34.289 210498 INFO neutron.wsgi [req-0ee6d2b6-6751-4bbc-9891-aa2e31ddd4fd 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:34] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.576624
2019-01-16 14:40:35.488 210498 INFO neutron.wsgi [req-9cc188b3-b6f3-44ab-9bf7-67b8151b53cb 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:35] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.323727
2019-01-16 14:40:40.870 210498 INFO neutron.wsgi [req-17ffe6e5-8f6c-4730-82f3-863620c2c371 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 0.649286
2019-01-16 14:40:45.713 210498 INFO neutron.wsgi [req-2c3fbdfc-6c7f-492c-b761-9969a14495cd 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:45] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.684487
2019-01-16 14:40:50.508 210498 INFO neutron.wsgi [req-ed0a1883-cf01-4932-9b78-cec6c877021b 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:50] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.967023
2019-01-16 14:40:52.349 210498 INFO neutron.wsgi [req-59509e28-c55b-484b-bf7d-beb1375fa352 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:52] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 0.862621
2019-01-16 14:40:52.380 210498 INFO neutron.wsgi [req-ddbdd926-7ce2-4a9f-8b91-2508501e9141 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:52] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 0.634230
2019-01-16 14:40:55.763 210498 INFO neutron.wsgi [req-b8b27fc5-83e7-444d-b27d-1d416f34c7c2 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:40:55] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 0.961653
2019-01-16 14:41:01.414 210498 INFO neutron.wsgi [req-1552266f-3819-44e5-8e22-063ba0be8eae 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:01] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.571252
2019-01-16 14:41:19.284 210498 INFO neutron.wsgi [req-bcce7c1c-72d6-4a01-b067-8a3139222516 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:19] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.748511
2019-01-16 14:41:19.807 210498 INFO neutron.wsgi [req-05c36e9c-de1c-4336-bc57-cedc21b69f94 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:19] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 1.282144
2019-01-16 14:41:24.781 210498 INFO neutron.wsgi [req-cfd650dd-8299-4820-8902-dc069b6ce374 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:24] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 0.541047
2019-01-16 14:41:32.876 210498 INFO neutron.wsgi [req-65cfcff3-578f-4a19-b32f-91e70040828c 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:32] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.951352
2019-01-16 14:41:35.483 210498 INFO neutron.wsgi [req-6126aa2b-8fce-44d5-8f6f-9bfee1df06f7 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:35] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 0.708825
2019-01-16 14:41:36.696 210498 INFO neutron.wsgi [req-4b1ed8a2-ec77-41f8-b343-3be710e8bebc 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:36] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.973826
2019-01-16 14:41:44.475 210498 INFO neutron.wsgi [req-0154c3ee-9e53-48c2-89d5-46f9e163e2ca 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:44] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 1.125709
2019-01-16 14:41:52.776 210498 INFO neutron.wsgi [req-8fb5633a-b6d4-4616-b063-c455e664de1e 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:52] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.861762
2019-01-16 14:41:53.723 210498 INFO neutron.wsgi [req-0391aa42-5270-4fd5-9796-d5a66ab8a408 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:53] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 701 0.522199
2019-01-16 14:41:57.665 210498 INFO neutron.wsgi [req-c06cfd94-b07a-451a-99f5-2a2acb415034 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:41:57] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.635149
2019-01-16 14:42:08.627 210498 INFO neutron.wsgi [req-8b723d35-18ea-4684-a778-a518dbd8f1b0 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:42:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 0.699129
2019-01-16 14:42:12.859 210498 INFO neutron.wsgi [req-66f0aa1a-f2c1-43c9-8353-51d3954cc194 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:42:12] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.912674
2019-01-16 14:42:13.040 210498 INFO neutron.wsgi [req-bd1005ef-287e-441e-958b-9a031e739143 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:42:13] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.555771
2019-01-16 14:42:27.209 210498 INFO neutron.wsgi [req-00b4a490-2e07-4e97-b75c-13ed4c3917ef 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:42:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.431893
2019-01-16 14:42:27.780 210498 INFO neutron.wsgi [req-6f61ebee-294a-4c14-b80c-405fb28f9e4c 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:42:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.521520
2019-01-16 14:44:42.658 210498 INFO neutron.wsgi [req-8f0ce410-582c-45ef-adf3-faade5cb8e16 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:44:42] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 1.553284
2019-01-16 14:44:51.654 210498 INFO neutron.wsgi [req-06e39d25-6a2b-473e-ab57-7b577b08220b 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:44:51] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 0.937651
2019-01-16 14:45:00.627 210498 INFO neutron.wsgi [req-ca1e0303-e1bb-4060-8046-c564e9573658 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:00] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 1.044528
2019-01-16 14:45:01.060 210498 INFO neutron.wsgi [req-4a3dd205-84e0-4010-aa06-4e5cae45a883 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:01] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 0.860162
2019-01-16 14:45:17.780 210498 INFO neutron.wsgi [req-efedb16f-473a-4949-9f73-7f879cfd93f1 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:17] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.776842
2019-01-16 14:45:22.139 210498 INFO neutron.wsgi [req-146fcc52-b194-4a29-b0a6-ac53540d349e 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:22] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.309854
2019-01-16 14:45:25.707 210498 INFO neutron.wsgi [req-7f233cd4-ee77-44de-aae5-babda0204f72 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:25] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.749563
2019-01-16 14:45:31.972 210498 INFO neutron.wsgi [req-8e66c150-7404-474d-bb9a-a27309310757 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:31] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 1.053860
2019-01-16 14:45:32.052 210498 INFO neutron.wsgi [req-f71b1515-cc9a-41fa-ae81-b3559b5828d7 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:32] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 706 0.440611
2019-01-16 14:45:41.418 210498 INFO neutron.wsgi [req-913096a2-bcdc-4b98-b8fa-a8c118d2dbe4 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:41] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 0.748176
2019-01-16 14:45:54.030 210498 INFO neutron.wsgi [req-8d9fd6c1-4420-4176-943e-28dfa133062f 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:54] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 702 0.756269
2019-01-16 14:45:57.086 210498 INFO neutron.wsgi [req-779da302-fb78-452f-8f71-9772a3ed0471 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:45:57] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.626127
2019-01-16 14:46:01.403 210498 INFO neutron.wsgi [req-e1074979-6b64-408c-9571-c54f34e7d57d 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:01] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 707 0.425969
2019-01-16 14:46:08.743 210498 INFO neutron.wsgi [req-0b4e2ca1-76ae-492f-8329-c19dddf1d760 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:08] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.612346
2019-01-16 14:46:13.637 210498 INFO neutron.wsgi [req-d9e24a4d-cf11-48c3-a7c3-0fecd120189f 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:13] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.480778
2019-01-16 14:46:17.978 210498 INFO neutron.wsgi [req-9aff0f08-828b-4242-87cc-e8e40821e85d 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:17] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.918394
2019-01-16 14:46:27.958 210498 INFO neutron.wsgi [req-67e49bb0-5c99-423e-a0fd-030461b2dc1b 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:27] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 1.042901
2019-01-16 14:46:29.034 210498 INFO neutron.wsgi [req-7ecc25a6-ba1d-4bda-aadc-24e9850253d5 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:29] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 0.753321
2019-01-16 14:46:37.068 210498 INFO neutron.wsgi [req-0a3fb9ca-3008-4954-ab79-4a78a72f8db1 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:37] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 704 1.660932
2019-01-16 14:46:38.646 210498 INFO neutron.wsgi [req-8eb305fd-a7fd-4d25-b0a0-5c6d9fd067d9 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:38] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 705 1.344436
2019-01-16 14:46:39.172 210498 INFO neutron.wsgi [req-954509f7-d81a-40a0-add1-ae71f4089668 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:39] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.624947
2019-01-16 14:46:40.364 210498 INFO neutron.wsgi [req-1b35256c-9c05-45c6-b112-6c7251dc9525 195746d8ad6a4cf5a3bcacedec02cbd4 4500dcbc17ff41249cc329bf1e38ffb7 - - -] 172.17.1.16 - - [16/Jan/2019 14:46:40] "POST /v2.0/security-group-rules.json HTTP/1.1" 201 703 0.865810
~~~
Comment 17 Brian Haley 2019-02-25 20:29:57 UTC 
David - the downstream change should be good to test, it seems to work correctly in my OSP 10 environment.
Comment 32 errata-xmlrpc 2019-04-30 16:58:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:0916
Note You need to log in before you can comment on or make changes to this bug.