Bug 1263521

Summary: [undercloud] swift proxy-server keep logging "Invalid user token - deferring reject downstream"
Product: Red Hat OpenStack Reporter: Jaison Raju <jraju>
Component: python-keystonemiddlewareAssignee: John Dennis <jdennis>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Abrams <mabrams>
Severity: medium Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: acanan, apevec, cschwede, dprince, egafford, hbrock, jraju, lhh, mburns, mori, pgrist, rhel-osp-director-maint, scohen, slinaber, thiago, zaitcev
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-22 07:31:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Config none

Description Jaison Raju 2015-09-16 04:32:54 UTC
Created attachment 1073892 [details]
Config

Description of problem:
After initial deployment of undercloud, swift proxy-server keeps logging the
following error :
Sep 16 05:21:07 instack.localdomain proxy-server[27467]: Invalid user token - deferring reject downstream
Sep 16 05:21:07 instack.localdomain proxy-server[27467]: 192.0.2.1 192.0.2.1 16/Sep/2015/09/21/07 GET /info HTTP/1.0 200 - - - - 738 - tx76694f16d3c74c27bf7d4-0055f93483 - 0.0006 - - 1442395267.932071924 1442395
Sep 16 05:21:09 instack.localdomain proxy-server[27467]: Invalid user token - deferring reject downstream

This is noticed in swift.log , messages or from systemd commands .
This is logged almost every second , hence log file increases by +500M in a day or two .
This error is notice on overcloud controllers & undercloud node .
openstack overcloud deploy --templates --ntp-server xx.xx.xx.xx --control-flavor control --compute-flavor compute --swift-storage-flavor control --ceph-storage-flavor control --block-storage-flavor control --control-scale 1 --compute-scale 1 --neutron-tunnel-types vxlan --neutron-network-type vxlan --neutron-public-interface eth4 --libvirt-type kvm


Version-Release number of selected component (if applicable):
OSP7 Director

How reproducible:
Yes

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
# date
Wed Sep 16 05:25:52 EDT 2015
[root@instack ~]# journalctl -u openstack-swift-proxy --since 05:25:52
-- Logs begin at Mon 2015-09-14 12:53:35 EDT, end at Wed 2015-09-16 05:25:57 EDT. --
Sep 16 05:25:52 instack.localdomain proxy-server[27467]: Invalid user token - deferring reject downstream
Sep 16 05:25:52 instack.localdomain proxy-server[27467]: 192.0.2.1 192.0.2.1 16/Sep/2015/09/25/52 GET /info HTTP/1.0 200 - - - - 738 - tx2f7370777c5b4d9fa5a71-0055f935a0 - 0.0007 - - 1442395552.575284004 1442395
Sep 16 05:25:54 instack.localdomain proxy-server[27467]: Invalid user token - deferring reject downstream
Sep 16 05:25:54 instack.localdomain proxy-server[27467]: 192.0.2.1 192.0.2.1 16/Sep/2015/09/25/54 GET /info HTTP/1.0 200 - - - - 738 - tx071e3d2d1c8f4067ad5ff-0055f935a2 - 0.0006 - - 1442395554.579209089 1442395
Sep 16 05:25:56 instack.localdomain proxy-server[27467]: Invalid user token - deferring reject downstream
Sep 16 05:25:56 instack.localdomain proxy-server[27467]: 192.0.2.1 192.0.2.1 16/Sep/2015/09/25/56 GET /info HTTP/1.0 200 - - - - 738 - tx0c098e5cd5d34383817a1-0055f935a4 - 0.0009 - - 1442395556.584398031 1442395
[root@instack ~]#  ls -lh /var/log/messages /var/log/swift/swift.log 
-rw-------. 1 root root 629M Sep 16 05:26 /var/log/messages
-rw-r--r--. 1 root root  31M Sep 16 05:26 /var/log/swift/swift.log

Comment 8 Mike Burns 2016-04-07 20:50:54 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 15 Paul Grist 2016-10-14 03:21:10 UTC
Can we take a look at whether this is just a case of excessive logging swift can't control or whether any log options need to be considered?

Comment 25 Pete Zaitcev 2017-04-28 17:07:06 UTC
The upstream review 441549 was approved today. We need to put this in
urgently to make GA, or at least 1-day fixes. I'm changing the component
to keystonemiddleware so that relevant people can act upon it.