Bug 1554796

Summary: [F-QE] heketi-cli node remove command returns Error: Token used before issued
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rachael <rgeorge>
Component: heketiAssignee: Michael Adam <madam>
Status: CLOSED ERRATA QA Contact: vinutha <vinug>
Severity: medium Docs Contact:
Priority: unspecified    
Version: cns-3.9CC: akrishna, asriram, bmohanra, hchiramm, jmulligan, kramdoss, madam, ndevos, pprakash, rhs-bugs, rtalur, sankarshan, storage-qa-internal, suprasad, vinug
Target Milestone: ---   
Target Release: CNS 3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, some heketi client requests failed with ‘Token used before issued’ error because time synchronization was not properly handled by JSON web tokens. With this fix This update adds a margin of 120 seconds to iat claim validation to ensure that client requests can succeed in this situation. This margin can be changed by editing the ‘HEKETI_JWT_IAT_LEEWAY_SECONDS’ environment variable.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 09:22:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1568861    

Comment 2 Rachael 2018-03-13 12:22:29 UTC
The topology info and heketi logs can be found here: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/BZ-1554796/

Comment 3 Raghavendra Talur 2018-03-13 13:34:05 UTC
This error comes from jwt and from the code it can be confirmed that the operation won't fail because of this error. More info here https://github.com/dgrijalva/jwt-go/blob/master/claims.go#L28

Comment 8 Michael Adam 2018-05-15 19:13:51 UTC
This is actually the same as Bug #1541323.

Comment 9 Michael Adam 2018-05-16 12:56:50 UTC
The fix is by preventing ntp from running in the gluster container

Comment 13 Saravanakumar 2018-05-28 07:13:13 UTC
NTP removed from gluster pods.
and ensured only in gluster and all other Nodes.

Corresponding upsteram patch:
https://github.com/gluster/gluster-containers/pull/81

Comment 14 Raghavendra Talur 2018-06-04 17:13:19 UTC
Make sure the node has a time synchronization service running. Gluster pods don't run any time sync service within them anymore.

Comment 23 Humble Chirammal 2018-07-19 06:48:32 UTC
Fixed in version : rhgs-volmanager-rhel7:3.3.1-20

Comment 38 John Mulligan 2018-09-07 17:57:28 UTC
Doc Text looks OK

Comment 40 errata-xmlrpc 2018-09-12 09:22:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686