Bug 1568820 - Add generated HMAC token in header for webhook calls
Summary: Add generated HMAC token in header for webhook calls
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: eventsapi
Version: mainline
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
Assignee: Aravinda VK
QA Contact:
URL:
Whiteboard:
Depends On: 1466129
Blocks: 1496363 1501864
TreeView+ depends on / blocked
 
Reported: 2018-04-18 09:44 UTC by Aravinda VK
Modified: 2018-06-20 18:05 UTC (History)
6 users (show)

Fixed In Version: glusterfs-v4.1.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1466129
Environment:
Last Closed: 2018-06-20 18:05:09 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Aravinda VK 2018-04-18 09:44:02 UTC
+++ This bug was initially created as a clone of Bug #1466129 +++

Description of problem:
Currently, there's no way to ascertain the identity/validity of a webhook requests sent by gluster-eventsd. If a Header was added that contains an HMAC token generated using content and secret key, this would ensure that server can validate the request is genuine before processing the event.


Version-Release number of selected component (if applicable):
3.3

How reproducible:
Always

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-29 02:47:37 EDT ---

This bug is automatically being proposed for the current release of Red Hat Gluster Storage 3 under active development, by setting the release flag 'rhgs‑3.3.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-07-06 08:48:25 EDT ---

This BZ having been considered, and subsequently not approved to be fixed at the RHGS 3.3.0 release, is being proposed for the next minor release of RHGS

--- Additional comment from Aravinda VK on 2017-09-27 00:54:42 EDT ---

As discussed over IRC, Events API will implement JWT(JSON Web Tokens). Additional argument will be added to webhook-add command to accept the secret which is required to generate JWT.

    gluster-eventsapi webhook-add <url> [--token <TOKEN>] \
        [--secret <SECRET>]

For shared secret approach use, `--secret` and for shared token approach use `--token`. With `--token` argument, Token header will be added as is.

    Authorization: Bearer <TOKEN>

In case of shared secret, Gluster will generate JWT token using the secret and then add it to Authorization header.

    Authorization: Bearer <GENERATED_TOKEN>

Secret/Token can be updated using `webhook-mod` command.

Generated token will include the following payload,

    {
        "iss": "gluster",
        "exp": EXPIRY_TIME,
        "sub": EVENT_TYPE,
        "iat": EVENT_TIME
    }

Where: iss - Issuer,  exp - Expiry Time, sub - Event Type used as Subject, iat - Event Time used as Issue Time

--- Additional comment from Aravinda VK on 2017-09-27 04:01:18 EDT ---

Upstream patch posted https://review.gluster.org/#/c/18405

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-11-13 10:35:36 EST ---

Since this bug has has been approved for the RHGS 3.4.0 release of Red Hat Gluster Storage 3, through release flag 'rhgs-3.4.0+', and through the Internal Whiteboard entry of '3.4.0', the Target Release is being automatically set to 'RHGS 3.4.0'

--- Additional comment from Atin Mukherjee on 2017-12-27 05:34:15 EST ---

downstream patch : https://code.engineering.redhat.com/gerrit/126551

--- Additional comment from Aravinda VK on 2017-12-28 04:54:25 EST ---

Another patch sent to Upstream which fixes the external library dependency issue.

https://review.gluster.org/19102

--- Additional comment from Sunil Kumar Acharya on 2017-12-28 05:25:13 EST ---

Second Downstream Patch:

https://code.engineering.redhat.com/gerrit/#/c/126619/

--- Additional comment from Sweta Anandpara on 2018-04-16 02:53:29 EDT ---

Validated this on the build glusterfs-3.12.2-7.el7rhgs.x86_64.

On a 3 node cluster with a webhook-added with default fields, we are able to see the events.. But the same webhook when added using the "--secret" field, no events are seen at the receiver. /var/log/glusterfs/events.log does show the event successfully being pushed, though. There's no indication of any error in the events logs. 

/var/log/messages shows the below traceback in glustereventsd.

Apr 16 02:26:59 dhcp47-187 glustereventsd: Exception in thread Thread-50:
Apr 16 02:26:59 dhcp47-187 glustereventsd: Traceback (most recent call last):
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Apr 16 02:26:59 dhcp47-187 glustereventsd: self.run()
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/lib64/python2.7/threading.py", line 765, in run
Apr 16 02:26:59 dhcp47-187 glustereventsd: self.__target(*self.__args, **self.__kwargs)
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/libexec/glusterfs/events/utils.py", line 247, in publish_to_webhook
Apr 16 02:26:59 dhcp47-187 glustereventsd: hashval = get_jwt_token(secret, event_type, event_ts)
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/libexec/glusterfs/events/utils.py", line 209, in get_jwt_token
Apr 16 02:26:59 dhcp47-187 glustereventsd: base64_urlencode(hmac.HMAC(secret, msg, sha256).digest())
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/lib64/python2.7/hmac.py", line 75, in __init__
Apr 16 02:26:59 dhcp47-187 glustereventsd: self.outer.update(key.translate(trans_5C))
Apr 16 02:26:59 dhcp47-187 glustereventsd: TypeError: character mapping must return integer, None or unicode

Moving this bug to assigned for the reason stated above. Sosreports of the server node, and the webhook logs will be copied at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/swetas/<bugnumber>

--- Additional comment from Sweta Anandpara on 2018-04-16 03:25:08 EDT ---

[qe@rhsqe-repo 1466129]$ hostname
rhsqe-repo.lab.eng.blr.redhat.com
[qe@rhsqe-repo 1466129]$ 
[qe@rhsqe-repo 1466129]$ pwd
/var/www/html/sosreports/swetas/1466129
[qe@rhsqe-repo 1466129]$ 
[qe@rhsqe-repo 1466129]$ ll
total 36616
-rwxr-xr-x. 1 qe qe 37490200 Apr 16 12:54 sosreport-dhcp47-187.lab.eng.blr.redhat.com-20180416025148.tar.xz
-rwxr-xr-x. 1 qe qe      533 Apr 16 12:22 webhook.py
[qe@rhsqe-repo 1466129]$

--- Additional comment from Aravinda VK on 2018-04-17 10:24:37 EDT ---

Root caused the issue, issue is due to secret stored as Unicode string. I will work on it and send patch by tomorrow.

Comment 1 Worker Ant 2018-04-18 15:19:49 UTC
REVIEW: https://review.gluster.org/19900 (eventsapi: Handle Unicode string during signing) posted (#1) for review on master by Aravinda VK

Comment 2 Worker Ant 2018-04-20 05:09:45 UTC
COMMIT: https://review.gluster.org/19900 committed in master by "Amar Tumballi" <amarts@redhat.com> with a commit message- eventsapi: Handle Unicode string during signing

Python 2.7 HMAC does not support Unicode strings. Secret is read
from file so it is possible that glustereventsd reads the content
as Unicode. This patch converts the secret to `str` type before
generating HMAC signature.

Fixes: bz#1568820
Change-Id: I7daa64499ac4ca02544405af26ac8af4b6b0bd95
Signed-off-by: Aravinda VK <avishwan@redhat.com>

Comment 3 Shyamsundar 2018-06-20 18:05:09 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report.

glusterfs-v4.1.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-June/000102.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.