Bug 1568820

+++ This bug was initially created as a clone of Bug #1466129 +++

Description of problem:
Currently, there's no way to ascertain the identity/validity of a webhook requests sent by gluster-eventsd. If a Header was added that contains an HMAC token generated using content and secret key, this would ensure that server can validate the request is genuine before processing the event.


Version-Release number of selected component (if applicable):
3.3

How reproducible:
Always

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-29 02:47:37 EDT ---

This bug is automatically being proposed for the current release of Red Hat Gluster Storage 3 under active development, by setting the release flag 'rhgs‑3.3.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-07-06 08:48:25 EDT ---

This BZ having been considered, and subsequently not approved to be fixed at the RHGS 3.3.0 release, is being proposed for the next minor release of RHGS

--- Additional comment from Aravinda VK on 2017-09-27 00:54:42 EDT ---

As discussed over IRC, Events API will implement JWT(JSON Web Tokens). Additional argument will be added to webhook-add command to accept the secret which is required to generate JWT.

    gluster-eventsapi webhook-add <url> [--token <TOKEN>] \
        [--secret <SECRET>]

For shared secret approach use, `--secret` and for shared token approach use `--token`. With `--token` argument, Token header will be added as is.

    Authorization: Bearer <TOKEN>

In case of shared secret, Gluster will generate JWT token using the secret and then add it to Authorization header.

    Authorization: Bearer <GENERATED_TOKEN>

Secret/Token can be updated using `webhook-mod` command.

Generated token will include the following payload,

    {
        "iss": "gluster",
        "exp": EXPIRY_TIME,
        "sub": EVENT_TYPE,
        "iat": EVENT_TIME
    }

Where: iss - Issuer,  exp - Expiry Time, sub - Event Type used as Subject, iat - Event Time used as Issue Time

--- Additional comment from Aravinda VK on 2017-09-27 04:01:18 EDT ---

Upstream patch posted https://review.gluster.org/#/c/18405

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-11-13 10:35:36 EST ---

Since this bug has has been approved for the RHGS 3.4.0 release of Red Hat Gluster Storage 3, through release flag 'rhgs-3.4.0+', and through the Internal Whiteboard entry of '3.4.0', the Target Release is being automatically set to 'RHGS 3.4.0'

--- Additional comment from Atin Mukherjee on 2017-12-27 05:34:15 EST ---

downstream patch : https://code.engineering.redhat.com/gerrit/126551

--- Additional comment from Aravinda VK on 2017-12-28 04:54:25 EST ---

Another patch sent to Upstream which fixes the external library dependency issue.

https://review.gluster.org/19102

--- Additional comment from Sunil Kumar Acharya on 2017-12-28 05:25:13 EST ---

Second Downstream Patch:

https://code.engineering.redhat.com/gerrit/#/c/126619/

--- Additional comment from Sweta Anandpara on 2018-04-16 02:53:29 EDT ---

Validated this on the build glusterfs-3.12.2-7.el7rhgs.x86_64.

On a 3 node cluster with a webhook-added with default fields, we are able to see the events.. But the same webhook when added using the "--secret" field, no events are seen at the receiver. /var/log/glusterfs/events.log does show the event successfully being pushed, though. There's no indication of any error in the events logs. 

/var/log/messages shows the below traceback in glustereventsd.

Apr 16 02:26:59 dhcp47-187 glustereventsd: Exception in thread Thread-50:
Apr 16 02:26:59 dhcp47-187 glustereventsd: Traceback (most recent call last):
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Apr 16 02:26:59 dhcp47-187 glustereventsd: self.run()
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/lib64/python2.7/threading.py", line 765, in run
Apr 16 02:26:59 dhcp47-187 glustereventsd: self.__target(*self.__args, **self.__kwargs)
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/libexec/glusterfs/events/utils.py", line 247, in publish_to_webhook
Apr 16 02:26:59 dhcp47-187 glustereventsd: hashval = get_jwt_token(secret, event_type, event_ts)
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/libexec/glusterfs/events/utils.py", line 209, in get_jwt_token
Apr 16 02:26:59 dhcp47-187 glustereventsd: base64_urlencode(hmac.HMAC(secret, msg, sha256).digest())
Apr 16 02:26:59 dhcp47-187 glustereventsd: File "/usr/lib64/python2.7/hmac.py", line 75, in __init__
Apr 16 02:26:59 dhcp47-187 glustereventsd: self.outer.update(key.translate(trans_5C))
Apr 16 02:26:59 dhcp47-187 glustereventsd: TypeError: character mapping must return integer, None or unicode

Moving this bug to assigned for the reason stated above. Sosreports of the server node, and the webhook logs will be copied at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/swetas/<bugnumber>

--- Additional comment from Sweta Anandpara on 2018-04-16 03:25:08 EDT ---

[qe@rhsqe-repo 1466129]$ hostname
rhsqe-repo.lab.eng.blr.redhat.com
[qe@rhsqe-repo 1466129]$ 
[qe@rhsqe-repo 1466129]$ pwd
/var/www/html/sosreports/swetas/1466129
[qe@rhsqe-repo 1466129]$ 
[qe@rhsqe-repo 1466129]$ ll
total 36616
-rwxr-xr-x. 1 qe qe 37490200 Apr 16 12:54 sosreport-dhcp47-187.lab.eng.blr.redhat.com-20180416025148.tar.xz
-rwxr-xr-x. 1 qe qe      533 Apr 16 12:22 webhook.py
[qe@rhsqe-repo 1466129]$

--- Additional comment from Aravinda VK on 2018-04-17 10:24:37 EDT ---

Root caused the issue, issue is due to secret stored as Unicode string. I will work on it and send patch by tomorrow.