Bug 1753587

Summary: https://build.gluster.org/job/compare-bug-version-and-git-branch/41059/ fails for public BZ
Product: [Community] GlusterFS Reporter: Nithya Balachandran <nbalacha>
Component: project-infrastructureAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6CC: bugs, dkhandel, gluster-infra, mscherer, pasik, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-27 06:04:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nithya Balachandran 2019-09-19 11:09:51 UTC
Description of problem:
https://build.gluster.org/job/compare-bug-version-and-git-branch/41059/ fails for a public bug.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Niels de Vos 2019-09-19 13:00:12 UTC
The error seems to be related to some missing Python package:

ImportError: No module named 'requests.packages.urllib3'


Maybe these additional RPM needs to be installed:
- python-requests.noarch : HTTP library, written in Python, for human beings
- python-urllib3.noarch : Python HTTP library with thread-safe connection pooling and file post

With those RPMs installed, the package requests.packages.urllib3 can be imported.

Comment 2 M. Scherer 2019-09-19 15:03:30 UTC
python-requests is installed, and so is python-urllib3. But it seems there was some fluck and we have a faulty package. I will fix.

Comment 3 M. Scherer 2019-09-19 15:36:37 UTC
ok, so I suspect it might have been partially due to a mix of pip package (for github3-py) and the rpms.  let's see if my cleanup is working.

Comment 4 M. Scherer 2019-09-20 10:30:45 UTC
so, now we face a interesting issue, gerrit produce invalid json:

~ $ curl -s "https://review.gluster.org/changes/glusterfs~master~I4fbb5c924fb3a144e415d2368126b784dde760ea/revisions/1/commit" |head -n 5
)]}'
{
  "commit": "13f8432734f8fd3fe59f05a48d297a7e72c33301",
  "parents": [
    {


Why, I have no idea. We flushed caches, etc just in case, and I will look at the DB side, but maybe we will need to restart gerrit quickly before I get to dig the source code.

Comment 5 Deepshikha khandelwal 2019-09-20 11:07:30 UTC
And this issue is not even persistent. We are getting random invalid JSON when using gerrit API and sometimes not.

Comment 6 Niels de Vos 2019-09-20 11:48:40 UTC
I think Gerrit does it on purpose. Quite a while ago I have used Gerrit(Hub) and scripts needed to drop the 1st line 'magic' value before parsing the response.

Comment 7 Niels de Vos 2019-09-20 11:53:26 UTC
From https://review.gluster.org/Documentation/rest-api.html#output :

To prevent against Cross Site Script Inclusion (XSSI) attacks, the JSON response body starts with a magic prefix line that must be stripped before feeding the rest of the response body to a JSON parser:

  )]}'
  [ ... valid JSON ... ]

Comment 8 M. Scherer 2019-09-20 12:48:06 UTC
mhh, then why did it broke suddenly ?

Comment 9 M. Scherer 2019-09-20 12:49:21 UTC
ok, so that explain the cleaned_output line in the code. So I guess we are back on the drawing board to figure why it break

Comment 10 M. Scherer 2019-09-20 14:28:36 UTC
now I can't reproduce much, so I have left debugging output and will wait until next run

Comment 11 Yaniv Kaul 2019-09-23 06:41:13 UTC
(In reply to M. Scherer from comment #10)
> now I can't reproduce much, so I have left debugging output and will wait
> until next run

It just happened to me several times, so it's certainly happening.
See https://build.gluster.org/job/bugzilla-post/16502/console

Comment 12 M. Scherer 2019-09-23 07:13:04 UTC
ok, so I am quite puzzled. The request made with python fail, but not with curl. But at least I can see why it give weird result.

Comment 13 M. Scherer 2019-09-23 07:59:02 UTC
So it seems the issue is that request do convert ~ to %7E and curl doesn't. After looking for a while, it seems that pip did decide to erase some of the rpm provided file, thus result in a weridly almost working mix, that broke when i tried to fix. I did clean everything and now, it should be working. We still need to make sure that github3-py do not interfere (ideally, the less we use pip, the better, cause this result into this kind of problem :/), but for now, I think thing should work.

Comment 14 Deepshikha khandelwal 2020-02-27 06:04:27 UTC
This seems to be working fine till now. Thank you, Michael.

Closing it.