Bug 1768840 - Gerrit is slow and fails to upload patches
Summary: Gerrit is slow and fails to upload patches
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: GlusterFS
Classification: Community
Component: project-infrastructure
Version: mainline
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-05 11:12 UTC by Xavi Hernandez
Modified: 2019-11-29 07:59 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-29 07:59:01 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Xavi Hernandez 2019-11-05 11:12:14 UTC
Description of problem:

When navigating through the gerrit web interface, response is very slow.

Trying to upload new patches is also slow and fails at the end with the following error:

     ! [remote rejected]     HEAD -> refs/for/master/ref-738 (internal server error: Error inserting change/patchset)
    error: failed to push some refs to 'ssh://xhernandez@review.gluster.org/glusterfs'

However the patch appears on gerrit and it seems fine. Not sure what failed exactly.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 M. Scherer 2019-11-05 14:34:33 UTC
so, it seems we have been getting a lot of requests on the web, i did blacklist crudely the range doing them. Not sure what we can do to protect from that right now, since the system is using different IP. Did things got better after doing that ?

Comment 2 Xavi Hernandez 2019-11-05 15:51:38 UTC
Navigation seems better now, but the error when patches are being uploaded is still there.

Comment 3 Xavi Hernandez 2019-11-05 15:58:46 UTC
The test patch I've just uploaded doesn't seem to appear in gerrit web. The other ones appeared regardless of the error.

Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 323 bytes | 323.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   https://review.gluster.org/#/c/glusterfs/+/23666 Test patch [WIP]
remote: 
To ssh://review.gluster.org/glusterfs
 ! [remote rejected]     HEAD -> refs/for/master/rfc (internal server error: Error inserting change/patchset)
error: failed to push some refs to 'ssh://xhernandez@review.gluster.org/glusterfs'

But going to the patch url gives a 404.

Navigation is still a bit slow, but seems faster than this morning.

Comment 4 Atin Mukherjee 2019-11-06 03:24:46 UTC
The problem still persists. Even I can't merge a patch from gerrit today.

Comment 5 Deepshikha khandelwal 2019-11-06 06:17:53 UTC
Looking at the error logs on gerrit server, I see 'Update aborted with one or more lock failures' 

[2019-11-06 01:27:45,187] [ReceiveCommits-2] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : [glusterfs-1573003654511-4ddffffd]Can't insert change/patch set for glusterfs
com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset
        at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:288)
        at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:279)
        at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:728)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:551)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:212)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:191)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:50)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:558)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.gerrit.server.update.UpdateException: com.google.gerrit.server.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[
  CREATE: 0000000000000000000000000000000000000000 79de40cec94dd117e869b55d1e1fde83f38ae401 refs/changes/72/23672/1  (LOCK_FAILURE)
  CREATE: 0000000000000000000000000000000000000000 995fbc9acb1cba9366eda955d86821791265bc78 refs/changes/72/23672/meta  (REJECTED_OTHER_REASON: transaction aborted)
]

I'm not sure how to deal with this. @misc, does it require any lock files cleanup?

Comment 6 M. Scherer 2019-11-06 10:41:03 UTC
So, I did stopped gerrit, did the git gc myself, and restarted. it didn't worked, because it wasn't really dead (I was under the impression that we were using systemd for gerrit, seems this wasn't the case).

So I restarted the server, the UI is back, can someone try a push ?

Comment 7 Xavi Hernandez 2019-11-06 11:49:49 UTC
Yes, I've been able to push a new patch without errors. Navigation also seems to be working fine.

I'm moving the bug to verified.

Comment 8 Xavi Hernandez 2019-11-29 07:59:01 UTC
The issue has been fixed. Closing the bug.


Note You need to log in before you can comment on or make changes to this bug.