so, it seems we have been getting a lot of requests on the web, i did blacklist crudely the range doing them. Not sure what we can do to protect from that right now, since the system is using different IP. Did things got better after doing that ?

Navigation seems better now, but the error when patches are being uploaded is still there.

The test patch I've just uploaded doesn't seem to appear in gerrit web. The other ones appeared regardless of the error.

Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 323 bytes | 323.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   https://review.gluster.org/#/c/glusterfs/+/23666 Test patch [WIP]
remote: 
To ssh://review.gluster.org/glusterfs
 ! [remote rejected]     HEAD -> refs/for/master/rfc (internal server error: Error inserting change/patchset)
error: failed to push some refs to 'ssh://xhernandez.org/glusterfs'

But going to the patch url gives a 404.

Navigation is still a bit slow, but seems faster than this morning.

The problem still persists. Even I can't merge a patch from gerrit today.

Looking at the error logs on gerrit server, I see 'Update aborted with one or more lock failures' 

[2019-11-06 01:27:45,187] [ReceiveCommits-2] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : [glusterfs-1573003654511-4ddffffd]Can't insert change/patch set for glusterfs
com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset
        at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:288)
        at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:279)
        at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:728)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:551)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:212)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:191)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:50)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:558)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.gerrit.server.update.UpdateException: com.google.gerrit.server.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[
  CREATE: 0000000000000000000000000000000000000000 79de40cec94dd117e869b55d1e1fde83f38ae401 refs/changes/72/23672/1  (LOCK_FAILURE)
  CREATE: 0000000000000000000000000000000000000000 995fbc9acb1cba9366eda955d86821791265bc78 refs/changes/72/23672/meta  (REJECTED_OTHER_REASON: transaction aborted)
]

I'm not sure how to deal with this. @misc, does it require any lock files cleanup?

So, I did stopped gerrit, did the git gc myself, and restarted. it didn't worked, because it wasn't really dead (I was under the impression that we were using systemd for gerrit, seems this wasn't the case).

So I restarted the server, the UI is back, can someone try a push ?

Yes, I've been able to push a new patch without errors. Navigation also seems to be working fine.

I'm moving the bug to verified.

The issue has been fixed. Closing the bug.