Bug 1323963

Summary:	ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE when deleting a project in a cluster
Product:	[Retired] JBoss BRMS Platform 6	Reporter:	Toshiya Kobayashi <tkobayas>
Component:	Business Central	Assignee:	Eder Ignatowicz <eignatow>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jiri Locker <jlocker>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.1.0
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-04-05 08:19:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Toshiya Kobayashi 2016-04-05 08:03:26 UTC

Description of problem:

Environment: BRMS 6.1.0, 2 node domain mode cluster, 1 helix/zookeeper.

If you delete a project in one node of the cluster, you get "org.eclipse.jgit.api.errors.ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE".

=======
[Server:server-one] 16:42:02,517 ERROR [org.jboss.as.ejb3.invocation] (Thread-92) JBAS014134: EJB Invocation failed on component IOWatchServiceExecutorImpl for method public abstract void org.uberfire.backend.server.io.watch.IOWatchServiceExecutor.execute(org.uberfire.java.nio.file.WatchKey,org.uberfire.backend.server.util.Filter): javax.ejb.EJBException: java.lang.RuntimeException: org.eclipse.jgit.api.errors.ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE
[Server:server-one] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:217) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
...
[Server:server-one] 	at org.uberfire.backend.server.io.watch.IOWatchServiceExecutor$$$view2.execute(Unknown Source) [uberfire-backend-server-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.backend.server.io.watch.AbstractIOWatchService$2.execute(AbstractIOWatchService.java:145) [uberfire-backend-server-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.backend.server.io.watch.AbstractIOWatchService$1.run(AbstractIOWatchService.java:86) [uberfire-backend-server-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_67]
[Server:server-one] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_67]
[Server:server-one] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_67]
[Server:server-one] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_67]
[Server:server-one] 	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_67]
[Server:server-one] Caused by: java.lang.RuntimeException: org.eclipse.jgit.api.errors.ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE
[Server:server-one] 	at org.uberfire.java.nio.fs.jgit.util.JGitUtil.commit(JGitUtil.java:608) [uberfire-nio2-jgit-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.java.nio.fs.jgit.util.JGitUtil.commit(JGitUtil.java:506) [uberfire-nio2-jgit-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.java.nio.fs.jgit.JGitFileSystemProvider.commit(JGitFileSystemProvider.java:1853) [uberfire-nio2-jgit-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.java.nio.fs.jgit.JGitFileSystemProvider.access$600(JGitFileSystemProvider.java:147) [uberfire-nio2-jgit-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.java.nio.fs.jgit.JGitFileSystemProvider$4.close(JGitFileSystemProvider.java:836) [uberfire-nio2-jgit-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.io.impl.AbstractIOService.write(AbstractIOService.java:609) [uberfire-io-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.io.impl.AbstractIOService.write(AbstractIOService.java:498) [uberfire-io-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.io.impl.AbstractIOService.write(AbstractIOService.java:506) [uberfire-io-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.uberfire.io.impl.cluster.IOServiceClusterImpl.write(IOServiceClusterImpl.java:775) [uberfire-io-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	at org.kie.workbench.common.screens.explorer.backend.server.ExplorerServiceImpl.cleanup(ExplorerServiceImpl.java:463) [kie-wb-common-project-explorer-backend-6.2.0.Final-redhat-4.jar:6.2.0.Final-redhat-4]
[Server:server-one] 	at org.kie.workbench.common.screens.explorer.backend.server.ExplorerServiceImpl.onProjectDelete(ExplorerServiceImpl.java:449) [kie-wb-common-project-explorer-backend-6.2.0.Final-redhat-4.jar:6.2.0.Final-redhat-4]
...
[Server:server-one] 	at org.kie.workbench.common.screens.explorer.backend.server.ExplorerServiceImpl$Proxy$_$$_WeldClientProxy.onProjectDelete(ExplorerServiceImpl$Proxy$_$$_WeldClientProxy.java) [kie-wb-common-project-explorer-backend-6.2.0.Final-redhat-4.jar:6.2.0.Final-redhat-4]
...
[Server:server-one] 	at org.guvnor.common.services.project.backend.server.PathObserverService.fireDeleteEvent(PathObserverService.java:63) [guvnor-project-backend-6.2.0.Final-redhat-4.jar:6.2.0.Final-redhat-4]
[Server:server-one] 	at org.guvnor.common.services.project.backend.server.PathObserverService.onBatchResourceChanges(PathObserverService.java:46) [guvnor-project-backend-6.2.0.Final-redhat-4.jar:6.2.0.Final-redhat-4]
[Server:server-one] 	at org.guvnor.common.services.project.backend.server.PathObserverService$Proxy$_$$_WeldClientProxy.onBatchResourceChanges(PathObserverService$Proxy$_$$_WeldClientProxy.java) [guvnor-project-backend-6.2.0.Final-redhat-4.jar:6.2.0.Final-redhat-4]
...
[Server:server-one] 	at org.uberfire.backend.server.io.watch.IOWatchServiceExecutorImpl.execute(IOWatchServiceExecutorImpl.java:110) [uberfire-backend-server-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
...
[Server:server-one] Caused by: org.eclipse.jgit.api.errors.ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE
[Server:server-one] 	at org.uberfire.java.nio.fs.jgit.util.JGitUtil.commit(JGitUtil.java:593) [uberfire-nio2-jgit-0.5.3.Final-redhat-1.jar:0.5.3.Final-redhat-1]
[Server:server-one] 	... 122 more
========

Because of the combination with BZ1323572, AsyncWatchService becomes unfunctional and servers get various issues.

How reproducible:

Always

Steps to Reproduce:
1. Start a BRMS cluster
2. Login to business-central
3. Create a project
4. Delete a project

Actual results:

"org.eclipse.jgit.api.errors.ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE" is thrown

Expected results:

No exception.

Comment 2 Toshiya Kobayashi 2016-04-05 08:19:51 UTC

This LOCK_FAILURE is caused by committing "/explorer/user.nav" of "upstream/brmsAdmin-uf-user". So the problem is that such a remote branch is retrieved in ExplorerServiceImpl.cleanup()

https://github.com/droolsjbpm/kie-wb-common/blob/6.1.x/kie-wb-common-screens/kie-wb-common-project-explorer/kie-wb-common-project-explorer-backend/src/main/java/org/kie/workbench/common/screens/explorer/backend/server/ExplorerServiceImpl.java#L959

Remote branches are retrieved because JGitFileSystemProvider.rescanForExistingRepositories() instantiates JGitFileSystem with ListMode.ALL.

https://github.com/uberfire/uberfire/blob/0.5.x/uberfire-nio2-backport/uberfire-nio2-impls/uberfire-nio2-jgit/src/main/java/org/uberfire/java/nio/fs/jgit/JGitFileSystemProvider.java#L397

This was accidentally fixed by another BZ.

https://bugzilla.redhat.com/show_bug.cgi?id=1269333
https://github.com/uberfire/uberfire/commit/a72a56efb7242749526cbf5ce3c5a03c86f66a39

So this issue no longer exists in BRMS 6.2.0.

I filed this BZ for record. I'm closing this BZ.