FULL PRODUCT VERSION :
java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"

A DESCRIPTION OF THE PROBLEM :
The race condition is as manifests as following:

One thread calls isAnnotationPresent on an annotated class where the annotation is not yet initialised for its defining classloader. This will result in a call on AnnotationType.getInstance, locking the class object for sun.reflect.annotation.AnnotationType. getInstance will result in a Class.initAnnotationsIfNecessary for that annotation, trying to acquire a lock on the class object of that annotation.

In the meanwhile, another thread has requested Class.getAnnotations for that annotation(!). Since getAnnotations locks the class object it was requested on, the first thread can't lock it when it runs into Class.initAnnotationsIfNecessary for that annotation. But the thread holding the lock will try to acquire the lock for the class object of sun.reflect.annotation.AnnotationType in AnnotationType.getInstance which is hold by the first thread, thus resulting in the deadlock.

Below is a kill -3 thread dump from the program I mention in "Steps to Reproduce":

Found one Java-level deadlock:
=============================
"pool-1-thread-2":
  waiting to lock monitor 0x00007f89d4026248 (object 0x000000073ecac700, a java.lang.Class),
  which is held by "pool-1-thread-1"
"pool-1-thread-1":
  waiting to lock monitor 0x0000000041f16cb0 (object 0x000000073ecc9f88, a java.lang.Class),
  which is held by "pool-1-thread-2"

Java stack information for the threads listed above:
===================================================
"pool-1-thread-2":
at java.lang.Class.initAnnotationsIfNecessary(Class.java:3067)
- waiting to lock <0x000000073ecac700> (a java.lang.Class for annotationinitialisationrace.domain.Annotation)
at java.lang.Class.getAnnotation(Class.java:3029)
at sun.reflect.annotation.AnnotationType.<init>(AnnotationType.java:113)
at sun.reflect.annotation.AnnotationType.getInstance(AnnotationType.java:66)
- locked <0x000000073ecc9f88> (a java.lang.Class for sun.reflect.annotation.AnnotationType)
at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:202)
at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
at java.lang.Class.initAnnotationsIfNecessary(Class.java:3070)
- locked <0x000000073ecb2c28> (a java.lang.Class for annotationinitialisationrace.domain.AnnotatedClass)
at java.lang.Class.getAnnotation(Class.java:3029)
at java.lang.Class.isAnnotationPresent(Class.java:3042)
at annotationinitialisationrace.run.IsAnnotationPresentRunnable.synchronisedRun(IsAnnotationPresentRunnable.java:19)
at annotationinitialisationrace.run.SynchronisedRunnableTemplate.run(SynchronisedRunnableTemplate.java:24)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"pool-1-thread-1":
at sun.reflect.annotation.AnnotationType.getInstance(AnnotationType.java:63)
- waiting to lock <0x000000073ecc9f88> (a java.lang.Class for sun.reflect.annotation.AnnotationType)
at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:202)
at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
at java.lang.Class.initAnnotationsIfNecessary(Class.java:3070)
- locked <0x000000073ecac700> (a java.lang.Class for annotationinitialisationrace.domain.Annotation)
at java.lang.Class.getAnnotations(Class.java:3050)
at annotationinitialisationrace.run.GetAnnotationsRunnable.synchronisedRun(GetAnnotationsRunnable.java:18)
at annotationinitialisationrace.run.SynchronisedRunnableTemplate.run(SynchronisedRunnableTemplate.java:24)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.

In the real world, this deadlock was provoked for us by a Spring application context initialisation in an OSGi container (Spring DM):

   at java.lang.Class.getAnnotations(Class.java:3050)
   at org.springframework.core.type.classreading.AnnotationMetadataReadingVisitor$1.visitEnd(AnnotationMetadataReadingVisitor.java:79)
   at org.springframework.asm.ClassReader.a(Unknown Source)
   at org.springframework.asm.ClassReader.accept(Unknown Source)
   at org.springframework.asm.ClassReader.accept(Unknown Source)
   at org.springframework.core.type.classreading.SimpleMetadataReader.getAnnotationMetadata(SimpleMetadataReader.java:55)

while another thread did the Class.isAnnotationPresent call for an anotated class.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Clone and compile https://bitbucket.org/fakraemer/annotation-initialisation-race, I was reliably running into the deadlock on every execution (4 cores).

If not, I'd suggest starting the program with a debugger attached, breaking annotationinitialisationrace.run.GetAnnotationsRunnable l18 and java.lang.Class.getAnnotation(Class<A>) l3029 (see above for the JSE I used, it should be on the initAnnotationsIfNecessary() call. You'll just have to step over the first hit on getAnnotation, thus you'll end up locking sun.reflect.annotation.AnnotationType but not yet annotationinitialisationrace.domain.Annotation. Then you switch over to the other thread which must be suspended on GetAnnotationsRunnable l18, and just resume it. It should be blocked now due to the wait for the AnnotationType lock. Then you switch over to the other thread and resume it, thus this thread should wait for locking annotationinitialisationrace.domain.Annotation.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Also the situation is rather exotic, I would expect the synchronisation to work since both calls are part of the standard reflection API.

REPRODUCIBILITY :
This bug can be reproduced occasionally.