Abstract: Virtual threads can deadlock, just like platform threads. Depending on what state they are in, this might be quite challenging to analyze. In this newsletter we explore some tricks on how to find and solve them.
Welcome to the 302nd edition of The Java(tm) Specialists' Newsletter. Summer is in full swing here on Crete, which means that instead of running on Kalathas Beach, we get to jog down to our stunning Tersanas Beach for a refresher. The municipality is upgrading the area and has created a riverbed straight down to the beach from our house. It is not a road, but our trusty little Suzuki Jimny makes short work of it. And it gives us amazing access to a quiet, beautiful little beach with cool clean water. What's not to like?
javaspecialists.teachable.com: Please visit our new self-study course catalog to see how you can upskill your Java knowledge.
After my last newsletter Gazillion Virtual Threads, I thought it was high time to update my concurrency course to include virtual threads. This new 4-day course does not make assumptions about how much the Java programmer already knows about good concurrent programming. We start at the beginning, and show how virtual threads and Project Loom fit in. As I was looking at my section on deadlocks, I began to wonder how these would manifest in virtual threads? Here is a simple class that demonstrates a lock-ordering deadlock, in this case between a platform and a virtual thread:
import java.lang.management.ManagementFactory; import java.lang.management.ThreadMXBean; import java.util.concurrent.Phaser; import java.util.concurrent.locks.ReentrantLock; public class SimpleLockOrderingDeadlockMixedThreads { public static void main(String... args) throws InterruptedException { var monitor1 = new Object(); var monitor2 = new Object(); var coop = new Phaser(2); Thread.ofPlatform().name("platform").start(() -> { synchronized (monitor1) { coop.arriveAndAwaitAdvance(); synchronized (monitor2) { System.out.println("All's well"); } } }); Thread.ofVirtual().name("virtual").start(() -> { synchronized (monitor2) { coop.arriveAndAwaitAdvance(); synchronized (monitor1) { System.out.println("All's well too"); } } }); Thread.sleep(100); ThreadMXBean tmb = ManagementFactory.getThreadMXBean(); long[] deadlocks = tmb.findDeadlockedThreads(); System.out.println("deadlocks = " + deadlocks); } }
When we run the code, we see the output:
deadlocks = null
. Currently, the method
findDeadlockedThreads()
specifically excludes
deadlocks that involve virtual threads. Furthermore, with
jstack
, the deadlock is also not explicitly
shown. Here is the abridged jstack output:
"platform" #30 cpu=1.75ms elapsed=4.42s waiting for monitor entry java.lang.Thread.State: BLOCKED (on object monitor) at SimpleLockOrderingDeadlockMixedThreads.lambda$main$0 - waiting to lock <0x000000043fce3d90> (a java.lang.Object) - locked <0x000000043fce3d80> (a java.lang.Object) at SimpleLockOrderingDeadlockMixedThreads$$Lambda$14 at java.lang.Thread.run "ForkJoinPool-1-worker-1" #32 daemon cpu=0.70ms elapsed=4.41s Carrying virtual thread #31 at jdk.internal.vm.Continuation.run at java.lang.VirtualThread.runContinuation at java.lang.VirtualThread$$Lambda$22 at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec at java.util.concurrent.ForkJoinTask.doExec at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec at java.util.concurrent.ForkJoinPool.scan at java.util.concurrent.ForkJoinPool.runWorker at java.util.concurrent.ForkJoinWorkerThread.run
We can see from the thread dump that the platform thread is
blocked waiting for a monitor 0x000000043fce3d90
,
but we do not see which thread holds that monitor. We also see
that ForkJoinPool-1-worker-1
, which is one of the
virtual thread carrier threads, is busy carrying virtual
thread #31, but we do not know from this stack dump what that
thread #31 is actually doing. We also do not see what monitors
are held by this carrier thread. This is not an oversight by
the authorsof Project Loom. In previous versions of Java, we
had a few thousand platform threads at most. But now we can
have millions of virtual threads! Even if the costs are only
linear, that is still a 1000x increase in potential costs.
Hopefully in the future the stack dumps will include the full
information, making it easier to diagnose deadlocks involving
virtual threads.
Let's continue our investigation. If we take a thread dump
and see that a normal platform thread is BLOCKED on a
monitor, but that monitor does not appear elsewhere in the
thread dump, we can suspect that it is held by a virtual
thread, especially if we see a carrier thread that is not
moving forward. With the current implementation of virtual
threads, all the carrier threads are from a fork join pool,
thus carrier threads would be named accordingly, e.g.
ForkJoinPool-1-worker-1
. The carrier thread
where the CPU time does not change over several thread dumps
is likely our culprit.
Once we have identified the carrier thread that is probably
to blame, we can see from the jstack
output
which virtual thread number it is currently carrying. In our
case, this is number #31. But how do we find what that
virtual thread #31 is doing?
Now it gets tricky. Unfortunately there is no way to find
that out without restarting our JVM with
-Djdk.trackAllThreads=true
.
Deadlocks are often hard to reproduce, such that if we have a
system that seems to deadlock, it might be better to always
run it with that setting. If we cannot reproduce the deadlock
after restarting with trackAllThreads
, then at
least we have the half of the deadlock that is in a platform
thread and we could find the solution from that. Assuming
that the deadlock occurs again, we now use
jcmd pid Thread.dump_to_file some_file
and then search for thread #31:
#31 "virtual" virtual SimpleLockOrderingDeadlockMixedThreads.lambda$main$1\ (SimpleLockOrderingDeadlockMixedThreads.java:22) java.base/java.lang.VirtualThread.run java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0 java.base/jdk.internal.vm.Continuation.enter0 java.base/jdk.internal.vm.Continuation.enter
If we have monitor deadlocks between multiple virtual threads,
then it becomes a bit harder to recognize the deadlock,
because we do not have one platform thread in the BLOCKED state.
We would see several carrier threads in what appears to be a
deadlocked state, and would have to again use jcmd
to find the virtual threads that are to blame.
The bad news is that the carrier thread will remain BLOCKED until the JVM is restarted. Usually we do not have that many deadlocks in our system as to use up all the carrier threads, but it could happen.
Slightly harder to discover are owned lock deadlocks with
ReentrantLock
. When the carrier thread
encounters the lock()
method, it swaps out the
virtual thread until the lock become available. Thus
jstack
will not show a stuck carrier thread. It
will only show the platform thread trying to lock an owned
lock. But there could be many reasons why this happens. For
example, another thread might have forgotten to unlock.
In this case, one possibility is to subclass
ReentrantLock
to reveal who owns the lock we
are trying to acquire, and to then combine that with a call
to tryLock()
. If that fails, then we reveal who
owns the lock:
import java.util.concurrent.locks.ReentrantLock; public class ReentrantLockOwnerReveal extends ReentrantLock { @Override public Thread getOwner() { return super.getOwner(); } @Override public String toString() { var owner = getOwner(); return super.toString() + ((owner == null) ? "" : " tid=" + owner.threadId()); } }
As I mentioned before, deadlocks are currently more difficult to discover in virtual threads than in platform threads. No doubt this will be addressed in future version of virtual threads. In the meantime, I hope this newsletter will help if you get stuck. As always, prevention is better than the cure, and in our Java Concurrency Course, we show coding techniques to avoid getting into deadlocks in the first place.
Kind regards
Heinz
P.S. Thanks again to Alan Bateman and the folks on the Loom Dev Mailing list for pointing me in the right direction on how to solve this :-)
We are always happy to receive comments from our readers. Feel free to send me a comment via email or discuss the newsletter in our JavaSpecialists Slack Channel (Get an invite here)
We deliver relevant courses, by top Java developers to produce more resourceful and efficient programmers within their organisations.