302Virtual Thread Deadlocks

Author: Dr Heinz M. KabutzDate: 2022-06-30Java Version: 19-ea+28Category: Concurrency

Abstract: Virtual threads can deadlock, just like platform threads. Depending on what state they are in, this might be quite challenging to analyze. In this newsletter we explore some tricks on how to find and solve them.

Welcome to the 302nd edition of The Java(tm) Specialists' Newsletter. Summer is in full swing here on Crete, which means that instead of running on Kalathas Beach, we get to jog down to our stunning Tersanas Beach for a refresher. The municipality is upgrading the area and has created a riverbed straight down to the beach from our house. It is not a road, but our trusty little Suzuki Jimny makes short work of it. And it gives us amazing access to a quiet, beautiful little beach with cool clean water. What's not to like?

javaspecialists.teachable.com: Please visit our new self-study course catalog to see how you can upskill your Java knowledge.

Virtual Thread Deadlocks

After my last newsletter Gazillion Virtual Threads, I thought it was high time to update my concurrency course to include virtual threads. This new 4-day course does not make assumptions about how much the Java programmer already knows about good concurrent programming. We start at the beginning, and show how virtual threads and Project Loom fit in. As I was looking at my section on deadlocks, I began to wonder how these would manifest in virtual threads? Here is a simple class that demonstrates a lock-ordering deadlock, in this case between a platform and a virtual thread:

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;
import java.util.concurrent.Phaser;
import java.util.concurrent.locks.ReentrantLock;

public class SimpleLockOrderingDeadlockMixedThreads {
  public static void main(String... args)
      throws InterruptedException {
    var monitor1 = new Object();
    var monitor2 = new Object();
    var coop = new Phaser(2);
    Thread.ofPlatform().name("platform").start(() -> {
      synchronized (monitor1) {
        coop.arriveAndAwaitAdvance();
        synchronized (monitor2) {
          System.out.println("All's well");
        }
      }
    });
    Thread.ofVirtual().name("virtual").start(() -> {
      synchronized (monitor2) {
        coop.arriveAndAwaitAdvance();
        synchronized (monitor1) {
          System.out.println("All's well too");
        }
      }
    });
    Thread.sleep(100);
    ThreadMXBean tmb = ManagementFactory.getThreadMXBean();
    long[] deadlocks = tmb.findDeadlockedThreads();
    System.out.println("deadlocks = " + deadlocks);
  }
}

When we run the code, we see the output: deadlocks = null. Currently, the method findDeadlockedThreads() specifically excludes deadlocks that involve virtual threads. Furthermore, with jstack, the deadlock is also not explicitly shown. Here is the abridged jstack output:

"platform" #30 cpu=1.75ms elapsed=4.42s waiting for monitor entry
  java.lang.Thread.State: BLOCKED (on object monitor)
    at SimpleLockOrderingDeadlockMixedThreads.lambda$main$0
    - waiting to lock <0x000000043fce3d90> (a java.lang.Object)
    - locked <0x000000043fce3d80> (a java.lang.Object)
    at SimpleLockOrderingDeadlockMixedThreads$$Lambda$14
    at java.lang.Thread.run

"ForkJoinPool-1-worker-1" #32 daemon cpu=0.70ms elapsed=4.41s
  Carrying virtual thread #31
    at jdk.internal.vm.Continuation.run
    at java.lang.VirtualThread.runContinuation
    at java.lang.VirtualThread$$Lambda$22
    at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec
    at java.util.concurrent.ForkJoinTask.doExec
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec
    at java.util.concurrent.ForkJoinPool.scan
    at java.util.concurrent.ForkJoinPool.runWorker
    at java.util.concurrent.ForkJoinWorkerThread.run

We can see from the thread dump that the platform thread is blocked waiting for a monitor 0x000000043fce3d90, but we do not see which thread holds that monitor. We also see that ForkJoinPool-1-worker-1, which is one of the virtual thread carrier threads, is busy carrying virtual thread #31, but we do not know from this stack dump what that thread #31 is actually doing. We also do not see what monitors are held by this carrier thread. This is not an oversight by the authorsof Project Loom. In previous versions of Java, we had a few thousand platform threads at most. But now we can have millions of virtual threads! Even if the costs are only linear, that is still a 1000x increase in potential costs. Hopefully in the future the stack dumps will include the full information, making it easier to diagnose deadlocks involving virtual threads.

Let's continue our investigation. If we take a thread dump and see that a normal platform thread is BLOCKED on a monitor, but that monitor does not appear elsewhere in the thread dump, we can suspect that it is held by a virtual thread, especially if we see a carrier thread that is not moving forward. With the current implementation of virtual threads, all the carrier threads are from a fork join pool, thus carrier threads would be named accordingly, e.g. ForkJoinPool-1-worker-1. The carrier thread where the CPU time does not change over several thread dumps is likely our culprit.

Once we have identified the carrier thread that is probably to blame, we can see from the jstack output which virtual thread number it is currently carrying. In our case, this is number #31. But how do we find what that virtual thread #31 is doing?

Now it gets tricky. Unfortunately there is no way to find that out without restarting our JVM with -Djdk.trackAllThreads=true. Deadlocks are often hard to reproduce, such that if we have a system that seems to deadlock, it might be better to always run it with that setting. If we cannot reproduce the deadlock after restarting with trackAllThreads, then at least we have the half of the deadlock that is in a platform thread and we could find the solution from that. Assuming that the deadlock occurs again, we now use jcmd pid Thread.dump_to_file some_file and then search for thread #31:

#31 "virtual" virtual
  SimpleLockOrderingDeadlockMixedThreads.lambda$main$1\
    (SimpleLockOrderingDeadlockMixedThreads.java:22)
  java.base/java.lang.VirtualThread.run
  java.base/java.lang.VirtualThread$VThreadContinuation.lambda$new$0
  java.base/jdk.internal.vm.Continuation.enter0
  java.base/jdk.internal.vm.Continuation.enter

If we have monitor deadlocks between multiple virtual threads, then it becomes a bit harder to recognize the deadlock, because we do not have one platform thread in the BLOCKED state. We would see several carrier threads in what appears to be a deadlocked state, and would have to again use jcmd to find the virtual threads that are to blame.

The bad news is that the carrier thread will remain BLOCKED until the JVM is restarted. Usually we do not have that many deadlocks in our system as to use up all the carrier threads, but it could happen.

Slightly harder to discover are owned lock deadlocks with ReentrantLock. When the carrier thread encounters the lock() method, it swaps out the virtual thread until the lock become available. Thus jstack will not show a stuck carrier thread. It will only show the platform thread trying to lock an owned lock. But there could be many reasons why this happens. For example, another thread might have forgotten to unlock. In this case, one possibility is to subclass ReentrantLock to reveal who owns the lock we are trying to acquire, and to then combine that with a call to tryLock(). If that fails, then we reveal who owns the lock:

import java.util.concurrent.locks.ReentrantLock;

public class ReentrantLockOwnerReveal extends ReentrantLock {
  @Override
  public Thread getOwner() {
    return super.getOwner();
  }

  @Override
  public String toString() {
    var owner = getOwner();
    return super.toString() + ((owner == null) ? "" :
        " tid=" + owner.threadId());
  }
}

As I mentioned before, deadlocks are currently more difficult to discover in virtual threads than in platform threads. No doubt this will be addressed in future version of virtual threads. In the meantime, I hope this newsletter will help if you get stuck. As always, prevention is better than the cure, and in our Java Concurrency Course, we show coding techniques to avoid getting into deadlocks in the first place.

Kind regards

Heinz

P.S. Thanks again to Alan Bateman and the folks on the Loom Dev Mailing list for pointing me in the right direction on how to solve this :-)

Comments

We are always happy to receive comments from our readers. Feel free to send me a comment via email or discuss the newsletter in our JavaSpecialists Slack Channel (Get an invite here)

Related Articles

Browse the Newsletter Archive