Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workstealing can lead to deadlocks #3945

Closed
1 task
Tracked by #3990
leonard84 opened this issue Sep 6, 2024 · 2 comments · Fixed by #3981
Closed
1 task
Tracked by #3990

Workstealing can lead to deadlocks #3945

leonard84 opened this issue Sep 6, 2024 · 2 comments · Fixed by #3981
Assignees
@marcphilipp @leonard84
Labels
component: Platform theme: concurrency type: bug
Milestone
5.11.1

Comments

@leonard84
Copy link
Collaborator

Steps to reproduce

This does not work every time as it is dependent on multiple factors.

  • Checkout spockframework/spock@b69f9f1
  • run ./gradlew :spock-specs:test --rerun -Dvariant=3.0 -DjavaVersion=21
  • You might need to try a few times. This depends a bit on your machine, but if it takes longer than 5m then it is most likely stuck.
  • You can attach a debugger to inspect the current executing tests

What is Broken?

As you can see in these two screenshots we switch from executing MockSpecInfoAnnotationSpec to ByteBuddyMockFactoryConcurrentSpec due to workstealing. The problem is that MockSpecInfoAnnotationSpec holds a read-lock for the org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEY but ByteBuddyMockFactoryConcurrentSpec needs a write-lock.

image
image

Context

  • Used versions (Jupiter/Vintage/Platform): 5.11.0/5.12.0-SNAPSHOT
  • Build Tool/IDE: Gradle

Deliverables

  • ...
@leonard84
Copy link
Collaborator Author

Idea: We can use a ThreadLocal to mark which locks are held and then either throw or somehow else reschedule an exclusive task that is incompatible. IIRC this can only happen with the GLOBAL_KEY in the first place, as a lock on a test class forces all children to be executed on the same thread.

@leonard84
Copy link
Collaborator Author

image

leonard84 added a commit to leonard84/junit5 that referenced this issue Sep 11, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService
3131277 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterward.

fixes junit-team#3945

Co-authored-by: Marc Philipp <[email protected]>
leonard84 added a commit to leonard84/junit5 that referenced this issue Sep 11, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService
6da042f 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterward.

fixes junit-team#3945

Co-authored-by: Marc Philipp <[email protected]>
@leonard84 leonard84 mentioned this issue Sep 11, 2024
Merged
6 tasks
leonard84 added a commit to leonard84/junit5 that referenced this issue Sep 11, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService
b2922c5 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterward.

fixes junit-team#3945

Co-authored-by: Marc Philipp <[email protected]>
leonard84 added a commit to leonard84/junit5 that referenced this issue Sep 11, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService
d6428b8 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterward.

fixes junit-team#3945

Co-authored-by: Marc Philipp <[email protected]>
@marcphilipp marcphilipp added this to the 5.11.1 milestone Sep 13, 2024
@sbrannen sbrannen modified the milestones: 5.10.4, 5.11.1 Sep 13, 2024
@sbrannen sbrannen mentioned this issue Sep 13, 2024
Closed
4 tasks
marcphilipp added a commit that referenced this issue Sep 17, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService (#3981)
88f8859 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterwards.

Fixes #3945.

---------

Co-authored-by: Marc Philipp <[email protected]>
marcphilipp added a commit that referenced this issue Sep 17, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService (#3981)
8e0e13f 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterwards.

Fixes #3945.

---------

Co-authored-by: Marc Philipp <[email protected]>
marcphilipp pushed a commit that referenced this issue Sep 23, 2024
@leonard84 @marcphilipp
Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService (#3981)
932fe5c 
The service now checks if the `ExclusiveTask` that should run
is executed on a thread that is already executing another task.
If this is scenario is detected, it checks if the lock is compatible to
the enclosing locks.
1. If compatible, it is executed and marked done
2. If incompatible, it is added to a list of deferred tasks and left
   unfinished. The deferred tasks will be re-forked afterwards.

Fixes #3945.

---------

Co-authored-by: Marc Philipp <[email protected]>
(cherry picked from commit 88f8859)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marcphilipp marcphilipp

@leonard84 leonard84

Labels
component: Platform theme: concurrency type: bug
Projects
None yet
Milestone
5.11.1
Development

Successfully merging a pull request may close this issue.

Fix deadlock in ForkJoinPoolHierarchicalTestExecutorService leonard84/junit5
3 participants
@sbrannen @marcphilipp @leonard84