Move it to ContactManager right next to ShouldCollide for fixtures. See if there is an optimization for the O(n) check on joints.