Can't leave ghost party on new server

Migrated my old save to a new server and now im stuck in a ghost party.

When trying to leave the ghost party i get this: https://i.imgur.com/zeUH6Mh.png

It seems like the party is being pulled from the client side.

How do I fix this?

