Failed Lease. Can't leave or join VG

[TProhofsky]

One of my server nodes in the cluster crashed while part of a VG. Some time went by and one of the other nodes lost their lease. Now I have an active LV on a node but no lock for the VG of that LV:

[root@node3 ~]# lvs
  VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
  Reading VG sbvg_drbdpool without a lock.
  LV                 VG            Attr       LSize    Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert
  csilv2eg47tjvu91qf sbvg_datalake rwi---r---  100.00g
  csilvnglz2r7n6i85  sbvg_datalake rwi---r--- 1000.00g
  csilvpvr6bs1lps8u  sbvg_datalake -wi-------  100.00g
  poctest_00000      sbvg_drbdpool Vwi-a-tz--   50.01g thinpool        5.74
  thinpool           sbvg_drbdpool twi-aotz--  931.31g                 0.31   10.55
[root@node3 ~]#

Attemping to start the lock completes immediately but has no effect:

[root@node3 ~]# /usr/sbin/vgchange --lock-start --lock-opt auto sbvg_drbdpool
  Starting locking.  Waiting until locks are ready...
[root@node3 ~]# lvs
  VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
  Reading VG sbvg_drbdpool without a lock.
  LV                 VG            Attr       LSize    Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert
  csilv2eg47tjvu91qf sbvg_datalake rwi---r---  100.00g
  csilvnglz2r7n6i85  sbvg_datalake rwi---r--- 1000.00g
  csilvpvr6bs1lps8u  sbvg_datalake -wi-------  100.00g
  poctest_00000      sbvg_drbdpool Vwi-a-tz--   50.01g thinpool        5.74
  thinpool           sbvg_drbdpool twi-aotz--  931.31g                 0.31   10.55
[root@node3 ~]#

Attempting to stop the locks thinking that might clear it shows this:

[root@node3 ~]# /usr/sbin/vgchange --lock-stop sbvg_drbdpool
  VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
  Reading VG sbvg_drbdpool without a lock.
  VG sbvg_drbdpool stop failed: LVs must first be deactivated

Drop and force did not clear issue:

 lvmlockctl --drop sbvg_datalake
  vgchange --lock-type none --lockopt force sbvg_drbdpool
  vgchange --lock-start --lock-opt auto sbvg_drbdpool

[TProhofsky]

Attempting to try it again after Node 1's issue with systemd was fix showed this:

[root@node3 ~]# /usr/sbin/vgchange --lock-start --lock-opt auto sbvg_drbdpool
  Starting locking.  Waiting until locks are ready...
[root@node3 ~]# lvs

Broadcast message from systemd-journald@node3 (Thu 2020-09-17 07:47:33 CDT):

lvmlockctl[15154]: Lost access to sanlock lease storage in VG sbvg_datalake.


Broadcast message from systemd-journald@node3 (Thu 2020-09-17 07:47:33 CDT):

lvmlockctl[15154]: Immediately deactivate LVs in VG sbvg_datalake.


Broadcast message from systemd-journald@node3 (Thu 2020-09-17 07:47:33 CDT):

lvmlockctl[15154]: Once VG is unused, run lvmlockctl --drop sbvg_datalake.


Message from syslogd@node3 at Sep 17 07:47:33 ...
 lvmlockctl:Immediately deactivate LVs in VG sbvg_datalake.

Message from syslogd@node3 at Sep 17 07:47:33 ...
 lvmlockctl:Once VG is unused, run lvmlockctl --drop sbvg_datalake.
  VG sbvg_drbdpool lock skipped: storage failed for sanlock leases
  Reading VG sbvg_drbdpool without a lock.
  LV                 VG            Attr       LSize    Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert
  csilv2eg47tjvu91qf sbvg_datalake rwi---r---  100.00g
  csilvnglz2r7n6i85  sbvg_datalake rwi---r--- 1000.00g
  csilvpvr6bs1lps8u  sbvg_datalake -wi-------  100.00g
  poctest_00000      sbvg_drbdpool Vwi-a-tz--   50.01g thinpool        5.74
  thinpool           sbvg_drbdpool twi-aotz--  931.31g                 0.31   10.55

[TProhofsky]

node3.lvmlockd.dump.txt