SDX-controller is not deleting Links upon receiving the topology update

[gretelliz]

When trying to remove a link from the network, SDX does not reflect the change.

This is the topology before removing a link:

{'name': 'Tenet-OXP', 'id': 'urn:sdx:topology:tenet.ac.za', 'version': 9, 'timestamp': '2024-10-18T19:10:49Z', 'model_version': '2.0.0', 'nodes': [{'name': 'Tenet02', 'id': 'urn:sdx:node:tenet.ac.za:Tenet02', 'location': {'address': 'Johanesburgo', 'latitude': -26.0, 'longitude': 28.0, 'iso3166_2_lvl4': 'ZA-GP', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet02:41', 'name': 'Tenet02-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax02:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:1', 'name': 'Tenet02-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:50', 'name': 'Tenet02-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet03', 'id': 'urn:sdx:node:tenet.ac.za:Tenet03', 'location': {'address': 'EastLondon', 'latitude': -33.0, 'longitude': 27.0, 'iso3166_2_lvl4': 'ZA-EC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet03:50', 'name': 'Tenet03-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet03:2', 'name': 'Tenet03-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet01', 'id': 'urn:sdx:node:tenet.ac.za:Tenet01', 'location': {'address': 'CapeTown', 'latitude': -33.0, 'longitude': 18.0, 'iso3166_2_lvl4': 'ZA-WC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet01:41', 'name': 'Tenet01-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax01:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:1', 'name': 'Tenet01-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:2', 'name': 'Tenet01-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:50', 'name': 'Tenet01-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}], 'links': [{'name': 'Tenet01/1_Tenet02/1', 'id': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'ports': ['urn:sdx:port:tenet.ac.za:Tenet01:1', 'urn:sdx:port:tenet.ac.za:Tenet02:1'], 'type': 'intra', 'bandwidth': 10, 'residual_bandwidth': 100, 'latency': 0, 'packet_loss': 0, 'availability': 0, 'status': 'up', 'state': 'enabled', 'private': ['packet_loss']}, {'name': 'Tenet01/2_Tenet03/2', 'id': 'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2', 'ports': ['urn:sdx:port:tenet.ac.za:Tenet01:2', 'urn:sdx:port:tenet.ac.za:Tenet03:2'], 'type': 'intra', 'bandwidth': 10, 'residual_bandwidth': 100, 'latency': 0, 'packet_loss': 0, 'availability': 0, 'status': 'up', 'state': 'enabled', 'private': ['packet_loss']}], 'services': ['l2vpn-ptp']}

This is the topology after removing a link:

{'name': 'Tenet-OXP', 'id': 'urn:sdx:topology:tenet.ac.za', 'version': 10, 'timestamp': '2024-10-18T19:12:09Z', 'model_version': '2.0.0', 'nodes': [{'name': 'Tenet02', 'id': 'urn:sdx:node:tenet.ac.za:Tenet02', 'location': {'address': 'Johanesburgo', 'latitude': -26.0, 'longitude': 28.0, 'iso3166_2_lvl4': 'ZA-GP', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet02:41', 'name': 'Tenet02-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax02:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:1', 'name': 'Tenet02-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:50', 'name': 'Tenet02-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet03', 'id': 'urn:sdx:node:tenet.ac.za:Tenet03', 'location': {'address': 'EastLondon', 'latitude': -33.0, 'longitude': 27.0, 'iso3166_2_lvl4': 'ZA-EC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet03:50', 'name': 'Tenet03-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet03:2', 'name': 'Tenet03-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'down', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet01', 'id': 'urn:sdx:node:tenet.ac.za:Tenet01', 'location': {'address': 'CapeTown', 'latitude': -33.0, 'longitude': 18.0, 'iso3166_2_lvl4': 'ZA-WC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet01:41', 'name': 'Tenet01-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax01:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:1', 'name': 'Tenet01-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:2', 'name': 'Tenet01-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'down', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:50', 'name': 'Tenet01-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}], 'links': [{'name': 'Tenet01/1_Tenet02/1', 'id': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'ports': ['urn:sdx:port:tenet.ac.za:Tenet01:1', 'urn:sdx:port:tenet.ac.za:Tenet02:1'], 'type': 'intra', 'bandwidth': 10, 'residual_bandwidth': 100, 'latency': 0, 'packet_loss': 0, 'availability': 0, 'status': 'up', 'state': 'enabled', 'private': ['packet_loss']}], 'services': ['l2vpn-ptp']}

The link Tenet01/2_Tenet03/2 was removed. However, this is not reflected in the topology that SDX controller exports.

[YufengXin]

@congwang09 @lmarinve you've tested the link removal/failure case, please comment here.

@gretelliz Thanks for reporting this. The event passes through from kytos->Napps->LC->controlller.

[YufengXin]

Closed with PR # #341

[italovalcy]

Hi team, as we discussed on Slack, this issue was not actually fixed.

[italovalcy]

Hi Cong,

So basically the procedure for reproducing this issue was:

1. Start a topology using the normal procedure as documented on https://sdx-docs.readthedocs.io/en/latest/sdx_deploy_single_server.html
2. On Mininet, add a new link and enabled that new link on Kytos. Once you do that, SDX-LC will send a updated topology for SDX-Controller with the new link for example the JSON below:

{'name': 'Tenet-OXP', 'id': 'urn:sdx:topology:tenet.ac.za', 'version': 8, 'timestamp': '2024-10-25T20:40:59Z', 'model_version': '2.0.0', 'nodes': [{'name': 'Tenet02', 'id': 'urn:sdx:node:tenet.ac.za:Tenet02', 'location': {'address': 'Johanesburgo', 'latitude': -26.0, 'longitude': 28.0, 'iso3166_2_lvl4': 'ZA-GP', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet02:41', 'name': 'Tenet02-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax02:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:1', 'name': 'Tenet02-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:50', 'name': 'Tenet02-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet03', 'id': 'urn:sdx:node:tenet.ac.za:Tenet03', 'location': {'address': 'EastLondon', 'latitude': -33.0, 'longitude': 27.0, 'iso3166_2_lvl4': 'ZA-EC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet03:50', 'name': 'Tenet03-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet03:2', 'name': 'Tenet03-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet01', 'id': 'urn:sdx:node:tenet.ac.za:Tenet01', 'location': {'address': 'CapeTown', 'latitude': -33.0, 'longitude': 18.0, 'iso3166_2_lvl4': 'ZA-WC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet01:41', 'name': 'Tenet01-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax01:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:1', 'name': 'Tenet01-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:2', 'name': 'Tenet01-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:50', 'name': 'Tenet01-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}], 'links': [{'name': 'Tenet01/1_Tenet02/1', 'id': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'ports': ['urn:sdx:port:tenet.ac.za:Tenet01:1', 'urn:sdx:port:tenet.ac.za:Tenet02:1'], 'type': 'intra', 'bandwidth': 10, 'residual_bandwidth': 100, 'latency': 0, 'packet_loss': 0, 'availability': 0, 'status': 'up', 'state': 'enabled', 'private': ['packet_loss']}, {'name': 'Tenet01/2_Tenet03/2', 'id': 'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2', 'ports': ['urn:sdx:port:tenet.ac.za:Tenet01:2', 'urn:sdx:port:tenet.ac.za:Tenet03:2'], 'type': 'intra', 'bandwidth': 10, 'residual_bandwidth': 100, 'latency': 0, 'packet_loss': 0, 'availability': 0, 'status': 'up', 'state': 'enabled', 'private': ['packet_loss']}], 'services': ['l2vpn-ptp']}

On the Topology JSON above you will notice two links:

>>> y["links"][0]["id"]
'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1'
>>> y["links"][1]["id"]
'urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2'

3. Then, we take steps to disable and remove the link on the OXP (simulating one link being decommissioned, for instance), and then the SDX-LC will send another update to SDX-Controller with the link removed, as you can see on the JSON below:

{'name': 'Tenet-OXP', 'id': 'urn:sdx:topology:tenet.ac.za', 'version': 9, 'timestamp': '2024-10-25T20:41:03Z', 'model_version': '2.0.0', 'nodes': [{'name': 'Tenet02', 'id': 'urn:sdx:node:tenet.ac.za:Tenet02', 'location': {'address': 'Johanesburgo', 'latitude': -26.0, 'longitude': 28.0, 'iso3166_2_lvl4': 'ZA-GP', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet02:41', 'name': 'Tenet02-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax02:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:1', 'name': 'Tenet02-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet02:50', 'name': 'Tenet02-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet02', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet03', 'id': 'urn:sdx:node:tenet.ac.za:Tenet03', 'location': {'address': 'EastLondon', 'latitude': -33.0, 'longitude': 27.0, 'iso3166_2_lvl4': 'ZA-EC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet03:50', 'name': 'Tenet03-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet03:2', 'name': 'Tenet03-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet03', 'type': '10GE', 'status': 'down', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}, {'name': 'Tenet01', 'id': 'urn:sdx:node:tenet.ac.za:Tenet01', 'location': {'address': 'CapeTown', 'latitude': -33.0, 'longitude': 18.0, 'iso3166_2_lvl4': 'ZA-WC', 'private': []}, 'ports': [{'id': 'urn:sdx:port:tenet.ac.za:Tenet01:41', 'name': 'Tenet01-eth41', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:port:sax.net:Sax01:41', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:1', 'name': 'Tenet01-eth1', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:2', 'name': 'Tenet01-eth2', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'down', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}, {'id': 'urn:sdx:port:tenet.ac.za:Tenet01:50', 'name': 'Tenet01-eth50', 'node': 'urn:sdx:node:tenet.ac.za:Tenet01', 'type': '10GE', 'status': 'up', 'state': 'enabled', 'mtu': 1500, 'nni': '', 'services': {'l2vpn-ptp': {'vlan_range': [[1, 4095]]}}, 'private': ['status']}], 'status': 'up', 'state': 'enabled'}], 'links': [{'name': 'Tenet01/1_Tenet02/1', 'id': 'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1', 'ports': ['urn:sdx:port:tenet.ac.za:Tenet01:1', 'urn:sdx:port:tenet.ac.za:Tenet02:1'], 'type': 'intra', 'bandwidth': 10, 'residual_bandwidth': 100, 'latency': 0, 'packet_loss': 0, 'availability': 0, 'status': 'up', 'state': 'enabled', 'private': ['packet_loss']}], 'services': ['l2vpn-ptp']}

You can see that now this json contains only one link:

>>> len(x["links"])
1
>>> x["links"][0]["id"]
'urn:sdx:link:tenet.ac.za:Tenet01/1_Tenet02/1'

So, expected behavior is: the link with ID urn:sdx:link:tenet.ac.za:Tenet01/2_Tenet03/2 should be removed from SDX-Controller. However, the SDX-Controller is not removing that link.

I'm also attaching the logs from SDX-Controller with relevant information
logs-sdx-controller.txt