Errors and inconsistent data when deleting cables terminated to circuits

[glennmatthews]

Environment

• Python version:
• Nautobot version: 1.3.2

Steps to Reproduce

1. Create a device and add interfaces to it
2. Create a circuit and add A and Z terminations to it
3. Connect the A termination to an interface of the device
4. Connect the Z termination to an interface of the device
5. Attempt to disconnect either termination

Expected Behavior

Cable to be deleted

Observed Behavior

Cable is reported as deleted, but still exists in the UI, even after a cacheops.invalidate_all() is run. On attempting to disconnect the cable a second time, the following traceback is seen:

14:56:07.429 ERROR   django.request :
  Internal Server Error: /dcim/cables/7f3a3352-0fb7-43f9-9e85-971037f553f3/delete/
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: '_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/source/nautobot/utilities/views.py", line 94, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/source/nautobot/core/views/generic.py", line 456, in post
    obj.delete()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 967, in delete
    return collector.delete()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/deletion.py", line 436, in delete
    sender=model, instance=obj, using=self.using
  File "/usr/local/lib/python3.7/site-packages/django/dispatch/dispatcher.py", line 182, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.7/site-packages/django/dispatch/dispatcher.py", line 182, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/source/nautobot/dcim/signals.py", line 177, in nullify_connected_endpoints
    instance.termination_b.save()
  File "/source/nautobot/dcim/models/device_components.py", line 512, in save
    return super().save(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 740, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 789, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "/usr/local/lib/python3.7/site-packages/django/dispatch/dispatcher.py", line 182, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.7/site-packages/django/dispatch/dispatcher.py", line 182, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/source/nautobot/utilities/utils.py", line 354, in _curried
    return _curried_func(*args, *moreargs, **{**kwargs, **morekwargs})
  File "/source/nautobot/extras/signals.py", line 72, in _handle_changed_object
    objectchange = instance.to_objectchange(action)
  File "/source/nautobot/dcim/models/device_components.py", line 93, in to_objectchange
    object_data_v2=serialize_object_v2(self),
  File "/source/nautobot/utilities/utils.py", line 181, in serialize_object_v2
    data = serializer_class(obj, context={"request": None}).data
  File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 555, in data
    ret = super().data
  File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 253, in data
    self._data = self.to_representation(self.instance)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 522, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "/usr/local/lib/python3.7/site-packages/rest_framework/fields.py", line 1886, in to_representation
    return method(value)
  File "/source/nautobot/dcim/api/serializers.py", line 163, in get_connected_endpoint
    if obj._path is not None and obj._path.destination is not None:
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 351, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 437, in get
    self.model._meta.object_name
nautobot.dcim.models.cables.CablePath.DoesNotExist: CablePath matching query does not exist.

[jathanism]

I confirmed that this behavior occurs both with or without caching enabled, so it's absolutely a bug in the business logic.

[jathanism]

Possibly related: Suor/django-cacheops#348 (comment)

[jathanism]

Under normal circumstances Interface._path should return None when unassigned. There's some state inconsistency that is being introduced here where the queryset is attempting to retrieve the value of _path from the internal field cache on the instance and it's failing. The pattern for deletion of the CablePath object is to be at play here and needs to be reviewed.

[jathanism]

Let's try to remove cacheops from the equation entirely and see if this issue persists, just to rule it out.

[jathanism]

Let's try to remove cacheops from the equation entirely and see if this issue persists, just to rule it out.

Still crashed. Here is a Sentry report for it: https://sentry.io/share/issue/59b831e46b4f4cc69485c49f97302654/

[glennmatthews]

Appears to be a pretty tangled chain of function calls what with a delete() call triggering a post_delete signal triggering a post_save signal...

[timber-schroeder]

👍 just ran into this one also on 1.3.2, with identical replication steps.

I have had CACHEOPS_ENABLED = False set since a long time before the circuit+termination in question existed so yeah its definitely not caching related.

Worked around for now in the few instances I had by just manually deleting all the relevant references and rows in the db.

[dd1245]

👍 just ran into this one also on 1.3.2, with identical replication steps.

I have had CACHEOPS_ENABLED = False set since a long time before the circuit+termination in question existed so yeah its definitely not caching related.

Worked around for now in the few instances I had by just manually deleting all the relevant references and rows in the db.

I think I'll have to do something manual as well, I can't really keep waiting for a fix. Which tables did you have to make manual delete operations on?