Infoblox -> Nautobot job fatally fails on 404 not found

[MidnightRamenX]

Environment

• Python version: Python 3.9.18
• Nautobot version: 2.3.10
• nautobot-ssot version: 2.8.1

Expected Behavior

Infoblox -> Nautobot job should gracefully handle a 404 not found on an import job.

Observed Behavior

Fatal failure with the following error and stack trace.

 Value	
{
    "exc_message": [
        "404 Client Error: Not Found for url: https://<infoblox_dn>/wapi/v2.12/record:ptr/ZG5<obscured_ref>ZWR1:<ip_addr>.in-addr.arpa/default?_return_fields=name%2Cptrdname%2Cipv4addr%2Cipv6addr%2Cview%2Ccomment. {'Error': 'AdmConDataNotFoundError: Reference record:ptr/ZG5<obscured_ref>ZWR1:<ip_arr>.in-addr.arpa/default not found', 'code': 'Client.Ibap.Data.NotFound', 'text': 'Reference record:ptr/ZG5<obscured_ref>ZWR1:<ip_arr>.in-addr.arpa/default not found'}"
    ],
    "exc_module": "requests.exceptions",
    "exc_type": "HTTPError"
}

Traceback (most recent call last):
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/utils/client.py", line 194, in _request
    resp.raise_for_status()
  File "/opt/nautobot/lib64/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://<infoblox_dn>/wapi/v2.12/record:ptr/ZG5<obscured_ref>ZWR1:<ip_arr>.in-addr.arpa/default?_return_fields=name%2Cptrdname%2Cipv4addr%2Cipv6addr%2Cview%2Ccomment

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobot/lib64/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/nautobot/lib64/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
    result = job(*args, **kwargs)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot/extras/jobs.py", line 149, in __call__
    return self.run(*args, **deserialized_kwargs)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/jobs.py", line 106, in run
    super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 317, in run
    self.sync_data(memory_profiling)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 136, in sync_data
    self.load_source_adapter()
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/jobs.py", line 88, in load_source_adapter
    self.source_adapter.load()
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/diffsync/adapters/infoblox.py", line 430, in load
    self.load_ipaddresses()
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/diffsync/adapters/infoblox.py", line 289, in load_ipaddresses
    self._load_dns_ptr_record_for_ip(ref=ptr_record_ref, ip_record=new_ip, namespace=namespace)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/diffsync/adapters/infoblox.py", line 356, in _load_dns_ptr_record_for_ip
    ptr_record = self.conn.get_ptr_record_by_ref(ref)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/utils/client.py", line 866, in get_ptr_record_by_ref
    response = self._request("GET", path=url_path, params=params)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/utils/client.py", line 197, in _request
    raise HTTPError(exc_msg, response=err.response) from err
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://<infoblox_dn>/wapi/v2.12/record:ptr/ZG5<obscured_ref>ZWR1:<ip_arr>.in-addr.arpa/default?_return_fields=name%2Cptrdname%2Cipv4addr%2Cipv6addr%2Cview%2Ccomment. {'Error': 'AdmConDataNotFoundError: Reference record:ptr/ZG5<obscured_ref>ZWR1:<ip_arr>.in-addr.arpa/default not found', 'code': 'Client.Ibap.Data.NotFound', 'text': 'Reference record:ptr/ZG5<obscured_ref>ZWR1:<ip_arr>.in-addr.arpa/default not found'}

Steps to Reproduce

1. Run the Infoblox -> Nautobot Job
2. Wait 30-40 minutes while job runs
3. Job fails

I'm unsure of what environmental causes might be causing this issue. It "may" be that the IP address is assigned to a DHCP lease after the job starts, but I can't verify that. I have verified that every instance of an A record failing, the IP address was part of a DHCP lease range and that the record was created around the same time the job started.

[chrislopezunt]

#922 / #934 doesn't fix this problem. There is a code issue in the logger:

File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_ssot/integrations/infoblox/diffsync/adapters/infoblox.py", line 342, in _load_dns_a_record_for_ip self.job.logger.warning(message=f"A record {ref} not found, likely dynamic and expired.") TypeError: warning() missing 1 required positional argument: 'msg'

The logger should instead look like:
self.job.logger.warning(f"A record {ref} not found, likely dynamic and expired.")