networktocode.nautobot.cable tries to use device_id in query to find a device, results in failure.

[scooterx3]

ISSUE TYPE

• Bug Report

SOFTWARE VERSIONS

pynautobot

2.1.0

Ansible:

core 2.14.0

Nautobot:

v2.0.5

Collection:

5.1.1

SUMMARY

Using networktocode.nautobot.cable results in error msg: '{"device_id":["Unknown filter field"]}'

STEPS TO REPRODUCE

Assuming the above versions, create a device with a power outlet (riley-server-1 in my examples) and another device with a power port (riley-outlet-1 in my examples)

Run the example test_cable.yml playbook:

ansible-playbook -i inventory/inventory.yaml test_cable.yml -vvvvvvvvvvvvvv

---
- name: Test cable connection
  hosts: <some single host in your inventory file>
  connection: local
  gather_facts: false
  tasks:
        
    - name: Plug power ports in
      networktocode.nautobot.cable:
        url: "{{ nautobot_url }}"
        token: "{{ nautobot_token }}"
        termination_a_type: dcim.powerport
        termination_a:
          device: "riley-server-1"
          name: "pwr0"
        termination_b_type: dcim.poweroutlet
        termination_b:
          device: "riley-outlet-1"
          name: "out0"
        status: Connected
        state: present

EXPECTED RESULTS

I expected a simple change that resulted in a cable connection like this:

ACTUAL RESULTS

Operation failed, returned message msg: '{"device_id":["Unknown filter field"]}'

$ ansible-playbook -i inventory/inventory.yaml test_cable.yml -vvvvvvvvvvvvvv
ansible-playbook [core 2.14.0]
  config file = /home/rloader/code/nautobot-ansible/ansible.cfg
  configured module search path = ['/home/rloader/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rloader/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/rloader/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/rloader/.local/bin/ansible-playbook
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
Using /home/rloader/code/nautobot-ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/rloader/code/nautobot-ansible/inventory/inventory.yaml as it did not pass its verify_file() method
script declined parsing /home/rloader/code/nautobot-ansible/inventory/inventory.yaml as it did not pass its verify_file() method
Skipping empty key (vars) in group (n10)
Skipping empty key (vars) in group (pod538)
Parsed /home/rloader/code/nautobot-ansible/inventory/inventory.yaml inventory source with yaml plugin
Loading collection networktocode.nautobot from /home/rloader/.ansible/collections/ansible_collections/networktocode/nautobot
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Loading collection community.general from /home/rloader/.local/lib/python3.10/site-packages/ansible_collections/community/general
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Loading callback plugin community.general.yaml of type stdout, v2.0 from /home/rloader/.local/lib/python3.10/site-packages/ansible_collections/community/general/plugins/callback/yaml.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: test_cable.yml ******************************************************************************************************************************************************************************************
Positional arguments: test_cable.yml
verbosity: 14
remote_user: root
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/rloader/code/nautobot-ansible/inventory/inventory.yaml',)
forks: 5
1 plays in test_cable.yml

PLAY [Test cable] *************************************************************************************************************************************************************************************************

TASK [Plug power ports in] ****************************************************************************************************************************************************************************************
task path: /home/rloader/code/nautobot-ansible/test_cable.yml:7
<riley-server-1> ESTABLISH LOCAL CONNECTION FOR USER: rloader
<riley-server-1> EXEC /bin/sh -c 'echo ~rloader && sleep 0'
<riley-server-1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/rloader/.ansible/tmp `"&& mkdir "` echo /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000 `" && echo ansible-tmp-1709330119.1504107-1296406-77223189425000="` echo /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000 `" ) && sleep 0'
Including module_utils file ansible/__init__.py
Including module_utils file ansible/module_utils/__init__.py
Including module_utils file ansible/module_utils/basic.py
Including module_utils file ansible/module_utils/_text.py
Including module_utils file ansible/module_utils/common/_collections_compat.py
Including module_utils file ansible/module_utils/common/__init__.py
Including module_utils file ansible/module_utils/common/_json_compat.py
Including module_utils file ansible/module_utils/common/_utils.py
Including module_utils file ansible/module_utils/common/arg_spec.py
Including module_utils file ansible/module_utils/common/file.py
Including module_utils file ansible/module_utils/common/locale.py
Including module_utils file ansible/module_utils/common/parameters.py
Including module_utils file ansible/module_utils/common/collections.py
Including module_utils file ansible/module_utils/common/process.py
Including module_utils file ansible/module_utils/common/sys_info.py
Including module_utils file ansible/module_utils/common/text/converters.py
Including module_utils file ansible/module_utils/common/text/__init__.py
Including module_utils file ansible/module_utils/common/text/formatters.py
Including module_utils file ansible/module_utils/common/validation.py
Including module_utils file ansible/module_utils/common/warnings.py
Including module_utils file ansible/module_utils/compat/selectors.py
Including module_utils file ansible/module_utils/compat/__init__.py
Including module_utils file ansible/module_utils/compat/_selectors2.py
Including module_utils file ansible/module_utils/compat/selinux.py
Including module_utils file ansible/module_utils/distro/__init__.py
Including module_utils file ansible/module_utils/distro/_distro.py
Including module_utils file ansible/module_utils/errors.py
Including module_utils file ansible/module_utils/parsing/convert_bool.py
Including module_utils file ansible/module_utils/parsing/__init__.py
Including module_utils file ansible/module_utils/pycompat24.py
Including module_utils file ansible/module_utils/six/__init__.py
Including module_utils file ansible_collections/networktocode/nautobot/plugins/module_utils/dcim.py
Including module_utils file ansible_collections/__init__.py
Including module_utils file ansible_collections/networktocode/__init__.py
Including module_utils file ansible_collections/networktocode/nautobot/__init__.py
Including module_utils file ansible_collections/networktocode/nautobot/plugins/__init__.py
Including module_utils file ansible_collections/networktocode/nautobot/plugins/module_utils/__init__.py
Including module_utils file ansible_collections/networktocode/nautobot/plugins/module_utils/utils.py
Including module_utils file ansible/module_utils/urls.py
Including module_utils file ansible/module_utils/compat/typing.py
<riley-server-1> Attempting python interpreter discovery
<riley-server-1> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'python3.11'"'"'; command -v '"'"'python3.10'"'"'; command -v '"'"'python3.9'"'"'; command -v '"'"'python3.8'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<riley-server-1> EXEC /bin/sh -c '/usr/bin/python3.10 && sleep 0'
Using module file /home/rloader/.ansible/collections/ansible_collections/networktocode/nautobot/plugins/modules/cable.py
<riley-server-1> PUT /home/rloader/.ansible/tmp/ansible-local-1296401m94rlqo1/tmp854_idx7 TO /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000/AnsiballZ_cable.py
<riley-server-1> EXEC /bin/sh -c 'chmod u+x /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000/ /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000/AnsiballZ_cable.py && sleep 0'
<riley-server-1> EXEC /bin/sh -c '/usr/bin/python3 /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000/AnsiballZ_cable.py && sleep 0'
<riley-server-1> EXEC /bin/sh -c 'rm -f -r /home/rloader/.ansible/tmp/ansible-tmp-1709330119.1504107-1296406-77223189425000/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/tmp/ansible_networktocode.nautobot.cable_payload_vtrnsjkx/ansible_networktocode.nautobot.cable_payload.zip/ansible_collections/networktocode/nautobot/plugins/module_utils/utils.py", line 518, in _nb_endpoint_get
    response = nb_endpoint.get(**query_params)
  File "/home/rloader/.local/lib/python3.10/site-packages/pynautobot/core/endpoint.py", line 156, in get
    filter_lookup = self.filter(**kwargs)
  File "/home/rloader/.local/lib/python3.10/site-packages/pynautobot/core/endpoint.py", line 248, in filter
    return response_loader(req.get(), self.return_obj, self)
  File "/home/rloader/.local/lib/python3.10/site-packages/pynautobot/core/query.py", line 350, in get
    return req_all()
  File "/home/rloader/.local/lib/python3.10/site-packages/pynautobot/core/query.py", line 308, in req_all
    req = self._make_call(add_params=add_params)
  File "/home/rloader/.local/lib/python3.10/site-packages/pynautobot/core/query.py", line 281, in _make_call
    raise RequestError(req)
fatal: [riley-server-1]: FAILED! => changed=false 
  ansible_facts:
    discovered_interpreter_python: /usr/bin/python3
  invocation:
    module_args:
      api_version: null
      color: null
      label: null
      length: null
      length_unit: null
      query_params: null
      state: present
      status: Connected
      termination_a:
        device: riley-server-1
        name: pwr0
      termination_a_type: dcim.powerport
      termination_b:
        device: riley-outlet-1
        name: out0
      termination_b_type: dcim.poweroutlet
      token: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      type: null
      url: http://127.0.0.1:8080
      validate_certs: true
  msg: '{"device_id":["Unknown filter field"]}'

PLAY RECAP ********************************************************************************************************************************************************************************************************
riley-server-1             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[joewesch]

At a first glance I'm going to guess that we need to add termination_a and termination_b here:

nautobot-ansible/plugins/module_utils/utils.py

Lines 357 to 373 in e10829a

	# Some API endpoints dropped '_id' in filter fields in 2.0, ignore them here.
	IGNORE_ADDING_IDS = {
	"ip_address_to_interface",
	"device_bay",
	"inventory_item",
	"circuit_termination",
	"rear_port",
	"front_port",
	"console_port",
	"console_server_port",
	"power_port",
	"power_outlet",
	"dcim.consoleport",
	"dcim.consoleserverport",
	"circuits.circuittermination",
	"services",
	}

[scooterx3]

Thanks for pointing that out. Although adding termination_a and _b didn't work, I did some experimentation. Here's the code that uses IGNORE_ADDING_IDS:

        for match in matches:
            if match in QUERY_PARAMS_IDS and parent not in IGNORE_ADDING_IDS:
                raise Exception(f'{parent = }')
                if child:
                    query_id = self._get_query_param_id(match, child)
                else:
                    query_id = self._get_query_param_id(match, module_data)
                query_dict.update({match + "_id": query_id})
            else:
                if child:
                    value = child.get(match)
                else:
                    value = module_data.get(match)
                query_dict.update({match: value})

This starts on line 664 in the same utils.py file.

Note I added the raise Exception temporarily as a quick and dirty way to understand what the conditional was seeing. It reported the parent as being dcim.powerport.

Looking back in here:

# Some API endpoints dropped '_id' in filter fields in 2.0, ignore them here.
IGNORE_ADDING_IDS = {
    "ip_address_to_interface",
    "device_bay",
    "inventory_item",
    "circuit_termination",
    "rear_port",
    "front_port",
    "console_port",
    "console_server_port",
    "dcim.powerport", #<<<<<
    "dcim.poweroutlet", #<<<<<
    "power_port",
    "power_outlet",
    "dcim.consoleport",
    "dcim.consoleserverport",
    "circuits.circuittermination",
    "services",
}

I see that power_port and power_outlet are present, but if I add dcim.powerport and dcim.poweroutlet then it seems to work. I have no idea what rippling effect this change may have though, so someone else would need to vet this 😁

[joewesch]

Thanks!

I have no idea what rippling effect this change may have though, so someone else would need to vet this

That's what we have tests for 😄.