From 7b73df4fcc9e1530cbcb2a2009059932b66d7907 Mon Sep 17 00:00:00 2001 From: James Williams Date: Fri, 15 Nov 2024 14:47:53 -0600 Subject: [PATCH 01/10] update v3 docs --- docs/custom-workflows.md | 36 ++++++ docs/drivers.md | 5 + docs/future-config.md | 118 ++++++++++++++++++ docs/getting-started.md | 47 +++++--- docs/gpt-based-remediation.md | 114 ++++++++++++++++++ docs/install.md | 4 +- docs/junos-style-syntax-remediation.md | 159 +++++++++++++++++++++++++ docs/tags.md | 146 +++++++++++++++++++++++ docs/unified-diff.md | 43 +++++++ mkdocs.yml | 9 +- 10 files changed, 658 insertions(+), 23 deletions(-) create mode 100644 docs/custom-workflows.md create mode 100644 docs/drivers.md create mode 100644 docs/future-config.md create mode 100644 docs/gpt-based-remediation.md create mode 100644 docs/junos-style-syntax-remediation.md create mode 100644 docs/tags.md create mode 100644 docs/unified-diff.md diff --git a/docs/custom-workflows.md b/docs/custom-workflows.md new file mode 100644 index 0000000..9856d86 --- /dev/null +++ b/docs/custom-workflows.md @@ -0,0 +1,36 @@ +# Creating Custom Workflows + +In some scenarios, configuration remediations may require handling beyond the standard negation and idempotency workflows that hier_config is designed to support. Hier Config accommodates these edge cases by allowing the creation of custom remediation workflows, enabling you to integrate them seamlessly with the existing remediation process. + +## Building a Remediation + +1. Import and Configuration Loading + +```python +import os +from hier_config import WorkflowRemediation, get_hconfig +from hier_config.models import Platform +``` +Necessary modules are imported, including the `WorkflowRemediation` class for handling remediation + +```python +running_config = load_device_config("./tests/fixtures/running_config_acl.conf") +generated_config = load_device_config("./tests/fixtures/generated_config_acl.conf") +``` + +Here, the current and intended configurations are loaded from files, serving as inputs for comparison and remediation. + +2. Initialize `WorkflowRemiation`: +```python +wfr = WorkflowRemediation( + running_config=get_hconfig(Platform.CISCO_IOS, running_config), + generated_config=get_hconfig(Platform.CISCO_IOS, generated_config) +) +``` +This initializes `WorkflowRemediation` with the configurations for a Cisco IOS platform, setting up the object that will manage the remediation workflow. + +## Extracting the Targeted Section of a Remediation + +## Building a Custom Remediation + +## Merging the Remediations diff --git a/docs/drivers.md b/docs/drivers.md new file mode 100644 index 0000000..0f87934 --- /dev/null +++ b/docs/drivers.md @@ -0,0 +1,5 @@ +# Drivers + +## Customizing Existing Drivers + +## Creating a Custom Driver diff --git a/docs/future-config.md b/docs/future-config.md new file mode 100644 index 0000000..46a71e9 --- /dev/null +++ b/docs/future-config.md @@ -0,0 +1,118 @@ +# Future Config + +The Future Config feature, introduced in version 2.2.0, attempts to predict the state of the running configuration after a change is applied. + +This feature is useful in scenarios where you need to determine the anticipated configuration state following a change, such as: +- Verifying that a configuration change was successfully applied to a device + - For example, checking if the post-change configuration matches the predicted future configuration +- Generating a future-state configuration that can be analyzed by tools like Batfish to assess the potential impact of a change +- Building rollback configurations: once the future configuration state is known, a rollback configuration can be generated by simply creating the remediation in reverse `(rollback = future.config_to_get_to(running))`. + - When building rollbacks for a series of configuration changes, you can use the future configuration from each change as input for the subsequent change. For example, use the future configuration after Change 1 as the input for determining the future configuration after Change 2, and so on. + ```shell + post_change_1_config = running_config.future(change_1_config) + change_1_rollback_config = post_change_1_config.config_to_get_to(running_config) + post_change_2_config = post_change_1_config.future(change_2_config) + change_2_rollback_config = post_change_2_config.config_to_get_to(post_change_1_config) + ... + ``` + +Currently, this algorithm does not account for: +- negate a numbered ACL when removing an item +- sectional exiting +- negate with +- idempotent command avoid +- idempotent_acl_check +- and likely others + +```bash +>>> from hier_config import get_hconfig +>>> from hier_config.model import Platform +>>> from hier_config.utils import load_device_config +>>> from pprint import pprint +>>> + +>>> running_config_text = load_device_config("./tests/fixtures/running_config.conf") +>>> generated_config_text = load_device_config("./tests/fixtures/remediation_config_without_tags.conf") +>>> +>>> running_config = get_hconfig(Platform.CISCO_IOS, running_config_text) +>>> remediation_config = get_hconfig(Platform.CISCO_IOS, remediation_config_text) +>>> +>>> print("Running Config") +Running Config +>>> for line in running_config.all_children(): +... print(line.cisco_style_text()) +... +hostname aggr-example.rtr +ip access-list extended TEST + 10 permit ip 10.0.0.0 0.0.0.7 any +vlan 2 + name switch_mgmt_10.0.2.0/24 +vlan 3 + name switch_mgmt_10.0.4.0/24 +interface Vlan2 + descripton switch_10.0.2.0/24 + ip address 10.0.2.1 255.255.255.0 + shutdown +interface Vlan3 + mtu 9000 + description switch_mgmt_10.0.4.0/24 + ip address 10.0.4.1 255.255.0.0 + ip access-group TEST in + no shutdown +>>> +>>> print("Remediation Config") +Remediation Config +>>> for line in remediation_config.all_children(): +... print(line.cisco_style_text()) +... +vlan 3 + name switch_mgmt_10.0.3.0/24 +vlan 4 + name switch_mgmt_10.0.4.0/24 +interface Vlan2 + mtu 9000 + ip access-group TEST in + no shutdown +interface Vlan3 + description switch_mgmt_10.0.3.0/24 + ip address 10.0.3.1 255.255.0.0 +interface Vlan4 + mtu 9000 + description switch_mgmt_10.0.4.0/24 + ip address 10.0.4.1 255.255.0.0 + ip access-group TEST in + no shutdown +>>> +>>> print("Future Config") +Future Config +>>> for line in running_config.future(remediation_config).all_children(): +... print(line.cisco_style_text()) +... +vlan 3 + name switch_mgmt_10.0.3.0/24 +vlan 4 + name switch_mgmt_10.0.4.0/24 +interface Vlan2 + mtu 9000 + ip access-group TEST in + descripton switch_10.0.2.0/24 + ip address 10.0.2.1 255.255.255.0 +interface Vlan3 + description switch_mgmt_10.0.3.0/24 + ip address 10.0.3.1 255.255.0.0 + mtu 9000 + ip access-group TEST in + no shutdown +interface Vlan4 + mtu 9000 + description switch_mgmt_10.0.4.0/24 + ip address 10.0.4.1 255.255.0.0 + ip access-group TEST in + no shutdown +hostname aggr-example.rtr +ip access-list extended TEST + 10 permit ip 10.0.0.0 0.0.0.7 any +vlan 2 + name switch_mgmt_10.0.2.0/24 +>>> +``` \ No newline at end of file diff --git a/docs/getting-started.md b/docs/getting-started.md index 9ab78a1..5c73906 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -7,7 +7,10 @@ Hier Config is a Python library that assists with remediating network configurat To use `WorkflowRemediation`, you’ll import it along with `get_hconfig` (for generating configuration objects) and `Platform` (for specifying the operating system driver). ```python -from hier_config import get_hconfig, Platform, WorkflowRemediation +>>> from hier_config import WorkflowRemediation, get_hconfig +>>> from hier_config.model import Platform +>>> from hier_config.utils import load_device_config +>>> ``` With the Host class imported, it can be utilized to create host objects. @@ -18,14 +21,14 @@ Use `get_hconfig` to create HConfig objects for both the running and intended co ```python # Define running and intended configurations as strings -running_config_text = open("./tests/fixtures/running_config.conf").read() -generated_config_text = open("./tests/fixtures/generated_config.conf").read() - +>>> running_config_text = load_device_config("./tests/fixtures/running_config.conf") +>>> generated_config_text = load_device_config("./tests/fixtures/remediation_config.conf") +>>> # Create HConfig objects for running and intended configurations -running_config = get_hconfig(Platform.CISCO_IOS, running_config_text) -generated_config = get_hconfig(Platform.CISCO_IOS, generated_config_text) - +>>> running_config = get_hconfig(Platform.CISCO_IOS, running_config_text) +>>> generated_config = get_hconfig(Platform.CISCO_IOS, generated_config_text) +>>> ``` ## Step 3: Initializing WorkflowRemediation and Generating Remediation @@ -34,18 +37,20 @@ With the HConfig objects created, initialize `WorkflowRemediation` to calculate ```python # Initialize WorkflowRemediation with the running and intended configurations -workflow = WorkflowRemediation(running_config, generated_config) +>>> workflow = WorkflowRemediation(running_config, generated_config) +>>> ``` -### Generating the Remediation Configuration +## Generating the Remediation Configuration -The `remediation_config` attribute generates the configuration needed to apply the intended changes to the device. +The `remediation_config` attribute generates the configuration needed to apply the intended changes to the device. Use `all_children_sorted()` to display the configuration in a readable format: ```python -print(workflow.remediation_config) -``` - -```text +>>> print("Remediation configuration:") +Remediation configuration: +>>> for line in workflow.remediation_config.all_children_sorted(): +... print(line.cisco_style_text()) +... vlan 3 name switch_mgmt_10.0.3.0/24 vlan 4 @@ -63,17 +68,20 @@ interface Vlan4 ip address 10.0.4.1 255.255.0.0 ip access-group TEST in no shutdown +>>> ``` -### Generating the Rollback Configuration +## Generating the Rollback Configuration Similarly, the `rollback_config` attribute generates a configuration that can revert the changes, restoring the device to its original state. ```python -print(workflow.rollback_config) -``` - -```text +# Generate and display the rollback configuration +>>> print("Rollback configuration:") +Rollback configuration: +>>> for line in workflow.rollback_config.all_children_sorted(): +... print(line.cisco_style_text()) +... no vlan 4 no interface Vlan4 vlan 3 @@ -85,4 +93,5 @@ interface Vlan2 interface Vlan3 description switch_mgmt_10.0.4.0/24 ip address 10.0.4.1 255.255.0.0 +>>> ``` \ No newline at end of file diff --git a/docs/gpt-based-remediation.md b/docs/gpt-based-remediation.md new file mode 100644 index 0000000..88fbbeb --- /dev/null +++ b/docs/gpt-based-remediation.md @@ -0,0 +1,114 @@ +# GPT-Based Custom Remediations + +In certain cases, configuration remediations may fall outside of the standard negation and idempotency workflows that `hier_config` is built to handle. Previously, addressing these edge cases required manually crafting specific remediation steps. However, with the advent of AI, particularly generative pre-trained transformers (GPT), it’s now possible to dynamically generate these custom remediations. + +The example below demonstrates how to integrate a GPT client for AI-driven remediations. While this example uses OpenAI's ChatGPT, `hier_config` can work with any compatible GPT model—such as Claude or self-hosted models like Ollama GPT—so long as the client provides similar functionality. + +### Code Walkthrough and Explanation + +1. Import and Configuration Loading +```python +import os +from hier_config import WorkflowRemediation, get_hconfig +from hier_config.utils import load_device_config +from hier_config.models import Platform, GPTRemediationRule, MatchRule, GPTRemediationExample +from hier_config.clients import ChatGPTClient +``` + +Necessary modules are imported, including the `WorkflowRemediation` class for handling remediation, as well as a client interface—in this case, `ChatGPTClient` for connecting to GPT. For other GPT models, you can replace `ChatGPTClient` with your own client class. + +```python +running_config = load_device_config("./tests/fixtures/running_config_acl.conf") +generated_config = load_device_config("./tests/fixtures/generated_config_acl.conf") +``` +Here, the current and intended configurations are loaded from files, serving as inputs for comparison and remediation. + +2. Initialize `WorkflowRemiation`: +```python +wfr = WorkflowRemediation( + running_config=get_hconfig(Platform.CISCO_IOS, running_config), + generated_config=get_hconfig(Platform.CISCO_IOS, generated_config) +) +``` +This initializes `WorkflowRemediation` with the configurations for a Cisco IOS platform, setting up the object that will manage the remediation workflow. + +3. Define a GPT-Based Remediation Rule: +```python +description = ( + "When remediating an access-list on Cisco IOS devices, follow these steps precisely:\n" + " 1. **Resequence the access-list:**\n" + " * **Action**: Resequence the access-list so that each sequence number is a multiple of 10.\n" + " * **Condition**: If a sequence number in the running config isn't divisible by 10, resequence it to the nearest 10, starting at 10.\n" + " 2. **Add Temporary Permit Statement:**\n" + " * **Action**: Insert a temporary `'permit any'` statement at sequence number `1`.\n" + " * **Purpose**: Ensures there's always a valid permit in place during modifications.\n" + " 3. **Apply Required Changes:**\n" + " * **Action**: Update the access-list with the new permit statements as per the generated configuration.\n" + " 4. **Remove Temporary Permit Statement:**\n" + " * **Action**: Remove the temporary `'permit any'` statement added at sequence number `1`.\n" + "\n" + " **Important**:\n" + " **When issuing `no` commands to remove existing entries after resequencing, USE THE" + " RESEQUENCED SEQUENCE NUMBERS, NOT THE ORIGINAL SEQUENCE NUMBERS FROM THE" + " RUNNING CONFIGURATION.**" +) +lineage = (MatchRule(startswith="ip access-list"),) +running = "ip access-list extended TEST\n 12 permit ip host 10.0.0.1 any" +remediation = "ip access-list resequence TEST 10 10\nip access-list extended TEST\n 1 permit ip any any\n no 10\n 10 permit ip host 10.0.0.2 any\n no 1" +example = GPTRemediationExample(running_config=running, remediation_config=remediation) +gpt_rule = GPTRemediationRule(description=description, lineage=lineage, example=example) +``` +Here: + * `description`: Outlines specific remediation steps that GPT should apply. + * `lineage`: Specifies MatchRule criteria to identify access-list configurations. + * `example`: Provides a running configuration and remediation example for added context. + * `gpt_rule`: Combines the description, lineage, and example into a GPTRemediationRule. + +4. Load AI Remediation Rules: +```python +wfr.running_config.driver.gpt_remediation_rules.append(gpt_rule) +``` +This appends the custom rule to `gpt_remediation_rules`, making it available for GPT to apply. + +5. Set Up and Load the GPT Client: +```python +# Example using ChatGPT, but can be replaced with other clients +gpt_client = ChatGPTClient(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o-mini") +wfr.set_gpt_client(gpt_client) +``` + +In this example, `ChatGPTClient` is used for integration with OpenAI's GPT, but you can replace it with any other client class—such as Claude’s API client or a self-hosted Ollama GPT model—so long as the client provides a `generate_plan` method (or equivalent) for producing responses from a given prompt. + +```python +# For a different client (e.g., Claude or Ollama) +# gpt_client = YourCustomGPTClient(api_key="your_api_key", model="claude-v1") +# wfr.set_gpt_client(gpt_client) +``` +6. Generate GPT-Based Remediation: +```python +remediation = wfr.gpt_remediation_config() +``` +This method builds the GPT-generated remediation based on the `gpt_rule`. + +### Behind the Scenes + +In the background, `hier_config` constructs the prompt and context to pass to GPT: + +1. Context Creation: +```python +contexts = [item for item in wfr._build_remediation_context()] +``` +This function builds the context needed for remediation by iterating over `_build_remediation_context`, which selects configurations matching the specified lineage. + +2. Prompt Construction: +``` +prompt = wfr._build_gpt_prompt(contexts[0]) +``` +Using the gathered context, `wfr._build_gpt_prompt` creates a structured prompt. The prompt includes: + + * `Description`: The remediation steps (e.g., resequencing and temporary changes). + * `Example`: A sample configuration state to guide GPT’s response. + +3. GPT Generates the Plan: GPT uses this prompt to generate commands following the custom rules, such as resequencing access-lists or adding/removing temporary statements. + +This setup makes it possible to leverage any GPT model for custom configuration adjustments in `hier_config`, providing flexibility across various network automation scenarios. diff --git a/docs/install.md b/docs/install.md index 1b8ebb1..651eb2d 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,11 +1,11 @@ # Install hier_config -> Hierarchical Configuration requires a minimum Python version of 3.8. +> Hierarchical Configuration requires a minimum Python version of 3.9. Hierarchical Configuration can be installed directly from GitHub or with pip: ## Pip -1. Install from [PyPI](https://pypi.org/project/hier-config/): `pip install hier-config` +1. Install from PyPI: `pip install hier-config` ## Github 1. [Install Poetry](https://python-poetry.org/docs/#installation) diff --git a/docs/junos-style-syntax-remediation.md b/docs/junos-style-syntax-remediation.md new file mode 100644 index 0000000..b12ea8a --- /dev/null +++ b/docs/junos-style-syntax-remediation.md @@ -0,0 +1,159 @@ +# JunOS-style Syntax Remediation +Operating systems that use "set"-based syntax can now be remediated experimentally. Below is an example of a JunOS-style remediation. + +```bash +$ cat ./tests/fixtures/running_config_flat_junos.conf +set system host-name aggr-example.rtr + +set firewall family inet filter TEST term 1 from source-address 10.0.0.0/29 +set firewall family inet filter TEST term 1 then accept + +set vlans switch_mgmt_10.0.2.0/24 vlan-id 2 +set vlans switch_mgmt_10.0.2.0/24 l3-interface irb.2 + +set vlans switch_mgmt_10.0.4.0/24 vlan-id 3 +set vlans switch_mgmt_10.0.4.0/24 l3-interface irb.3 + +set interfaces irb unit 2 family inet address 10.0.2.1/24 +set interfaces irb unit 2 family inet description "switch_10.0.2.0/24" +set interfaces irb unit 2 family inet disable + +set interfaces irb unit 3 family inet address 10.0.4.1/16 +set interfaces irb unit 3 family inet filter input TEST +set interfaces irb unit 3 family inet mtu 9000 +set interfaces irb unit 3 family inet description "switch_mgmt_10.0.4.0/24" + + +$ python3 +>>> from hier_config import WorkflowRemediation, get_hconfig +>>> from hier_config.model import Platform +>>> from hier_config.utils import load_device_config +>>> +>>> running_config_text = load_device_config("./tests/fixtures/running_config_flat_junos.conf") +>>> generated_config_text = load_device_config("./tests/fixtures/generated_config_flat_junos.conf") +# Create HConfig objects for the running and generated configurations using JunOS syntax +>>> running_config = get_hconfig(Platform.JUNIPER_JUNOS, running_config_text) +>>> generated_config = get_hconfig(Platform.JUNIPER_JUNOS, generated_config_text) +>>> +# Initialize WorkflowRemediation with the running and generated configurations +>>> workflow = WorkflowRemediation(running_config, generated_config) +>>> +# Generate and display the remediation configuration +>>> print("Remediation configuration:") +Remediation configuration: +>>> print(str(workflow.remediation_config)) +delete vlans switch_mgmt_10.0.4.0/24 vlan-id 3 +delete vlans switch_mgmt_10.0.4.0/24 l3-interface irb.3 +delete interfaces irb unit 2 family inet disable +delete interfaces irb unit 3 family inet address 10.0.4.1/16 +delete interfaces irb unit 3 family inet description "switch_mgmt_10.0.4.0/24" +set vlans switch_mgmt_10.0.3.0/24 vlan-id 3 +set vlans switch_mgmt_10.0.3.0/24 l3-interface irb.3 +set vlans switch_mgmt_10.0.4.0/24 vlan-id 4 +set vlans switch_mgmt_10.0.4.0/24 l3-interface irb.4 +set interfaces irb unit 2 family inet filter input TEST +set interfaces irb unit 2 family inet mtu 9000 +set interfaces irb unit 3 family inet address 10.0.3.1/16 +set interfaces irb unit 3 family inet description "switch_mgmt_10.0.3.0/24" +set interfaces irb unit 4 family inet address 10.0.4.1/16 +set interfaces irb unit 4 family inet filter input TEST +set interfaces irb unit 4 family inet mtu 9000 +set interfaces irb unit 4 family inet description "switch_mgmt_10.0.4.0/24" +>>> +``` + +Configurations loaded into Hier Config with Juniper-style syntax are converted to a flat, `set`-based format. Remediation steps are then generated using this `set` syntax. + +```bash +$ cat ./tests/fixtures/running_config_junos.conf +system { + host-name aggr-example.rtr; +} + +firewall { + family inet { + filter TEST { + term 1 { + from { + source-address 10.0.0.0/29; + } + then { + accept; + } + } + } + } +} + +vlans { + switch_mgmt_10.0.2.0/24 { + vlan-id 2; + l3-interface irb.2; + } + switch_mgmt_10.0.4.0/24 { + vlan-id 3; + l3-interface irb.3; + } +} + +interfaces { + irb { + unit 2 { + family inet { + address 10.0.2.1/24; + description "switch_10.0.2.0/24"; + disable; + } + } + unit 3 { + family inet { + address 10.0.4.1/16; + filter { + input TEST; + } + mtu 9000; + description "switch_mgmt_10.0.4.0/24"; + } + } + } +} + +$ python3 +>>> from hier_config import WorkflowRemediation, get_hconfig +>>> from hier_config.model import Platform +>>> from hier_config.utils import load_device_config +>>> +>>> running_config_text = load_device_config("./tests/fixtures/running_config_junos.conf") +>>> generated_config_text = load_device_config("./tests/fixtures/generated_config_junos.conf") +# Create HConfig objects for the running and generated configurations using JunOS syntax +>>> running_config = get_hconfig(Platform.JUNIPER_JUNOS, running_config_text) +>>> generated_config = get_hconfig(Platform.JUNIPER_JUNOS, generated_config_text) +>>> +# Initialize WorkflowRemediation with the running and generated configurations +>>> workflow = WorkflowRemediation(running_config, generated_config) +>>> +# Generate and display the remediation configuration +>>> print("Remediation configuration:") +Remediation configuration: +>>> print(str(workflow.remediation_config)) +delete vlans switch_mgmt_10.0.4.0/24 vlan-id 3 +delete vlans switch_mgmt_10.0.4.0/24 l3-interface irb.3 +delete interfaces irb unit 2 family inet description "switch_10.0.2.0/24" +delete interfaces irb unit 2 family inet disable +delete interfaces irb unit 3 family inet address 10.0.4.1/16 +delete interfaces irb unit 3 family inet description "switch_mgmt_10.0.4.0/24" +set vlans switch_mgmt_10.0.3.0/24 vlan-id 3 +set vlans switch_mgmt_10.0.3.0/24 l3-interface irb.3 +set vlans switch_mgmt_10.0.4.0/24 vlan-id 4 +set vlans switch_mgmt_10.0.4.0/24 l3-interface irb.4 +set interfaces irb unit 2 family inet filter input TEST +set interfaces irb unit 2 family inet mtu 9000 +set interfaces irb unit 2 family inet description "switch_mgmt_10.0.2.0/24" +set interfaces irb unit 3 family inet address 10.0.3.1/16 +set interfaces irb unit 3 family inet description "switch_mgmt_10.0.3.0/24" +set interfaces irb unit 4 family inet address 10.0.4.1/16 +set interfaces irb unit 4 family inet filter input TEST +set interfaces irb unit 4 family inet mtu 9000 +set interfaces irb unit 4 family inet description "switch_mgmt_10.0.4.0/24" +>>> +``` \ No newline at end of file diff --git a/docs/tags.md b/docs/tags.md new file mode 100644 index 0000000..cbac8d8 --- /dev/null +++ b/docs/tags.md @@ -0,0 +1,146 @@ +# Working with Tags + +## MatchRules + +MatchRules, written in YAML, help users identify either highly specific sections or more generalized lines within a configuration. For instance, if you want to target interface descriptions, you could set up MatchRules as follows: + +```yaml +- match_rules: + - startswith: interface + - startswith: description +``` + +This setup directs hier_config to search for configuration lines that begin with `interface` and, under each interface, locate lines that start with `description`​​. + +With MatchRules, you can specify the level of detail needed, whether focusing on general configuration lines or diving into specific subsections. For example, to check for the presence or absence of HTTP, SSH, SNMP, and logging commands in a configuration, you could use a single rule as follows: + +```yaml +- match_rules: + - startswith: + - ip ssh + - no ip ssh + - ip http + - no ip http + - snmp-server + - no snmp-server + - logging + - no logging +``` + +This rule will look for configuration lines that start with any of the listed keywords​. + +To check whether BGP IPv4 AFIs (Address Family Identifiers) are activated, you can use the following rule: + +```yaml +- match_rules: + - startswith: router bgp + - startswith: address-family ipv4 + - endswith: activate +``` + +In this example, the `activate` keyword is used to identify active BGP neighbors. Available keywords for MatchRules include: +- startswith +- endswith +- contains +- equals +- re_search (for regular expressions) + +These options allow you to target configuration lines with precision based on the desired pattern​. + +You can also combine the previous examples into a single set of MatchRules, like this: + +```yaml +- match_rules: + - startswith: interface + - startswith: description +- match_rules: + - startswith: + - ip ssh + - no ip ssh + - ip http + - no ip http + - snmp-server + - no snmp-server + - logging + - no logging +- match_rules: + - startswith: router bgp + - startswith: address-family ipv4 + - endswith: activate +``` + +When `hier_config` processes MatchRules, it treats each as a separate rule, evaluating them individually to match the specified configuration patterns​. + +## Working with Tags + +With a solid understanding of MatchRules, you can unlock more advanced capabilities in `hier_config`, such as tagging specific configuration sections to control remediation output based on tags. This feature is particularly useful during maintenance, allowing you to focus on low-risk changes or isolate high-risk changes for detailed inspection. + +Tagging builds on MatchRules by adding the **apply_tags** keyword to target specific configurations. + +For example, suppose your running configuration contains an NTP server setup like this: + +```text +ntp server 192.0.2.1 prefer version 2 +``` + +But your intended configuration uses publicly available NTP servers: + +```text +ip name-server 1.1.1.1 +ip name-server 8.8.8.8 +ntp server time.nist.gov +``` + +You can create a MatchRule to tag this specific remediation with "ntp" as follows: + +```yaml +- match_rules: + - startswith: + - ip name-server + - no ip name-server + - ntp + - no ntp + apply_tags: [ntp] +``` + +With the tags loaded, you can create a targeted remediation based on those tags as follows: + +```python +#!/usr/bin/env python3 + +# Import necessary libraries +from hier_config import WorkflowRemediation, get_hconfig +from hier_config.models import Platform +from hier_config.utils import load_device_config, load_hier_config_tags + +# Load the running and generated configurations from files +running_config = load_device_config("./tests/fixtures/running_config.conf") +generated_config = load_device_config("./tests/fixtures/generated_config.conf") + +# Load tag rules from a file +tags = load_hier_config_tags("./tests/fixtures/tag_rules_ios.yml") + +# Initialize a WorkflowRemediation object with the running and intended configurations +wfr = WorkflowRemediation( + running_config=get_hconfig(Platform.CISCO_IOS, running_config), + generated_config=get_hconfig(Platform.CISCO_IOS, generated_config) +) + +# Apply the tag rules to filter remediation steps by tags +wfr.apply_remediation_tag_rules(tags) + +# Generate the remediation steps +wfr.remediation_config + +# Display remediation steps filtered to include only the "ntp" tag +print(wfr.remediation_config_filtered_text(include_tags={"ntp"}, exclude_tags={})) +``` + +The resulting remediation output appears as follows: + +```text +no ntp server 192.0.2.1 prefer version 2 +ip name-server 1.1.1.1 +ip name-server 8.8.8.8 +ntp server time.nist.gov +``` \ No newline at end of file diff --git a/docs/unified-diff.md b/docs/unified-diff.md new file mode 100644 index 0000000..54de534 --- /dev/null +++ b/docs/unified-diff.md @@ -0,0 +1,43 @@ +# Unified diff + +The Unified Diff feature, introduced in version 2.1.0, provides output similar to `difflib.unified_diff()` but with added awareness of out-of-order lines and parent-child relationships in the Hier Config model of configurations being compared. + +This feature is particularly useful when comparing configurations from two network devices, such as redundant pairs, or when validating differences between running and intended configurations. + +Currently, the algorithm does not account for duplicate child entries (e.g., multiple `endif` statements in an IOS-XR route-policy) or enforce command order in sections where it may be critical, such as Access Control Lists (ACLs). For accurate ordering in ACLs, sequence numbers should be used if command order is important. + +```bash +>>> from hier_config import get_hconfig +>>> from hier_config.model import Platform +>>> from pprint import pprint +>>> +>>> running_config_text = load_device_config("./tests/fixtures/running_config.conf") +>>> generated_config_text = load_device_config("./tests/fixtures/generated_config.conf") +>>> +>>> running_config = get_hconfig(Platform.CISCO_IOS, running_config_text) +>>> generated_config = get_hconfig(Platform.CISCO_IOS, generated_config_text) +>>> +>>> pprint(list(running_config.unified_diff(generated_config))) +['vlan 3', + ' - name switch_mgmt_10.0.4.0/24', + ' + name switch_mgmt_10.0.3.0/24', + 'interface Vlan2', + ' - shutdown', + ' + mtu 9000', + ' + ip access-group TEST in', + ' + no shutdown', + 'interface Vlan3', + ' - description switch_mgmt_10.0.4.0/24', + ' - ip address 10.0.4.1 255.255.0.0', + ' + description switch_mgmt_10.0.3.0/24', + ' + ip address 10.0.3.1 255.255.0.0', + '+ vlan 4', + ' + name switch_mgmt_10.0.4.0/24', + '+ interface Vlan4', + ' + mtu 9000', + ' + description switch_mgmt_10.0.4.0/24', + ' + ip address 10.0.4.1 255.255.0.0', + ' + ip access-group TEST in', + ' + no shutdown'] +>>> +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index c829397..2fd06ca 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -8,5 +8,10 @@ nav: - Home: index.md - Install: install.md - Getting Started: getting-started.md -- Advanced Topics: advanced-topics.md -- Experimental Features: experimental-features.md +- Drivers: drivers.md +- Custom Workflows: custom-workflows.md +- Future Config: future-config.md +- GPT Based Remediation: gpt-based-remediation.md +- JunOS Style Syntax Remediation: junos-style-sytax-remediation.md +- Unified Diff: unified-diff.md +- Working with Tags: tags.md From 09300ed07118dcb1ec3e126502e42eb3a030b8cb Mon Sep 17 00:00:00 2001 From: James Williams Date: Fri, 15 Nov 2024 14:52:03 -0600 Subject: [PATCH 02/10] add utils and update docs --- docs/gpt-based-remediation.md | 114 ---------------------------------- hier_config/utils.py | 33 ++++++++++ mkdocs.yml | 1 - tests/test_utils.py | 100 +++++++++++++++++++++++++++++ 4 files changed, 133 insertions(+), 115 deletions(-) delete mode 100644 docs/gpt-based-remediation.md create mode 100644 hier_config/utils.py create mode 100644 tests/test_utils.py diff --git a/docs/gpt-based-remediation.md b/docs/gpt-based-remediation.md deleted file mode 100644 index 88fbbeb..0000000 --- a/docs/gpt-based-remediation.md +++ /dev/null @@ -1,114 +0,0 @@ -# GPT-Based Custom Remediations - -In certain cases, configuration remediations may fall outside of the standard negation and idempotency workflows that `hier_config` is built to handle. Previously, addressing these edge cases required manually crafting specific remediation steps. However, with the advent of AI, particularly generative pre-trained transformers (GPT), it’s now possible to dynamically generate these custom remediations. - -The example below demonstrates how to integrate a GPT client for AI-driven remediations. While this example uses OpenAI's ChatGPT, `hier_config` can work with any compatible GPT model—such as Claude or self-hosted models like Ollama GPT—so long as the client provides similar functionality. - -### Code Walkthrough and Explanation - -1. Import and Configuration Loading -```python -import os -from hier_config import WorkflowRemediation, get_hconfig -from hier_config.utils import load_device_config -from hier_config.models import Platform, GPTRemediationRule, MatchRule, GPTRemediationExample -from hier_config.clients import ChatGPTClient -``` - -Necessary modules are imported, including the `WorkflowRemediation` class for handling remediation, as well as a client interface—in this case, `ChatGPTClient` for connecting to GPT. For other GPT models, you can replace `ChatGPTClient` with your own client class. - -```python -running_config = load_device_config("./tests/fixtures/running_config_acl.conf") -generated_config = load_device_config("./tests/fixtures/generated_config_acl.conf") -``` -Here, the current and intended configurations are loaded from files, serving as inputs for comparison and remediation. - -2. Initialize `WorkflowRemiation`: -```python -wfr = WorkflowRemediation( - running_config=get_hconfig(Platform.CISCO_IOS, running_config), - generated_config=get_hconfig(Platform.CISCO_IOS, generated_config) -) -``` -This initializes `WorkflowRemediation` with the configurations for a Cisco IOS platform, setting up the object that will manage the remediation workflow. - -3. Define a GPT-Based Remediation Rule: -```python -description = ( - "When remediating an access-list on Cisco IOS devices, follow these steps precisely:\n" - " 1. **Resequence the access-list:**\n" - " * **Action**: Resequence the access-list so that each sequence number is a multiple of 10.\n" - " * **Condition**: If a sequence number in the running config isn't divisible by 10, resequence it to the nearest 10, starting at 10.\n" - " 2. **Add Temporary Permit Statement:**\n" - " * **Action**: Insert a temporary `'permit any'` statement at sequence number `1`.\n" - " * **Purpose**: Ensures there's always a valid permit in place during modifications.\n" - " 3. **Apply Required Changes:**\n" - " * **Action**: Update the access-list with the new permit statements as per the generated configuration.\n" - " 4. **Remove Temporary Permit Statement:**\n" - " * **Action**: Remove the temporary `'permit any'` statement added at sequence number `1`.\n" - "\n" - " **Important**:\n" - " **When issuing `no` commands to remove existing entries after resequencing, USE THE" - " RESEQUENCED SEQUENCE NUMBERS, NOT THE ORIGINAL SEQUENCE NUMBERS FROM THE" - " RUNNING CONFIGURATION.**" -) -lineage = (MatchRule(startswith="ip access-list"),) -running = "ip access-list extended TEST\n 12 permit ip host 10.0.0.1 any" -remediation = "ip access-list resequence TEST 10 10\nip access-list extended TEST\n 1 permit ip any any\n no 10\n 10 permit ip host 10.0.0.2 any\n no 1" -example = GPTRemediationExample(running_config=running, remediation_config=remediation) -gpt_rule = GPTRemediationRule(description=description, lineage=lineage, example=example) -``` -Here: - * `description`: Outlines specific remediation steps that GPT should apply. - * `lineage`: Specifies MatchRule criteria to identify access-list configurations. - * `example`: Provides a running configuration and remediation example for added context. - * `gpt_rule`: Combines the description, lineage, and example into a GPTRemediationRule. - -4. Load AI Remediation Rules: -```python -wfr.running_config.driver.gpt_remediation_rules.append(gpt_rule) -``` -This appends the custom rule to `gpt_remediation_rules`, making it available for GPT to apply. - -5. Set Up and Load the GPT Client: -```python -# Example using ChatGPT, but can be replaced with other clients -gpt_client = ChatGPTClient(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o-mini") -wfr.set_gpt_client(gpt_client) -``` - -In this example, `ChatGPTClient` is used for integration with OpenAI's GPT, but you can replace it with any other client class—such as Claude’s API client or a self-hosted Ollama GPT model—so long as the client provides a `generate_plan` method (or equivalent) for producing responses from a given prompt. - -```python -# For a different client (e.g., Claude or Ollama) -# gpt_client = YourCustomGPTClient(api_key="your_api_key", model="claude-v1") -# wfr.set_gpt_client(gpt_client) -``` -6. Generate GPT-Based Remediation: -```python -remediation = wfr.gpt_remediation_config() -``` -This method builds the GPT-generated remediation based on the `gpt_rule`. - -### Behind the Scenes - -In the background, `hier_config` constructs the prompt and context to pass to GPT: - -1. Context Creation: -```python -contexts = [item for item in wfr._build_remediation_context()] -``` -This function builds the context needed for remediation by iterating over `_build_remediation_context`, which selects configurations matching the specified lineage. - -2. Prompt Construction: -``` -prompt = wfr._build_gpt_prompt(contexts[0]) -``` -Using the gathered context, `wfr._build_gpt_prompt` creates a structured prompt. The prompt includes: - - * `Description`: The remediation steps (e.g., resequencing and temporary changes). - * `Example`: A sample configuration state to guide GPT’s response. - -3. GPT Generates the Plan: GPT uses this prompt to generate commands following the custom rules, such as resequencing access-lists or adding/removing temporary statements. - -This setup makes it possible to leverage any GPT model for custom configuration adjustments in `hier_config`, providing flexibility across various network automation scenarios. diff --git a/hier_config/utils.py b/hier_config/utils.py new file mode 100644 index 0000000..3219ca5 --- /dev/null +++ b/hier_config/utils.py @@ -0,0 +1,33 @@ +from pathlib import Path + +import yaml +from pydantic import TypeAdapter + +from hier_config.models import TagRule + + +def load_device_config(file_path: str) -> str: + """Reads a device configuration file and loads its contents into memory. + + Args: + file_path (str): The path to the configuration file. + + Returns: + str: The configuration file contents as a string. + + """ + return Path(file_path).read_text(encoding="utf-8") + + +def load_hier_config_tags(tags_file: str) -> tuple[TagRule, ...]: + """Loads and validates Hier Config tags from a YAML file. + + Args: + tags_file (str): Path to the YAML file containing the tags. + + Returns: + Tuple[TagRule, ...]: A tuple of validated TagRule objects. + + """ + tags_data = yaml.safe_load(Path(tags_file).read_text(encoding="utf-8")) + return TypeAdapter(tuple[TagRule, ...]).validate_python(tags_data) diff --git a/mkdocs.yml b/mkdocs.yml index 2fd06ca..fee1067 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -11,7 +11,6 @@ nav: - Drivers: drivers.md - Custom Workflows: custom-workflows.md - Future Config: future-config.md -- GPT Based Remediation: gpt-based-remediation.md - JunOS Style Syntax Remediation: junos-style-sytax-remediation.md - Unified Diff: unified-diff.md - Working with Tags: tags.md diff --git a/tests/test_utils.py b/tests/test_utils.py new file mode 100644 index 0000000..8f71f79 --- /dev/null +++ b/tests/test_utils.py @@ -0,0 +1,100 @@ +from pathlib import Path # Use pathlib for handling file paths +from typing import Any, Union + +import pytest +import yaml +from pydantic import ValidationError + +from hier_config.models import TagRule +from hier_config.utils import load_device_config, load_hier_config_tags + +TAGS_FILE_PATH = "./tests/fixtures/tag_rules_ios.yml" + + +# Helper function to create a temporary file for testing +@pytest.fixture +def temp_file(tmp_path: Path) -> tuple[Path, str]: + file_path = tmp_path / "temp_config.conf" + content = "interface GigabitEthernet0/1\n ip address 192.168.1.1 255.255.255.0\n no shutdown" + file_path.write_text(content) + return file_path, content + + +def test_load_device_config_success(temp_file: tuple[Path, str]) -> None: + """Test that the function successfully loads a valid configuration file.""" + file_path, expected_content = temp_file + result = load_device_config(str(file_path)) + assert result == expected_content, "File content should match expected content." + + +def test_load_device_config_file_not_found() -> None: + """Test that the function raises FileNotFoundError when the file does not exist.""" + with pytest.raises(FileNotFoundError): + load_device_config("non_existent_file.conf") + + +def test_load_device_config_io_error( + monkeypatch: pytest.MonkeyPatch, temp_file: tuple[Path, str] +) -> None: + """Test that the function raises OSError for an unexpected file access error.""" + file_path, _ = temp_file + + def mock_read_text(*args: Any, **kwargs: Union[dict, None]) -> None: # noqa: ANN401, ARG001 + msg = "Mocked IO error" + raise OSError(msg) + + monkeypatch.setattr(Path, "read_text", mock_read_text) + + with pytest.raises(OSError, match="Mocked IO error"): + load_device_config(str(file_path)) + + +def test_load_device_config_empty_file(tmp_path: Path) -> None: + """Test that the function correctly handles an empty configuration file.""" + empty_file = tmp_path / "empty.conf" + empty_file.write_text("") # Create an empty file + result = load_device_config(str(empty_file)) + assert not result, "Empty file should return an empty string." + + +def test_load_hier_config_tags_success() -> None: + """Test that valid tags from the tag_rules_ios.yml file load and validate successfully.""" + result = load_hier_config_tags(TAGS_FILE_PATH) + + assert isinstance(result, tuple), "Result should be a tuple of TagRule objects." + assert len(result) == 4, "There should be four TagRule objects." + assert isinstance(result[0], TagRule), "Each element should be a TagRule object." + assert result[0].apply_tags == { + "safe" + }, "First tag should have 'safe' as an applied tag." + assert result[3].apply_tags == { + "manual" + }, "Last tag should have 'manual' as an applied tag." + + +def test_load_hier_config_tags_file_not_found() -> None: + """Test that the function raises FileNotFoundError for a missing file.""" + with pytest.raises(FileNotFoundError): + load_hier_config_tags("non_existent_file.yml") + + +def test_load_hier_config_tags_invalid_yaml(tmp_path: Path) -> None: + """Test that the function raises yaml.YAMLError for invalid YAML syntax.""" + invalid_file = tmp_path / "invalid_tags.yml" + invalid_file.write_text(""" + - match_rules: + - equals: no ip http server + apply_tags [safe] # Missing colon causes a syntax error + """) + + with pytest.raises(yaml.YAMLError): + load_hier_config_tags(str(invalid_file)) + + +def test_load_hier_config_tags_empty_file(tmp_path: Path) -> None: + """Test that the function raises ValidationError for an empty YAML file.""" + empty_file = tmp_path / "empty.yml" + empty_file.write_text("") # Create an empty file + + with pytest.raises(ValidationError): + load_hier_config_tags(str(empty_file)) From fad5c115c9d912ee0d2ac71ab47d5876901612b1 Mon Sep 17 00:00:00 2001 From: James Williams Date: Fri, 15 Nov 2024 15:05:56 -0600 Subject: [PATCH 03/10] update v3-docs --- docs/advanced-topics.md | 157 ------------------------- docs/custom-workflows.md | 4 +- docs/future-config.md | 4 +- docs/getting-started.md | 3 +- docs/junos-style-syntax-remediation.md | 6 +- docs/tags.md | 3 +- docs/unified-diff.md | 3 +- tests/test_utils.py | 30 +---- 8 files changed, 13 insertions(+), 197 deletions(-) delete mode 100644 docs/advanced-topics.md diff --git a/docs/advanced-topics.md b/docs/advanced-topics.md deleted file mode 100644 index 7b92e7b..0000000 --- a/docs/advanced-topics.md +++ /dev/null @@ -1,157 +0,0 @@ -# Advanced Topics - -## MatchRules - -MatchRules, written in YAML, help users identify either highly specific sections or more generalized lines within a configuration. For instance, if you want to target interface descriptions, you could set up MatchRules as follows: - -```yaml -- match_rules: - - startswith: interface - - startswith: description -``` - -This setup directs hier_config to search for configuration lines that begin with `interface` and, under each interface, locate lines that start with `description`​​. - -With MatchRules, you can specify the level of detail needed, whether focusing on general configuration lines or diving into specific subsections. For example, to check for the presence or absence of HTTP, SSH, SNMP, and logging commands in a configuration, you could use a single rule as follows: - -```yaml -- match_rules: - - startswith: - - ip ssh - - no ip ssh - - ip http - - no ip http - - snmp-server - - no snmp-server - - logging - - no logging -``` - -This rule will look for configuration lines that start with any of the listed keywords​. - -To check whether BGP IPv4 AFIs (Address Family Identifiers) are activated, you can use the following rule: - -```yaml -- match_rules: - - startswith: router bgp - - startswith: address-family ipv4 - - endswith: activate -``` - -In this example, the `activate` keyword is used to identify active BGP neighbors. Available keywords for MatchRules include: -- startswith -- endswith -- contains -- equals -- re_search (for regular expressions) - -These options allow you to target configuration lines with precision based on the desired pattern​. - -You can also combine the previous examples into a single set of MatchRules, like this: - -```yaml -- match_rules: - - startswith: interface - - startswith: description -- match_rules: - - startswith: - - ip ssh - - no ip ssh - - ip http - - no ip http - - snmp-server - - no snmp-server - - logging - - no logging -- match_rules: - - startswith: router bgp - - startswith: address-family ipv4 - - endswith: activate -``` - -When `hier_config` processes MatchRules, it treats each as a separate rule, evaluating them individually to match the specified configuration patterns​. - -## Working with Tags - -With a solid understanding of MatchRules, you can unlock more advanced capabilities in `hier_config`, such as tagging specific configuration sections to control remediation output based on tags. This feature is particularly useful during maintenance, allowing you to focus on low-risk changes or isolate high-risk changes for detailed inspection. - -Tagging builds on MatchRules by adding the **apply_tags** keyword to target specific configurations. - -For example, suppose your running configuration contains an NTP server setup like this: - -```text -ntp server 192.0.2.1 prefer version 2 -``` - -But your intended configuration uses publicly available NTP servers: - -```text -ip name-server 1.1.1.1 -ip name-server 8.8.8.8 -ntp server time.nist.gov -``` - -You can create a MatchRule to tag this specific remediation with "ntp" as follows: - -```yaml -- match_rules: - - startswith: - - ip name-server - - no ip name-server - - ntp - - no ntp - apply_tags: [ntp] -``` - -With the tags loaded, you can create a targeted remediation based on those tags as follows: - -```python -#!/usr/bin/env python3 - -# Import necessary libraries -import yaml -from pydantic import TypeAdapter -from hier_config import WorkflowRemediation, get_hconfig -from hier_config.models import Platform, TagRule - -# Load the running and generated configurations from files -with open("./tests/fixtures/running_config.conf") as f: - running_config = f.read() - -with open("./tests/fixtures/generated_config.conf") as f: - generated_config = f.read() - -# Load tag rules from a file -with open("./tests/fixtures/tag_rules_ios.yml") as f: - tags = yaml.safe_load(f) - -# Validate and format tags using the TagRule model -tag_rules = TypeAdapter(tuple[TagRule, ...]).validate_python(tags) - -# Initialize a WorkflowRemediation object with the running and intended configurations -wfr = WorkflowRemediation( - running_config=get_hconfig(Platform.CISCO_IOS, running_config), - generated_config=get_hconfig(Platform.CISCO_IOS, generated_config) -) - -# Apply the tag rules to filter remediation steps by tags -wfr.apply_remediation_tag_rules(tag_rules) - -# Display remediation steps filtered to include only the "ntp" tag -print(wfr.remediation_config_filtered_text(include_tags={"ntp"}, exclude_tags={})) -``` - -The resulting remediation output appears as follows: - -```text -no ntp server 192.0.2.1 prefer version 2 -ip name-server 1.1.1.1 -ip name-server 8.8.8.8 -ntp server time.nist.gov -``` - -## Drivers - -## Custom hier_config Workflows - -Coming soon... diff --git a/docs/custom-workflows.md b/docs/custom-workflows.md index 9856d86..bc44be5 100644 --- a/docs/custom-workflows.md +++ b/docs/custom-workflows.md @@ -7,9 +7,7 @@ In some scenarios, configuration remediations may require handling beyond the st 1. Import and Configuration Loading ```python -import os -from hier_config import WorkflowRemediation, get_hconfig -from hier_config.models import Platform +from hier_config import WorkflowRemediation, get_hconfig, Platform ``` Necessary modules are imported, including the `WorkflowRemediation` class for handling remediation diff --git a/docs/future-config.md b/docs/future-config.md index 46a71e9..672973d 100644 --- a/docs/future-config.md +++ b/docs/future-config.md @@ -25,10 +25,8 @@ Currently, this algorithm does not account for: - and likely others ```bash ->>> from hier_config import get_hconfig ->>> from hier_config.model import Platform +>>> from hier_config import get_hconfig, Platform >>> from hier_config.utils import load_device_config ->>> from pprint import pprint >>> >>> running_config_text = load_device_config("./tests/fixtures/running_config.conf") diff --git a/docs/getting-started.md b/docs/getting-started.md index 5c73906..9ced331 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -7,8 +7,7 @@ Hier Config is a Python library that assists with remediating network configurat To use `WorkflowRemediation`, you’ll import it along with `get_hconfig` (for generating configuration objects) and `Platform` (for specifying the operating system driver). ```python ->>> from hier_config import WorkflowRemediation, get_hconfig ->>> from hier_config.model import Platform +>>> from hier_config import WorkflowRemediation, get_hconfig, Platform >>> from hier_config.utils import load_device_config >>> ``` diff --git a/docs/junos-style-syntax-remediation.md b/docs/junos-style-syntax-remediation.md index b12ea8a..9126930 100644 --- a/docs/junos-style-syntax-remediation.md +++ b/docs/junos-style-syntax-remediation.md @@ -25,8 +25,7 @@ set interfaces irb unit 3 family inet description "switch_mgmt_10.0.4.0/24" $ python3 ->>> from hier_config import WorkflowRemediation, get_hconfig ->>> from hier_config.model import Platform +>>> from hier_config import WorkflowRemediation, get_hconfig, Platform >>> from hier_config.utils import load_device_config >>> >>> running_config_text = load_device_config("./tests/fixtures/running_config_flat_junos.conf") @@ -119,8 +118,7 @@ interfaces { } $ python3 ->>> from hier_config import WorkflowRemediation, get_hconfig ->>> from hier_config.model import Platform +>>> from hier_config import WorkflowRemediation, get_hconfig, Platform >>> from hier_config.utils import load_device_config >>> >>> running_config_text = load_device_config("./tests/fixtures/running_config_junos.conf") diff --git a/docs/tags.md b/docs/tags.md index cbac8d8..101f48e 100644 --- a/docs/tags.md +++ b/docs/tags.md @@ -109,8 +109,7 @@ With the tags loaded, you can create a targeted remediation based on those tags #!/usr/bin/env python3 # Import necessary libraries -from hier_config import WorkflowRemediation, get_hconfig -from hier_config.models import Platform +from hier_config import WorkflowRemediation, get_hconfig, Platform from hier_config.utils import load_device_config, load_hier_config_tags # Load the running and generated configurations from files diff --git a/docs/unified-diff.md b/docs/unified-diff.md index 54de534..75d07a6 100644 --- a/docs/unified-diff.md +++ b/docs/unified-diff.md @@ -7,8 +7,7 @@ This feature is particularly useful when comparing configurations from two netwo Currently, the algorithm does not account for duplicate child entries (e.g., multiple `endif` statements in an IOS-XR route-policy) or enforce command order in sections where it may be critical, such as Access Control Lists (ACLs). For accurate ordering in ACLs, sequence numbers should be used if command order is important. ```bash ->>> from hier_config import get_hconfig ->>> from hier_config.model import Platform +>>> from hier_config import get_hconfig, Platform >>> from pprint import pprint >>> >>> running_config_text = load_device_config("./tests/fixtures/running_config.conf") diff --git a/tests/test_utils.py b/tests/test_utils.py index 8f71f79..ddb42f8 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,5 +1,4 @@ -from pathlib import Path # Use pathlib for handling file paths -from typing import Any, Union +from pathlib import Path import pytest import yaml @@ -11,18 +10,17 @@ TAGS_FILE_PATH = "./tests/fixtures/tag_rules_ios.yml" -# Helper function to create a temporary file for testing @pytest.fixture -def temp_file(tmp_path: Path) -> tuple[Path, str]: +def temporary_file_fixture(tmp_path: Path) -> tuple[Path, str]: file_path = tmp_path / "temp_config.conf" content = "interface GigabitEthernet0/1\n ip address 192.168.1.1 255.255.255.0\n no shutdown" file_path.write_text(content) return file_path, content -def test_load_device_config_success(temp_file: tuple[Path, str]) -> None: +def test_load_device_config_success(temporary_file_fixture: tuple[Path, str]) -> None: """Test that the function successfully loads a valid configuration file.""" - file_path, expected_content = temp_file + file_path, expected_content = temporary_file_fixture result = load_device_config(str(file_path)) assert result == expected_content, "File content should match expected content." @@ -33,26 +31,10 @@ def test_load_device_config_file_not_found() -> None: load_device_config("non_existent_file.conf") -def test_load_device_config_io_error( - monkeypatch: pytest.MonkeyPatch, temp_file: tuple[Path, str] -) -> None: - """Test that the function raises OSError for an unexpected file access error.""" - file_path, _ = temp_file - - def mock_read_text(*args: Any, **kwargs: Union[dict, None]) -> None: # noqa: ANN401, ARG001 - msg = "Mocked IO error" - raise OSError(msg) - - monkeypatch.setattr(Path, "read_text", mock_read_text) - - with pytest.raises(OSError, match="Mocked IO error"): - load_device_config(str(file_path)) - - def test_load_device_config_empty_file(tmp_path: Path) -> None: """Test that the function correctly handles an empty configuration file.""" empty_file = tmp_path / "empty.conf" - empty_file.write_text("") # Create an empty file + empty_file.write_text("") result = load_device_config(str(empty_file)) assert not result, "Empty file should return an empty string." @@ -94,7 +76,7 @@ def test_load_hier_config_tags_invalid_yaml(tmp_path: Path) -> None: def test_load_hier_config_tags_empty_file(tmp_path: Path) -> None: """Test that the function raises ValidationError for an empty YAML file.""" empty_file = tmp_path / "empty.yml" - empty_file.write_text("") # Create an empty file + empty_file.write_text("") with pytest.raises(ValidationError): load_hier_config_tags(str(empty_file)) From bb12fb92cb0ba055c70afcc0dbc48ff15460bcc1 Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 15:22:06 -0600 Subject: [PATCH 04/10] resolve pylint issue --- tests/test_utils.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_utils.py b/tests/test_utils.py index ddb42f8..91682b7 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -20,6 +20,7 @@ def temporary_file_fixture(tmp_path: Path) -> tuple[Path, str]: def test_load_device_config_success(temporary_file_fixture: tuple[Path, str]) -> None: """Test that the function successfully loads a valid configuration file.""" + # pylint: disable=redefined-outer-name file_path, expected_content = temporary_file_fixture result = load_device_config(str(file_path)) assert result == expected_content, "File content should match expected content." From eefeed1b6a387fa4597b3099e82491a3618907ed Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 16:55:27 -0600 Subject: [PATCH 05/10] Add driver docs --- docs/drivers.md | 603 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 602 insertions(+), 1 deletion(-) diff --git a/docs/drivers.md b/docs/drivers.md index 0f87934..2089af9 100644 --- a/docs/drivers.md +++ b/docs/drivers.md @@ -1,5 +1,606 @@ -# Drivers +# Drivers in Hier Config + +Drivers represent a modern approach to handling operating system-specific options within Hier Config. Prior to version 3, Hier Config utilized `options` or `hconfig_options`, which were defined as dictionaries, to specify OS-specific parameters. Starting with version 3, these options have been replaced by drivers, which are implemented as Pydantic models and loaded as Python classes, offering improved structure and validation. + +> **Note:** Many of the options available in the Hier Config version 3 driver format are similar to those in the version 2 options format. However, some options have been removed because they are no longer used in version 3, or their names have been updated for consistency or clarity. + +## What is a Driver? + +A driver in Hier Config defines a structured and systematic approach to managing operating system-specific configurations for network devices. It acts as a framework that encapsulates the rules, transformations, and behaviors required to process and normalize device configurations. + +Drivers provide a consistent way to handle configurations by applying a set of specialized logic, including: + +1. **Negation Handling**: Ensures commands are properly negated or reset according to the operating system's syntax and behavior, maintaining consistency in enabling or disabling features. + +2. **Sectional Exiting Rules**: Defines how to navigate in and out of hierarchical configuration sections, ensuring commands are logically grouped and the configuration maintains its structural integrity. + +3. **Command Ordering**: Establishes the sequence in which commands should be applied based on dependencies or importance, preventing conflicts or misconfigurations during deployment. + +4. **Line Substitutions**: Cleans up unnecessary or temporary data in configurations, such as metadata, system-generated comments, or obsolete commands, resulting in a streamlined and standardized output. + +5. **Idempotency Management**: Identifies and enforces commands that should not be duplicated, ensuring repeated application of the configuration does not lead to redundant or conflicting entries. + +6. **Post-Processing Callbacks**: Performs additional adjustments or enhancements after the initial configuration is processed, such as refining access control lists or applying custom transformations specific to the device's operating system. + +By defining these rules and behaviors in a reusable way, a driver enables Hier Config to adapt seamlessly to different operating systems while maintaining a consistent interface for configuration management. This abstraction allows users to work with configurations in a predictable and efficient manner, regardless of the underlying system-specific requirements. + +--- + +## Built-In Drivers in Hier Config + +The following drivers are included in Hier Config: + +- **ARISTA_EOS** +- **CISCO_IOS** +- **CISCO_XR** +- **CISCO_NXOS** +- **GENERIC** +- **HP_COMWARE5** +- **HP_PROCURVE** +- **JUNIPER_JUNOS** +- **VYOS** + +To activate a driver, use the `get_hconfig_driver` utility provided by Hier Config: + +```python +from hier_config import get_hconfig_driver, Platform + +# Example: Activating the CISCO_IOS driver +driver = get_hconfig_driver(Platform.CISCO_IOS) +``` + +### Structure of Each Section and How Rules Are Built + +In Hier Config, the rules within a driver are organized into sections, each targeting a specific aspect of device configuration processing. These sections use Pydantic models to define the behavior and ensure consistency. Here's a breakdown of each section and its associated models: + +--- + +### 1. Negation Rules +**Purpose**: Define how to negate commands or reset them to a default state. + +- **Models**: + - **`NegationDefaultWithRule`**: + - `match_rules`: A tuple of `MatchRule` objects defining the conditions under which the rule applies. + - `use`: The text to use as the negation command. + + - **`NegationDefaultWhenRule`**: + - `match_rules`: A tuple of `MatchRule` objects for matching conditions where negation is default. + +--- + +### 2. Sectional Exiting +**Purpose**: Manage hierarchical configuration sections by defining commands for properly exiting each section. + +- **Models**: + - **`SectionalExitingRule`**: + - `match_rules`: A tuple of `MatchRule` objects defining the section's boundaries. + - `exit_text`: The command used to exit the section. + +--- + +### 3. Ordering +**Purpose**: Assign weights to commands to control the order of operations during configuration application. + +- **Models**: + - **`OrderingRule`**: + - `match_rules`: A tuple of `MatchRule` objects defining the commands to be ordered. + - `weight`: An integer determining the order (lower weights are processed earlier). + +--- + +### 4. Per-Line Substitutions +**Purpose**: Modify or clean up specific lines in the configuration. + +- **Models**: + - **`PerLineSubRule`**: + - `search`: A string or regex to search for. + - `replace`: The replacement text. + + - **`FullTextSubRule`**: + - Similar to `PerLineSubRule`, but applies to the entire text rather than individual lines. + +--- + +### 5. Idempotent Commands +**Purpose**: Ensure commands are not repeated unnecessarily in the configuration. + +- **Models**: + - **`IdempotentCommandsRule`**: + - `match_rules`: A tuple of `MatchRule` objects defining idempotent commands. + + - **`IdempotentCommandsAvoidRule`**: + - `match_rules`: Specifies commands that should be avoided during idempotency checks. + +--- + +### 6. Post-Processing Callbacks +**Purpose**: Apply additional transformations after initial configuration processing. + +- **Implementation**: + - A list of functions or methods called after the driver rules are applied, enabling custom logic specific to the platform. + +--- + +### 7. Tagging and Overwriting +**Purpose**: Apply tags to configuration lines or define overwriting behavior for specific sections. + +- **Models**: + - **`TagRule`**: + - `match_rules`: A tuple of `MatchRule` objects defining the lines to tag. + - `apply_tags`: A frozenset of tags to apply. + + - **`SectionalOverwriteRule`**: + - `match_rules`: Defines sections that can be overwritten. + + - **`SectionalOverwriteNoNegateRule`**: + - Similar to `SectionalOverwriteRule`, but prevents negation. + +--- + +### 8. Indentation Adjustments +**Purpose**: Define start and end points for adjusting indentation within configurations. + +- **Models**: + - **`IndentAdjustRule`**: + - `start_expression`: Regex or text marking the start of an adjustment. + - `end_expression`: Regex or text marking the end of an adjustment. + +--- + +### 9. Match Rules +**Purpose**: Provide a flexible way to define conditions for matching configuration lines. + +- **Models**: + - **`MatchRule`**: + - `equals`: Matches lines that are exactly equal. + - `startswith`: Matches lines that start with the specified text or tuple of text. + - `endswith`: Matches lines that end with the specified text or tuple of text. + - `contains`: Matches lines that contain the specified text or tuple of text. + - `re_search`: Matches lines using a regular expression. + +--- + +### 10. Instance Metadata +**Purpose**: Manage metadata for configuration instances, such as tags and comments. + +- **Models**: + - **`Instance`**: + - `id`: A unique positive integer identifier. + - `comments`: A frozenset of comments. + - `tags`: A frozenset of tags. + +--- + +### 11. Dumping Configuration +**Purpose**: Represent and handle the output of processed configuration lines. + +- **Models**: + - **`DumpLine`**: + - `depth`: Indicates the hierarchy level of the line. + - `text`: The configuration text. + - `tags`: A frozenset of tags associated with the line. + - `comments`: A frozenset of comments associated with the line. + - `new_in_config`: A boolean indicating if the line is new. + + - **`Dump`**: + - `lines`: A tuple of `DumpLine` objects representing the processed configuration. + +--- + +### General Rule-Building Patterns + +1. **Define Matching Conditions**: + - Use `MatchRule` to specify conditions for each rule, ensuring flexible and precise control over which configuration lines a rule applies to. + +2. **Apply Context-Specific Logic**: + - Use specialized models like `SectionalExitingRule` or `IdempotentCommandsRule` to tailor behavior to hierarchical or idempotency-related scenarios. + +3. **Maintain Immutability**: + - All models use Pydantic’s immutability and validation to enforce the integrity of rules and configurations. + +This structure ensures that drivers are modular, extensible, and capable of handling diverse configuration scenarios across different platforms. ## Customizing Existing Drivers +This guide provides two examples of how to extend the rules for a Cisco IOS driver in Hier Config. The first example involves subclassing the driver to customize and add rules. The second example demonstrates extending the driver rules dynamically after the driver has already been instantiated. + +--- + +### Example 1: Subclassing the Driver to Extend Rules + +In this approach, you create a new class that subclasses the base Cisco IOS driver and overrides its `_instantiate_rules` method to customize the rules. + +```python +from hier_config.models import ( + MatchRule, + NegationDefaultWithRule, + SectionalExitingRule, + OrderingRule, + PerLineSubRule, + IdempotentCommandsRule, +) +from hier_config.drivers.cisco_ios import HConfigDriverCiscoIOS + + +class ExtendedHConfigDriverCiscoIOS(HConfigDriverCiscoIOS): + @staticmethod + def _instantiate_rules(): + # Start with the base rules + base_rules = super()._instantiate_rules() + + # Extend negation rules + base_rules.negate_with.append( + NegationDefaultWithRule( + match_rules=(MatchRule(startswith="ip route "),), + use="no ip route" + ) + ) + + # Extend sectional exiting rules + base_rules.sectional_exiting.append( + SectionalExitingRule( + match_rules=( + MatchRule(startswith="policy-map"), + MatchRule(startswith="class"), + ), + exit_text="exit", + ) + ) + + # Add additional ordering rules + base_rules.ordering.append( + OrderingRule( + match_rules=( + MatchRule(startswith="access-list"), + MatchRule(startswith="permit "), + ), + weight=50, + ) + ) + + # Add new per-line substitutions + base_rules.per_line_sub.append( + PerLineSubRule( + search="^!.*Generated by system.*$", replace="" + ) + ) + + # Add new idempotent commands + base_rules.idempotent_commands.append( + IdempotentCommandsRule( + match_rules=( + MatchRule(startswith="interface "), + MatchRule(startswith="speed "), + ) + ) + ) + + return base_rules +``` + +#### Using the Subclassed Driver + +```python +from hier_config import Platform + +# Example function to activate the extended driver +def get_extended_hconfig_driver(platform: Platform): + if platform == Platform.CISCO_IOS: + return ExtendedHConfigDriverCiscoIOS() + raise ValueError(f"Unsupported platform: {platform}") + +# Activate the extended driver +driver = get_extended_hconfig_driver(Platform.CISCO_IOS) +``` + +### Example 2: Dynamically Extending Rules for an Instantiated Driver + +If you already have the driver instantiated, you can modify its rules dynamically by directly appending to the appropriate sections. + +```python +from hier_config import get_hconfig_driver, Platform +from hier_config.models import ( + MatchRule, + NegationDefaultWithRule, + SectionalExitingRule, + OrderingRule, + PerLineSubRule, + IdempotentCommandsRule, +) + +# Instantiate the driver +driver = get_hconfig_driver(Platform.CISCO_IOS) + +# Dynamically extend negation rules +driver.rules.negate_with.append( + NegationDefaultWithRule( + match_rules=(MatchRule(startswith="ip route "),), + use="no ip route" + ) +) + +# Dynamically extend sectional exiting rules +driver.rules.sectional_exiting.append( + SectionalExitingRule( + match_rules=( + MatchRule(startswith="policy-map"), + MatchRule(startswith="class"), + ), + exit_text="exit", + ) +) + +# Add additional ordering rules dynamically +driver.rules.ordering.append( + OrderingRule( + match_rules=( + MatchRule(startswith="access-list"), + MatchRule(startswith="permit "), + ), + weight=50, + ) +) + +# Add new per-line substitutions dynamically +driver.rules.per_line_sub.append( + PerLineSubRule( + search="^!.*Generated by system.*$", replace="" + ) +) + +# Add new idempotent commands dynamically +driver.rules.idempotent_commands.append( + IdempotentCommandsRule( + match_rules=( + MatchRule(startswith="interface "), + MatchRule(startswith="speed "), + ) + ) +) +``` + +#### Explanation + +* **Dynamic Rule Extension:** You directly modify the driver.rules attributes to append new rules dynamically. +* **Flexibility:** This approach is useful when the driver is instantiated by external code, and subclassing is not feasible. + +Both approaches allow you to extend the functionality of the Cisco IOS driver: + +1. **Subclassing:** Recommended for reusable, modular extensions where the driver logic can be encapsulated in a new class. +2. **Dynamic Modification:** Useful when the driver is instantiated dynamically, and you need to modify the rules at runtime. + ## Creating a Custom Driver + +This guide walks you through the process of creating a custom driver using the `HConfigDriverBase` class from the `hier_config.platforms.driver_base` module. Custom drivers allow you to define operating system-specific rules and behaviors for managing device configurations. + +--- + +### Overview of `HConfigDriverBase` + +The `HConfigDriverBase` class provides a foundation for defining driver-specific rules and behaviors. It encapsulates configuration rules and methods for handling idempotency, negation, and more. You will extend this class to create a new driver. + +Key Components: +1. **`HConfigDriverRules`**: A collection of rules for handling configuration logic. +2. **Methods to Override**: Define custom behavior by overriding the `_instantiate_rules` method. +3. **Properties**: Adjust behavior for negation and declaration prefixes. + +--- + +### Steps to Create a Custom Driver + +#### Step 1: Subclass `HConfigDriverBase` +Begin by subclassing `HConfigDriverBase` to define a new driver. + +```python +from hier_config.platforms.driver_base import HConfigDriverBase, HConfigDriverRules +from hier_config.models import ( + MatchRule, + NegationDefaultWithRule, + SectionalExitingRule, + OrderingRule, + PerLineSubRule, + IdempotentCommandsRule, +) + + +class CustomHConfigDriver(HConfigDriverBase): + """Custom driver for a specific operating system.""" + + @staticmethod + def _instantiate_rules() -> HConfigDriverRules: + """Define the rules for this custom driver.""" + return HConfigDriverRules( + negate_with=[ + NegationDefaultWithRule( + match_rules=(MatchRule(startswith="ip route "),), + use="no ip route" + ) + ], + sectional_exiting=[ + SectionalExitingRule( + match_rules=( + MatchRule(startswith="policy-map"), + MatchRule(startswith="class"), + ), + exit_text="exit" + ) + ], + ordering=[ + OrderingRule( + match_rules=(MatchRule(startswith="interface"),), + weight=10 + ) + ], + per_line_sub=[ + PerLineSubRule( + search="^!.*Generated by system.*$", + replace="" + ) + ], + idempotent_commands=[ + IdempotentCommandsRule( + match_rules=(MatchRule(startswith="interface"),) + ) + ], + ) +``` + +#### Step 2: Customize Negation or Declaration Prefixes (Optional) +Override the `negation_prefix` or `declaration_prefix` properties to customize their behavior. + +```python + @property + def negation_prefix(self) -> str: + """Customize the negation prefix.""" + return "disable " + + @property + def declaration_prefix(self) -> str: + """Customize the declaration prefix.""" + return "enable " +``` + +#### Step 3: Use the Custom Driver + +This section describes how to use the custom driver by extending the `get_hconfig_driver` function and adding a new platform to the `Platform` model. It also covers how to load the driver into Hier Config and utilize it for remediation workflows. + +--- + +##### 1. Extend `get_hconfig_driver` to Include the Custom Driver + +First, modify the `get_hconfig_driver` function to include the new custom driver: + +```python +from hier_config.platforms.driver_base import HConfigDriverBase +from hier_config import get_hconfig_driver +from .custom_driver import CustomHConfigDriver # Import your custom driver +from hier_config.models import Platform + +def get_custom_hconfig_driver(platform: Union[CustomPlatform,Platform]) -> HConfigDriverBase: + """Create base options on an OS level.""" + if platform == CustomPlatform.CUSTOM_DRIVER: + return CustomHConfigDriver() + return get_hconfig_driver(platform) +``` + +##### 2. Create a custom `Platform` to Include the Custom Driver + +```python +from enum import Enum, auto + +class CustomPlatform(str, Enum): + CUSTOM_PLATFORM = auto() +``` + +##### 3. Load the Driver into `HConfig` + +```python +from .custom_platform import CustomPlatform +from hier_config import get_hconfig +from hier_config.utils import load_device_config + +# Load running and intended configurations from files +running_config_text = load_device_config("./tests/fixtures/running_config.conf") +generated_config_text = load_device_config("./tests/fixtures/remediation_config.conf") + +# Create HConfig objects for running and intended configurations +running_config = get_hconfig(CustomPlatform.CUSTOM_DRIVER, running_config_text) +generated_config = get_hconfig(CustomPlatform.CUSTOM_DRIVER, generated_config_text) +``` + +##### 4. Instantiate a `WorkflowRemediation` + +```python +from hier_config import WorkflowRemediation + +# Instantiate the remediation workflow +workflow = WorkflowRemediation(running_config, generated_config) +``` + + +### Key Methods in HConfigDriverBase + +1. `idempotent_for`: + * Matches configurations against idempotent rules to prevent duplication. + +```python +def idempotent_for( + self, + config: HConfigChild, + other_children: Iterable[HConfigChild], +) -> Optional[HConfigChild]: + ... +``` + +2. `negate_with`: + * Provides a negation command based on rules. + +```python +def negate_with(self, config: HConfigChild) -> Optional[str]: + ... +``` + +3. `swap_negation`: + * Toggles the negation of a command. + +```python +def swap_negation(self, child: HConfigChild) -> HConfigChild: + ... +``` + +4. Properties: + * `negation_prefix`: Default is `"no "`. + * `declaration_prefix`: Default is `""`. + +### Example Rule Definitions + +#### Negation Rules +Define commands that require specific negation handling: + +```python +negate_with=[ + NegationDefaultWithRule( + match_rules=(MatchRule(startswith="ip route "),), + use="no ip route" + ) +] +``` + +#### Sectional Exiting +Define how to exit specific configuration sections: + +```python +sectional_exiting=[ + SectionalExitingRule( + match_rules=( + MatchRule(startswith="policy-map"), + MatchRule(startswith="class"), + ), + exit_text="exit" + ) +] +``` + +#### Command Ordering +Set the execution order of specific commands: + +```python +ordering=[ + OrderingRule( + match_rules=(MatchRule(startswith="interface"),), + weight=10 + ) +] +``` + +#### Per-Line Substitution +Clean up unwanted lines in the configuration: + +```python +per_line_sub=[ + PerLineSubRule( + search="^!.*Generated by system.*$", + replace="" + ) +] +``` From eb0e47a3554577bc1eeabe0f558da8a7737283e4 Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 17:38:04 -0600 Subject: [PATCH 06/10] fix url in readthedocs --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index fee1067..5aeaab7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,5 +1,5 @@ site_name: Hierarchical Configuration -site_url: https://hier_config.readthedocs.io/ +site_url: https://hier-config.readthedocs.io/ repo_url: https://github.com/netdevops/hier_config theme: name: readthedocs From 51c62ebbe5e51110921020afd48fae3151d9de2a Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 17:59:30 -0600 Subject: [PATCH 07/10] add docs about config views --- docs/config-view.md | 90 +++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 91 insertions(+) create mode 100644 docs/config-view.md diff --git a/docs/config-view.md b/docs/config-view.md new file mode 100644 index 0000000..b9f9df1 --- /dev/null +++ b/docs/config-view.md @@ -0,0 +1,90 @@ +# Config View + +A Config View is an abstraction layer for network device configurations. It provides a structured, Pythonic way to interact with and extract information from raw configuration data. Config Views are especially useful for standardizing how configuration elements are accessed across different platforms and devices. + +The framework uses a combination of abstract base classes (e.g., `ConfigViewInterfaceBase`, `HConfigViewBase`) and platform-specific implementations (e.g., `ConfigViewInterfaceCiscoIOS`, `HConfigViewCiscoIOS`) to provide a unified interface for interacting with configurations while accounting for the unique syntax and semantics of each vendor or platform. + +## Why Use Config Views? + +1. **Vendor Abstraction:** Network devices from different vendors (Cisco, Arista, Juniper, etc.) have varied configuration formats. Config Views standardize access, making it easier to work across platforms. + +2. **Simplified Interface:** Accessing configuration data becomes more intuitive through Python properties and methods rather than manually parsing text. + +3. **Extensibility:** Easily extendable to support new platforms or devices by implementing platform-specific subclasses. + +4. **Error Reduction:** Encapsulates parsing logic, reducing the risk of errors due to configuration syntax differences. + +## Available Config Views + +| **Property/Method** | **Type** | **Description** | +|-----------------------------|--------------------------------|------------------------------------------------------------------------------| +| `bundle_interface_views` | `Iterable` | Yields interfaces configured as bundles. | +| `config` | `HConfig` | Root configuration object. | +| `dot1q_mode_from_vlans` | `Callable` | Determines 802.1Q mode based on VLANs and tagging. | +| `hostname` | `Optional[str]` | Retrieves the device hostname. | +| `interface_names_mentioned` | `frozenset[str]` | Set of all interface names mentioned. | +| `interface_view_by_name` | `Callable` | Returns view of a specific interface by name. | +| `interface_views` | `Iterable` | Yields all interface views. | +| `interfaces` | `Iterable[HConfigChild]` | Yields raw configuration objects for all interfaces. | +| `interfaces_names` | `Iterable[str]` | Yields the names of all interfaces. | +| `ipv4_default_gw` | `Optional[IPv4Address]` | Retrieves the IPv4 default gateway. | +| `location` | `str` | Returns the SNMP location. | +| `module_numbers` | `Iterable[int]` | Yields module numbers from interfaces. | +| `stack_members` | `Iterable[StackMember]` | Yields stack members configured on the device. | +| `vlan_ids` | `frozenset[int]` | Set of VLAN IDs configured. | +| `vlans` | `Iterable[Vlan]` | Yields VLAN objects, including ID and name. | + +## Example: Cisco IOS Config View + +### Step 1: Parse Configuration + +Assume we have a Cisco IOS configuration file as a string. + +```python +from hier_config import Platform, get_hconfig + + +raw_config = """ +hostname router1 +interface GigabitEthernet0/1 + description Uplink to Switch + switchport access vlan 10 + ip address 192.168.1.1 255.255.255.0 + shutdown +! +vlan 10 + name DATA +""" + +hconfig = get_hconfig(Platform.CISCO_IOS, raw_config) +``` + +### Step 2: Create Config View + +```python +from hier_config.platforms.cisco_ios.view import HConfigViewCiscoIOS + + +config_view = HConfigViewCiscoIOS(hconfig) +``` + +### Step 3: Access Configuration Details + +Access properties to interact with the configuration programmatically: + +```python +# Get the hostname +print(config_view.hostname) # Output: router1 + +# List all interface names +print(list(config_view.interfaces_names)) # Output: ['GigabitEthernet0/1'] + +# Check if an interface is enabled +for interface_view in config_view.interface_views: + print(interface_view.name, "Enabled:", interface_view.enabled) + +# Get all VLANs +for vlan in config_view.vlans: + print(f"VLAN {vlan.id}: {vlan.name}") + +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 5aeaab7..7f73533 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -8,6 +8,7 @@ nav: - Home: index.md - Install: install.md - Getting Started: getting-started.md +- Config View: config-view.md - Drivers: drivers.md - Custom Workflows: custom-workflows.md - Future Config: future-config.md From 63a7ccb5937f51fb37ca2bdea596d0e3f16f4c2e Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 21:18:00 -0600 Subject: [PATCH 08/10] update custom workflow doc --- docs/custom-workflows.md | 166 +++++++++++++++++++++-- tests/fixtures/generated_config_acl.conf | 35 +++++ tests/fixtures/running_config_acl.conf | 22 +++ 3 files changed, 213 insertions(+), 10 deletions(-) create mode 100644 tests/fixtures/generated_config_acl.conf create mode 100644 tests/fixtures/running_config_acl.conf diff --git a/docs/custom-workflows.md b/docs/custom-workflows.md index bc44be5..2711a8b 100644 --- a/docs/custom-workflows.md +++ b/docs/custom-workflows.md @@ -1,34 +1,180 @@ # Creating Custom Workflows -In some scenarios, configuration remediations may require handling beyond the standard negation and idempotency workflows that hier_config is designed to support. Hier Config accommodates these edge cases by allowing the creation of custom remediation workflows, enabling you to integrate them seamlessly with the existing remediation process. +Certain scenarios demand remediation strategies that go beyond the standard negation and idempotency workflows Hier Config is designed to handle. To address these edge cases, Hier Config allows for custom remediation workflows that integrate seamlessly with the existing remediation process. -## Building a Remediation +---- -1. Import and Configuration Loading +## Building a Remediation Workflow + +1. Importing Modules and Loading Configurations + +Start by importing the necessary modules and loading the running and intended configurations for comparison. ```python from hier_config import WorkflowRemediation, get_hconfig, Platform +from hier_config.utils import load_device_config ``` -Necessary modules are imported, including the `WorkflowRemediation` class for handling remediation + +Load the configurations from files: ```python running_config = load_device_config("./tests/fixtures/running_config_acl.conf") generated_config = load_device_config("./tests/fixtures/generated_config_acl.conf") ``` -Here, the current and intended configurations are loaded from files, serving as inputs for comparison and remediation. +These configurations represent the current and desired states of the device. + +---- + +2. Initializing the Workflow Remediation Object: + +Initialize the WorkflowRemediation object for a Cisco IOS platform: + -2. Initialize `WorkflowRemiation`: ```python wfr = WorkflowRemediation( running_config=get_hconfig(Platform.CISCO_IOS, running_config), generated_config=get_hconfig(Platform.CISCO_IOS, generated_config) ) ``` -This initializes `WorkflowRemediation` with the configurations for a Cisco IOS platform, setting up the object that will manage the remediation workflow. +This object manages the remediation workflow between the running and generated configurations. + +---- + +## Extracting and Analyzing Remediation Sections + +### Example: Access-List Custom Remediation + +**Current (Running) Configuration** + +```python +print(wfr.running_config.get_child(startswith="ip access-list")) +``` + +Output: + +``` +ip access-list extended TEST + 12 permit ip 10.0.0.0 0.0.0.7 any + exit +``` + +**Intended (Generated) Configuration** + +```python +print(wfr.generated_config.get_child(startswith="ip access-list")) +``` + +Output: + +``` +ip access-list extended TEST + 10 permit ip 10.0.1.0 0.0.0.255 any + 20 permit ip 10.0.0.0 0.0.0.7 any + exit +``` + +**Default Remediation Configuration** + +```python +print(wfr.remediation_config.get_child(startswith="ip access-list")) +``` + +Output: + +``` +ip access-list extended TEST + no 12 permit ip 10.0.0.0 0.0.0.7 any + 10 permit ip 10.0.1.0 0.0.0.255 any + 20 permit ip 10.0.0.0 0.0.0.7 any + exit +``` + +#### Issues with the Default Remediation: + +1. **Invalid Command:** `no 12 permit ip 10.0.0.0 0.0.0.7 any` is invalid in Cisco IOS. The valid command is `no 12`. +2. **Risk of Lockout:** Removing a line currently matched by traffic could cause a connectivity outage. +3. **Unnecessary Changes:** `permit ip 10.0.0.0 0.0.0.7 any` is a valid line aside from sequence numbers. In large ACLs, this might be unnecessary to delete and re-add. -## Extracting the Targeted Section of a Remediation +---- -## Building a Custom Remediation +#### Goals for Safe Access-List Remediation -## Merging the Remediations +To avoid outages during production changes: + +1. **Resequence the ACL:** Adjust sequence numbers using the ip access-list resequence command. + * For demonstration, resequence to align 12 to 20. +2. **Temporary Allow-All:** Add a temporary rule (1 permit ip any any) to prevent lockouts. +2. **Cleanup:** Remove the temporary rule (no 1) after applying the changes. + +---- + +## Building the Custom Remediation + +1. Create a Custom `HConfig` Object + +```python +from hier_config import HConfig + +custom_remediation = HConfig(wfr.running_config.driver) +``` + +2. Add Resequencing and Extract ACL Remediation + +```python +custom_remediation.add_child("ip access-list resequence TEST 10 10") +custom_remediation.add_child("ip access-list extended TEST") +remediation = wfr.remediation_config.get_child(equals="ip access-list extended TEST") +``` + +3. Build the Custom ACL Remediation + +```python +acl = custom_remediation.get_child(equals="ip access-list extended TEST") +acl.add_child("1 permit ip any any") # Temporary allow-all + +for line in remediation.all_children(): + if line.text.startswith("no "): + # Adjust invalid sequence negation + parts = line.text.split() + rounded_number = round(int(parts[1]), -1) + acl.add_child(f"{parts[0]} {rounded_number}") + else: + acl.add_child(line.text) + +acl.add_child("no 1") # Cleanup temporary rule +``` + +### Output of Custom Remediation + +```python +print(custom_remediation) +``` + +Output: + +``` +ip access-list resequence TEST 10 10 +ip access-list extended TEST + 1 permit ip any any + no 10 + 10 permit ip 10.0.1.0 0.0.0.255 any + 20 permit ip 10.0.0.0 0.0.0.7 any + no 1 + exit +``` + +## Applying the Custom Remediation + +### Remove Invalid Remediation + +```python +invalid_remediation = wfr.remediation_config.get_child(equals="ip access-list extended TEST") +wfr.remediation_config.delete_child(invalid_remediation) +``` + +### Add Custom Remediation + +```python +wfr.remediation_config.merge(custom_remediation) +``` diff --git a/tests/fixtures/generated_config_acl.conf b/tests/fixtures/generated_config_acl.conf new file mode 100644 index 0000000..c5d441c --- /dev/null +++ b/tests/fixtures/generated_config_acl.conf @@ -0,0 +1,35 @@ +hostname aggr-example.rtr +! +ip access-list extended TEST + 10 permit ip 10.0.1.0 0.0.0.255 any + 20 permit ip 10.0.0.0 0.0.0.7 any +! +vlan 2 + name switch_mgmt_10.0.2.0/24 +! +vlan 3 + name switch_mgmt_10.0.3.0/24 +! +vlan 4 + name switch_mgmt_10.0.4.0/24 +! +interface Vlan2 + mtu 9000 + descripton switch_10.0.2.0/24 + ip address 10.0.2.1 255.255.255.0 + ip access-group TEST in + no shutdown +! +interface Vlan3 + mtu 9000 + description switch_mgmt_10.0.3.0/24 + ip address 10.0.3.1 255.255.0.0 + ip access-group TEST in + no shutdown +! +interface Vlan4 + mtu 9000 + description switch_mgmt_10.0.4.0/24 + ip address 10.0.4.1 255.255.0.0 + ip access-group TEST in + no shutdown diff --git a/tests/fixtures/running_config_acl.conf b/tests/fixtures/running_config_acl.conf new file mode 100644 index 0000000..c4fe45b --- /dev/null +++ b/tests/fixtures/running_config_acl.conf @@ -0,0 +1,22 @@ +hostname aggr-example.rtr +! +ip access-list extended TEST + 12 permit ip 10.0.0.0 0.0.0.7 any +! +vlan 2 + name switch_mgmt_10.0.2.0/24 +! +vlan 3 + name switch_mgmt_10.0.4.0/24 +! +interface Vlan2 + descripton switch_10.0.2.0/24 + ip address 10.0.2.1 255.255.255.0 + shutdown +! +interface Vlan3 + mtu 9000 + description switch_mgmt_10.0.4.0/24 + ip address 10.0.4.1 255.255.0.0 + ip access-group TEST in + no shutdown From d36e26f61923975b92d7d41990d1692a76fd5270 Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 21:19:41 -0600 Subject: [PATCH 09/10] update custom remediation --- docs/custom-workflows.md | 41 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/custom-workflows.md b/docs/custom-workflows.md index 2711a8b..0b18c9c 100644 --- a/docs/custom-workflows.md +++ b/docs/custom-workflows.md @@ -178,3 +178,44 @@ wfr.remediation_config.delete_child(invalid_remediation) ```python wfr.remediation_config.merge(custom_remediation) ``` + +### Output of Updated Remediation + +```python +print(wfr.remediation_config) +``` + +Output: + +``` +vlan 3 + name switch_mgmt_10.0.3.0/24 + exit +vlan 4 + name switch_mgmt_10.0.4.0/24 + exit +interface Vlan2 + mtu 9000 + ip access-group TEST in + no shutdown + exit +interface Vlan3 + description switch_mgmt_10.0.3.0/24 + ip address 10.0.3.1 255.255.0.0 + exit +interface Vlan4 + mtu 9000 + description switch_mgmt_10.0.4.0/24 + ip address 10.0.4.1 255.255.0.0 + ip access-group TEST in + no shutdown + exit +ip access-list resequence TEST 10 10 +ip access-list extended TEST + 1 permit ip any any + no 10 + 10 permit ip 10.0.1.0 0.0.0.255 any + 20 permit ip 10.0.0.0 0.0.0.7 any + no 1 + exit +``` \ No newline at end of file From fe69854ce69804d3388348749cb155f2b760f403 Mon Sep 17 00:00:00 2001 From: jtdub Date: Sat, 16 Nov 2024 21:35:29 -0600 Subject: [PATCH 10/10] fix typo --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 7f73533..fc8a703 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -12,6 +12,6 @@ nav: - Drivers: drivers.md - Custom Workflows: custom-workflows.md - Future Config: future-config.md -- JunOS Style Syntax Remediation: junos-style-sytax-remediation.md +- JunOS Style Syntax Remediation: junos-style-syntax-remediation.md - Unified Diff: unified-diff.md - Working with Tags: tags.md