Design Lifecycle (#162)

* feat: Add a new mode that tracks the design deployment providing capacity to updates and decommission (life cycle management) * feat: Provide data protection (optional) for data that has been created or modified by a design deployment. --------- Co-authored-by: Leo Kirchner <[email protected]> Co-authored-by: Gerasimos Tzakis <[email protected]> Co-authored-by: Leo Kirchner <[email protected]> Co-authored-by: Christian Adell <[email protected]> Co-authored-by: Josh VanDeraa <[email protected]> Co-authored-by: Adam Byczkowski <[email protected]>
nautobot · Jun 7, 2024 · a70523a · a70523a
1 parent b17545d
commit a70523a
Show file tree

Hide file tree

Showing 98 changed files with 5,931 additions and 775 deletions.
diff --git a/.bandit.yml b/.bandit.yml
@@ -2,5 +2,5 @@
 skips: []
 # No need to check for security issues in the test scripts!
 exclude_dirs:
-  - "./tests/"
+  - "./nautobot_design_builder/tests/"
   - "./.venv/"
diff --git a/.gitignore b/.gitignore
@@ -305,6 +305,9 @@ invoke.yml
 
 # Docs
 public
+
+# Remove Local Tests
+
 /compose.yaml
 /dump.sql
 /nautobot_design_builder/static/nautobot_design_builder/docs
diff --git a/.yamllint.yml b/.yamllint.yml
@@ -10,4 +10,5 @@ rules:
     quote-type: "double"
 ignore: |
   .venv/
+  .vscode/
   compose.yaml
diff --git a/README.md b/README.md
@@ -15,6 +15,8 @@
 
 Design Builder is a Nautobot application for easily populating data within Nautobot using standardized design files. These design files are just Jinja templates that describe the Nautobot objects to be created or updated.
 
+It also introduces the concept of a design-oriented Source of Truth with a complete lifecycle management of the design deployments (i.e., an instantiation of a design with concrete input data). With this approach, the users of the application can not only create (or populate) data within Nautobot but also update or decommission it while enforcing data protection and dependency.
+
 ## Documentation
 
 Full documentation for this App can be found over on the [Nautobot Docs](https://docs.nautobot.com) website:

diff --git a/development/Dockerfile b/development/Dockerfile
@@ -70,12 +70,12 @@ RUN sort poetry_freeze_base.txt poetry_freeze_all.txt | uniq -u > poetry_freeze_
 # Install all local project as editable, constrained on Nautobot version, to get any additional
 #   direct dependencies of the app
 RUN --mount=type=cache,target="/root/.cache/pip",sharing=locked \
-    pip install -c constraints.txt -e .[all]
+  pip install -c constraints.txt -e .[all]
 
 # Install any dev dependencies frozen from Poetry
 # Can be improved in Poetry 1.2 which allows `poetry install --only dev`
 RUN --mount=type=cache,target="/root/.cache/pip",sharing=locked \
-    pip install -c constraints.txt -r poetry_freeze_dev.txt
+  pip install -c constraints.txt -r poetry_freeze_dev.txt
 
 COPY development/nautobot_config.py ${NAUTOBOT_ROOT}/nautobot_config.py
 # !!! USE CAUTION WHEN MODIFYING LINES ABOVE
diff --git a/development/docker-compose.dev.yml b/development/docker-compose.dev.yml
@@ -11,6 +11,8 @@ services:
     volumes:
       - "./nautobot_config.py:/opt/nautobot/nautobot_config.py"
       - "../:/source"
+      - "../examples/custom_design/designs:/opt/nautobot/designs:cached"
+      - "../examples/custom_design/jobs:/opt/nautobot/jobs:cached"
     healthcheck:
       test: ["CMD", "true"]  # Due to layering, disable: true won't work. Instead, change the test
   docs:
@@ -31,6 +33,8 @@ services:
     volumes:
       - "./nautobot_config.py:/opt/nautobot/nautobot_config.py"
       - "../:/source"
+      - "../examples/custom_design/designs:/opt/nautobot/designs:cached"
+      - "../examples/custom_design/jobs:/opt/nautobot/jobs:cached"
     healthcheck:
       test: ["CMD", "true"]  # Due to layering, disable: true won't work. Instead, change the test
 # To expose postgres or redis to the host uncomment the following

diff --git a/development/nautobot_config.py b/development/nautobot_config.py
@@ -25,6 +25,9 @@
     if "debug_toolbar.middleware.DebugToolbarMiddleware" not in MIDDLEWARE:  # noqa: F405
         MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")  # noqa: F405
 
+if "nautobot_design_builder.middleware.GlobalRequestMiddleware" not in MIDDLEWARE:  # noqa: F405
+    MIDDLEWARE.insert(0, "nautobot_design_builder.middleware.GlobalRequestMiddleware")  # noqa: F405
+
 #
 # Misc. settings
 #
@@ -164,4 +167,18 @@
         subprocess.check_call(command, shell=False)  # nosec
     PLUGINS.append("nautobot_bgp_models")
 
-PLUGINS_CONFIG = {"design_builder": {"context_repository": os.getenv("DESIGN_BUILDER_CONTEXT_REPO_SLUG", None)}}
+
+def pre_decommission_hook_example(design_instance):
+    return True, "Everything good!"
+
+
+PLUGINS_CONFIG = {
+    "nautobot_design_builder": {
+        "context_repository": os.getenv("DESIGN_BUILDER_CONTEXT_REPO_SLUG", None),
+        "pre_decommission_hook": pre_decommission_hook_example,
+        "protected_models": [("dcim", "region"), ("dcim", "device"), ("dcim", "interface")],
+        "protected_superuser_bypass": False,
+    }
+}
+
+STRICT_FILTERING = False
diff --git a/docs/admin/compatibility_matrix.md b/docs/admin/compatibility_matrix.md
@@ -1,5 +1,6 @@
 # Compatibility Matrix
 
 | Nautobot Design Builder Version | Nautobot First Support Version | Nautobot Last Support Version |
-| ------------- | -------------------- | ------------- |
-| 1.0.X         | 1.6.0                | 2.9999         |
+| ------------------------------- | ------------------------------ | ----------------------------- |
+| 1.1.X                           | 1.6.0                          | 1.99.99                       |
+| 2.X                             | 2.0.0                          | 2.99.99                       |
diff --git a/docs/admin/install.md b/docs/admin/install.md
@@ -47,6 +47,46 @@ PLUGINS = ["nautobot_design_builder"]
 # }
 ```
 
+### Data Protection
+
+Data protection allows enforcing consistent protection of data owned by designs.
+
+There are two data protection configuration settings, and this is how you can manage them.
+
+#### Define the Protected Data Models
+
+By default, no data models are protected. To enable data protection, you should add it under the `PLUGINS_CONFIG`:
+
+```python
+PLUGINS_CONFIG = {
+    "nautobot_design_builder": {
+        "protected_models": [("dcim", "location"), ("dcim", "device")],
+        ...
+    }
+}
+```
+
+In this example, data protection feature will be only taken into account for locations and devices.
+
+#### Bypass Data Protection for Super Users
+
+First, you have to enable a middleware that provides request information in all the Django processing.
+
+```python
+MIDDLEWARE.insert(0, "nautobot_design_builder.middleware.GlobalRequestMiddleware")
+```
+
+Finally, you have to tune the default behavior of allowing superuser bypass of protection (i.e., `True`).
+
+```python
+PLUGINS_CONFIG = {
+    "nautobot_design_builder": {
+        "protected_superuser_bypass": False,
+        ...
+    }
+}
+```
+
 Once the Nautobot configuration is updated, run the Post Upgrade command (`nautobot-server post_upgrade`) to run migrations and clear any cache:
 
 ```shell

diff --git a/docs/dev/contributing.md b/docs/dev/contributing.md
@@ -24,7 +24,11 @@ Example designs should be placed in the top level `examples/` directory, as appr
 
 ## Branching Policy
 
-The active branch in Design Builder is the `develop` branch. However, commits are not allowed directly to this branch. Instead, fork the code and open a pull request to `develop`.
+The branching policy includes the following tenets:
+
+- The `develop` branch is the branch of the next major and minor paired version planned.
+- PRs intended to add new features should be sourced from the `develop` branch.
+- PRs intended to fix issues in the Nautobot LTM compatible release should be sourced from the latest `ltm-<major.minor>` branch instead of `develop`.
 
 ## Release Policy
 

diff --git a/docs/images/screenshots/design-deployment-decommissioning.png b/docs/images/screenshots/design-deployment-decommissioning.png
diff --git a/docs/images/screenshots/design-deployment-detail.png b/docs/images/screenshots/design-deployment-detail.png
diff --git a/docs/images/screenshots/design-deployments.png b/docs/images/screenshots/design-deployments.png
diff --git a/docs/images/screenshots/designs.png b/docs/images/screenshots/designs.png
diff --git a/docs/user/app_getting_started.md b/docs/user/app_getting_started.md
@@ -12,6 +12,8 @@ The easiest way to experience Design Builder is to either add the [demo-designs]
 
 ## What are the next steps?
 
+<!-- TODO: update with the new Design Navigation and new screenshoots -->
+
 The Design Builder demo designs ship with some sample designs to demonstrate capabilities. Once the application stack is ready, you should have several jobs listed under the "Jobs" -> "Jobs" menu item.
 
 ![Jobs list](../images/screenshots/sample-design-jobs-list.png)

diff --git a/docs/user/app_overview.md b/docs/user/app_overview.md
@@ -7,12 +7,16 @@ This document provides an overview of the App including critical information and
 
 ## Description
 
-Design Builder provides a system where standardized network designs can be developed to produce collections of objects within Nautobot. These designs are text based templates that can create and update hierarchical data structures within Nautobot.
+Design Builder provides a system where standardized network designs can be developed to produce or update collections of objects within Nautobot. These designs are text based templates that can create and update hierarchical data structures within Nautobot.
+
+The deployment of a design allows a complete lifecycle management of all the changes connected as a single entity. This means that a design deployment (i.e., a concrete combination of input data with a design) can be updated or decommissioned after its creation, and all the data changes introduced can be enforced even when accessing the data outside of the design builder app.
 
 ## Audience (User Personas) - Who should use this App?
 
 - Network engineers who want to have reproducible sets of Nautobot objects based on some standard design.
 - Automation engineers who want to be able to automate the creation of Nautobot objects based on a set of standard designs.
+- Users who want to leverage abstracted network services defined by network engineers in a simplfied way.
+- Network Managers who need a design-driven point of view of the network more abstract than per device. For example, getting the bill of materials (BOM) for a concrete deployment.
 
 ## Authors and Maintainers
 
@@ -21,4 +25,5 @@ Design Builder provides a system where standardized network designs can be devel
 
 ## Nautobot Features Used
 
-This application interacts directly with Nautobot's Object Relational Mapping (ORM) system.
+- This application interacts directly with Nautobot's Object Relational Mapping (ORM) system.
+- It uses (optionally) `CustomValidators` and `pre_delete` signals to enforce data protection for existing design deployments.
diff --git a/docs/user/design_development.md b/docs/user/design_development.md
@@ -16,7 +16,7 @@ For the remainder of this tutorial we will focus solely on the Design Job, Desig
 
 Designs can be loaded either from local files or from a git repository. Either way, the structure of the actual designs and all the associated files is the same. Since, fundamentally, all designs are Nautobot Jobs, everything must be in a top level `jobs` python package (meaning the directory must contain the file `__init__.py`) and all design classes must be either defined in this `jobs` module or be imported to it. The following directory layout is from the [demo designs repository](hhttps://github.com/nautobot/demo-designs):
 
-``` bash
+```bash
 jobs
 ├── __init__.py
 ├── core_site
@@ -78,7 +78,7 @@ Primary Purpose:
 - Provide the user inputs
 - Define the Design Context and Design Templates
 
-As previously stated, the entry point for all designs is the `DesignJob` class.  New designs should include this class in their ancestry. Design Jobs are an extension of Nautobot Jobs with several additional metadata attributes. Here is the initial data job from our sample design:
+As previously stated, the entry point for all designs is the `DesignJob` class. New designs should include this class in their ancestry. Design Jobs are an extension of Nautobot Jobs with several additional metadata attributes. Here is the initial data job from our sample design:
 
 ```python
 --8<-- "https://raw.githubusercontent.com/nautobot/demo-designs/main/jobs/initial_data/__init__.py"
@@ -106,6 +106,10 @@ Design file specifies the Jinja template that should be used to produce the inpu
 
 Design files specifies a list of Jinja template that should be used to produce the input for the design builder. The builder will resolve the files' locations relative to the location of the design job class. Exactly one of `design_file` or `design_files` must be present in the design's Metadata. If `design_files` is used for a list of design templates, each one is evaluated in order. The same context and builder are used for all files. Since a single builder instance is used, references can be created in one design file and then accessed in a later design file.
 
+### `design_mode`
+
+The `design_mode` indicates how this design's state will be tracked in Nautobot. By default, the design mode is set to `CLASSIC` mode, which means that objects are created in an ad-hoc fashion and no change sets are created. Classic mode is how all designs worked prior to the introduction of the design lifecycle features. If a design is intended to be tracked as a deployment, then design mode should be set to `DEPLOYMENT` in order to implement the full lifecycle.
+
 ### `context_class`
 
 The value of the `context_class` metadata attribute should be any Python class that inherits from the `nautobot_design_builder.Context` base class. Design builder will create an instance of this class and use it for the Jinja rendering environment in the first stage of implementation.
@@ -114,6 +118,18 @@ The value of the `context_class` metadata attribute should be any Python class t
 
 This attribute is optional. A report is a Jinja template that is rendered once the design has been implemented. Like `design_file` the design builder will look for this template relative to the filename that defines the design job. This is helpful to generate a custom view of the data that was built during the design build.
 
+### `version`
+
+It's an optional string attribute that is used to define the versioning reference of a design job. This will enable in the future the versioning lifecycle of design deployments. For example, one a design evolves from one version to another, the design deployment will be able to accommodate the new changes.
+
+### `description`
+
+This optional attribute that is a string that provides a high-level overview of the intend of the design job. This description is displayed int the design detail view.
+
+### `docs`
+
+This attribute is also displayed on the design detail view. The `docs` attribute can utilize markdown format and should provide more detailed information than the description. This should help the users of the `Design` to understand the goal of the design and the impact of the input data.
+
 ## Design Context
 
 Primary Purpose:
@@ -141,6 +157,8 @@ Now let's inspect the context YAML file:
 
 This context YAML creates two variables that will be added to the design context: `core_1_loopback` and `core_2_loopback`. The values of both of these variables are computed using a jinja template. The template uses a jinja filter from the `netutils` project to compute the address using the user-supplied `site_prefix`. When the design context is created, the variables will be added to the context. The values (from the jinja template) are rendered when the variables are looked up during the design template rendering process.
 
+> Note: The `Context` class also contains a property to retrieve the `Tag` associated with the design and attached to all the objects with full_control. With this tag you can check for data in objects already created when the design is updated, for example: `.filter(tags__in=[self.design_instance_tag]`.
+
 ### Context Validations
 
 Sometimes design data needs to be validated before a design can be built. The Design Builder provides a means for a design context to determine if it is valid and can/should the implementation proceed. After a design job creates and populates a design context, the job will call any methods on the context where the method name begins with `validate_`. These methods should not accept any arguments other than `self` and should either return `None` when valid or should raise `nautobot_design_builder.DesignValidationError`. In the above Context example, the design context checks to see if a site with the same name already exists, and if so it raises an error. Any number of validation methods can exist in a design context. Each will be called in the order it is defined in the class.
@@ -178,8 +196,8 @@ Double underscores between a `field` and a `relatedfield` cause design builder t
 
 ```yaml
 devices:
-- name: "switch1"
-  platform__name: "Arista EOS"
+  - name: "switch1"
+    platform__name: "Arista EOS"
 ```
 
 This template will attempt to find the `platform` with the name `Arista EOS` and then assign the object to the `platform` field on the `device`. The value for query fields can be a scalar or a dictionary. In the case above (`platform__name`) the scalar value `"Arista EOS"` expands the the equivalent ORM query: `Platform.objects.get(name="Arista EOS")` with the returned object being assigned to the `platform` attribute of the device.
@@ -188,10 +206,10 @@ If a query field's value is a dictionary, then more complex lookups can be perfo
 
 ```yaml
 devices:
-- name: "switch1"
-  platform: 
-    name: "Arista EOS"
-    napalm_driver: "eos"
+  - name: "switch1"
+    platform:
+      name: "Arista EOS"
+      napalm_driver: "eos"
 ```
 
 The above query expands to the following ORM code: `Platform.objects.get(name="Arista EOS", napalm_driver="eos")` with the returned value being assigned to the `platform` attribute of the device.
@@ -255,7 +273,7 @@ When used as a YAML mapping key, `!ref` will store a reference to the current Na
 ```jinja
 # Creating a reference to spine interfaces.
 #
-# In the rendered YAML this ends up being something like 
+# In the rendered YAML this ends up being something like
 # "spine_switch1:Ethernet1", "spine_switch1:Ethernet2", etc
 #
 #
@@ -275,7 +293,7 @@ When used as the value for a key `!ref:<reference_name>` will return the the pre
 
 ```jinja
 # Looking up a reference to previously created spine interfaces.
-# 
+#
 # In the rendered YAML "!ref:{{ spine.name }}:{{ interface }}" will become something like
 # "!ref:spine_switch1:Ethernet1", "!ref:spine_switch1:Ethernet2", etc
 # ObjectCreator will be able to assign the cable termination A side to the previously created objects.