Merge pull request #1647 from ilianiliev-redis/RDSC-3626-adding-opcode-example

RDSC RDSC-3626 - adding opcode examples

Author: andy-stark-redis

content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example.md

@@ -1,5 +1,5 @@
 ---
-Title: Add the opcode to the Redis output
+Title: Using the operation code
 alwaysopen: false
 categories:
 - docs
@@ -8,43 +8,90 @@ categories:
 - rdi
 description: null
 group: di
-linkTitle: Add the opcode to the Redis output
+linkTitle: Using the operation code
 summary: Redis Data Integration keeps Redis in sync with the primary database in near
   real time.
 type: integration
 weight: 100
 ---
 
-In the example below, the data is captured from the source table named `employee` and is written to the Redis database as a JSON document. When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`. 
+The operation code (`opcode`) is a metadata field that indicates the type of operation that generated the change in the source database. It can be useful for tracking changes and understanding the context of the data being processed.
+
+The opcode is only available in the [full row format]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format#full" >}}), and can be accessed in the `transform` and `output` sections of the job file.
+
+It has one of the following values:
+
+- r - Read (applies to only snapshots)
+- c - Create
+- u - Update
+- d - Delete
+- t = Truncate (PostgreSQL specific)
+- m = Message (PostgreSQL specific)
+
+
+You can add the value of the operation code to the output, and also use it in a conditional expression to modify the behavior of the job. The following examples demonstrate the different use-cases.
+
+### Adding the operation code to the output
+
+Use the `add_field` transformation to add a new field that contains the value of the `opcode` field from the source data. Note that the fields must be prefixed with `after` to be included in the output.
 
-Here, the result will be Redis JSON documents with fields captured from the source table
-(`employeeid`, `firstname`, `lastname`) and also with
-an extra field `my_opcode` added using the `merge` update strategy (see the
-[JSON job example]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example" >}})
-for more information). The `opcode` expression refers to the operation code captured from
-the source. This is a database-specific value that indicates which type of operation generated
-the change (insert, update, etc).
 
 ```yaml
 source:
   schema: public
   table: employee
   row_format: full
+
 transform:
+  # add the operation code to the data
   - uses: add_field
     with:
-      field: after.my_opcode
+      field: after.operation_code
       expression: opcode
       language: jmespath
-output:
-  - uses: redis.write
+```
+
+
+### Filtering operation by output code.
+
+In some cases you may want to ignore certain operations (for example, you may not be interested in deletions). Use the `filter` transformation to filter out any operations you don't need to process.
+
+```yaml
+source:
+  schema: public
+  table: employee
+  row_format: full
+
+transform:
+  - uses: filter
+    with:
+      expression: opcode != 'd'
+      language: jmespath
+```
+
+### Modifying the output based on the operation code
+
+The previous example filters out specific operations, but you can also modify the output based on the operation code. For example, you can add a new field that tracks the status of the record based on the operation code.
+
+Note that when a source record is deleted, you must modify the value of the `opcode` field if you want to prevent the corresponding record in the target database from being removed automatically.
+
+```yaml
+source:
+  schema: public
+  table: employee
+  row_format: full
+
+transform:
+  - uses: add_field
     with:
-      data_type: json
-      mapping:
-        - employeeid
-        - firstname
-        - lastname
-        - my_opcode
-      connection: target
-      on_update: merge
-```
+      fields:
+        # Here you set the value of the field based on the value of the opcode field
+        - field: after.status
+          expression: opcode == 'd' && 'inactive' || 'active'
+          language: jmespath
+
+        # You have to change the value of the opcode field to prevent deletion
+        - field: opcode
+          expression: opcode == 'd' && 'u' || opcode
+          language: jmespath
+```

content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md

@@ -63,7 +63,7 @@ With `row_format: full` the input value is a JSON object with the following stru
 - `key` - An object containing the attributes of the primary key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.
 - `before` - An object containing the previous value of the row.
 - `after` - An object containing the current value of the row.
-- `opcode` - The operation code. Different databases use different values for the operation code. See [operation code values]({{< relref "#operation-codes" >}}) below for more information.
+- `opcode` - The operation code. See [Using the operation code]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example" >}}) for more information about the possible opcode values and how to use them.
 - `db` - The database name.
 - `table` - The table name.
 - `schema` - The schema name. 
@@ -102,12 +102,3 @@ output:
         expression: concat(['addresses-full', '#', values(key)[0]])
         language: jmespath
 ```
-
-## Operation code values {#operation-codes}
-
-- r - Read (applies to only snapshots)
-- c - Create
-- u - Update
-- d - Delete
-- t = truncate (PostgreSQL specific)
-- m = message (PostgreSQL specific)