docs(ldif-producer): add detailed test and architecture documentation

Author: Y0-L0

README.md

@@ -20,10 +20,9 @@ The repository does include a compose file to start things up quickly. It will
 start three services:
 
 - `ldap-server` - The OpenLDAP server.
-- `ldap-notifier` - The Univention Directory Notifier.
+- `ldif-producer` - Generates a provisioning message from every LDAP write transaction, replaces notifier/listener
+- `ldap-notifier` - The Univention Directory Notifier. (deprecated, about to be removed)
 - `ldap-admin` - An instance of phpLDAPadmin as a web UI to access
-  `ldap-server`.
-
 
 To set it up:
 
@@ -32,27 +31,35 @@ To set it up:
 
 2. Bring up the services by running:
 
-   ```
-   docker compose up
-   ```
+```sh
+docker compose up
+```
 
-The web UI is by default available at <http://localhost:8001>.
+To always start with a fresh LDAP database,
+always build with the latest changes
+and check for mew remote images, run this optimized command:
 
+```sh
+docker compose down -v && docker compose up --pull always --build \
+ldif-producer ldap-server nats
+```
 
+The web UI is by default available at <http://localhost:8001>.
 
 ## Interacting with `ldap-server`
 
 
 From the command line if you have the required tools available:
 
-```
+```sh
 ldapwhoami -H ldap://localhost:389 -x -D cn=admin,dc=univention-organization,dc=intranet -w univention
 
 ldapsearch -H ldap://localhost:389 -x -D cn=admin,dc=univention-organization,dc=intranet -w univention -b dc=univention-organization,dc=intranet
 ```
 
+## Interacting with the `ldap-server`
 
-## Interacting with the `ldap-notifier`
+## Interacting with the `ldap-notifier` (about to be deprecated)
 
 One option is to connect the base listener to the running notifier, this does
 involve manual tweaking at the moment though. The process is roughly as follows:
@@ -61,9 +68,10 @@ involve manual tweaking at the moment though. The process is roughly as follows:
   via `docker compose`. Set the `.env.listener` according to your local
   containers.
 
-
 ## Manually testing a full round trip
 
+**deprecated** because the phpLDAPadmin doesn't have the necessary LDAP Controls.
+
 The easiest way is to open phpLDAPadmin and change the description of the admin
 user.
 
@@ -89,7 +97,6 @@ Have the `container-listener-base` and the services from this repository running
 ## Linting
 
 You may run the pre-commit linter as follows:
-
 ```
 docker compose run pre-commit
 ```
@@ -130,7 +137,7 @@ as a comma-separated list of values found in the [OpenLDAP documentation](https:
 
 The default is `ldap/debug/level: stats`.
 
-## Notifier Data Files
+## Notifier Data Files (deprecated, will soon be removed)
 
 ### OpenLDAP translog output file
 

docs/ldif_producer_activity_diagram.puml

@@ -12,8 +12,10 @@ fork
 
 partition "Send Messages to NATS" {
 repeat
-:get message from queue;
-:send message to NATS;
+:get message from
+`outgoing_queue`;
+:send message to NATS
+into LDIF_PRODUCER queue;
 :remove message from journal;
 repeat while ( no SIGINT / SIGTERM )
 }
@@ -24,16 +26,15 @@ fork again
     fork
         repeat
             :listen for messages in the socket;
-            :handle_request()
+            :slapdsock.service
+            .SlapdSockServer.handle_request()
             ====
             handle_request simply puts
-            messages in a python queue
+            incoming socket messages into
+            the `requests` python queue
             so that multiple threads
             can work on them.;
 
-            :slapdsock.service
-            .SlapdSockServer.handle_request()
-            put message into queue;
         repeat while ( no SIGINT / SIGTERM )
 
     fork again
@@ -48,20 +49,24 @@ fork again
                 and determine the ldap operation;
                 switch (ldap_operation_type)
                 case ( RESULT )
-                    partition "do_result()" {
+                    partition "LDAPHandler.\ndo_result()" {
                         :write message to journal;
                         :put message into `outgoing_queue`;
                         :release back-pressure
-                        by removing an object from the `backpressure_queue`;
+                        by removing an object from
+                        the `backpressure_queue`;
                     }
                 case ( DELETE / MODIFY / MODRDN )
-                    :do_delete()
+                    :LDAPHandler.
+                    do_add()
+                    do_delete()
                     do_modify()
                     do_modrdn()
                     ====
-                    add object to `backpressure_queue`;
+                    add object to
+                    `backpressure_queue`;
                 case ( Everything else )
-                    :do_add()
+                    :LDAPHandler.
                     do_bind()
                     do_search()
                     ...
@@ -73,12 +78,6 @@ fork again
     fork again
         partition "SocketServer\nWorker  Thread" {
         }
-    fork again
-        partition "SocketServer\nWorker  Thread" {
-        }
-
-    end fork
-        }
 
 end fork
 

docs/ldif_producer_activity_diagram.svg

@@ -0,0 +1,124 @@
+# LDIF-Producer
+
+The LDIF-Producer produces a provisioning message
+for every LDAP write transaction (`ADD`, `DELETE`, `MODIFY`, `MODRDN`)
+and pushes them into a NATS queue.
+
+The messages are still in the LDAP Object representation.
+A separate component is responsible for converting the Object
+to it's UDM representation.
+
+## High-level architecture
+
+The LDIF-Producer combines two worlds.
+
+The slapd-sock-server based on `socketserver`, which is itself `threading`-based.
+And the official NATS python client library which is `asyncio`-based.
+
+The socketserver main thread and the asyncio even loop
+run in separate threads and are connected via the `outgoing_queue`
+synchronous python queue.
+The socketserver internally uses two additional python queues.
+They will be explained later.
+
+the ldif_producer.controller python module
+configures and starts both components,
+owns the `outgoing_queue` and initializes a graceful shutdown
+via a `signal_handler`.
+
+## slapd-sock-server
+
+the LDIF-Producer hooks into the LDAP-Server
+via the back-sock LDAP overlay module.
+It provides Pre-Transaction and Post-Transaction hook capabilities
+via a UNIX TCP Socket.
+One problem with this approach is, that LDAP requests
+can only be blocked in the Pre-hook (ADDRequset, DELETERequest, MODIFYRequest...)
+and only the Post-hook (RESULTRequest)
+provides the necessary LDAP object / LDIF data
+to construct the provisioning message.
+
+This leads to the requirements that:
+
+1. the Post-hook must locally persist the message content as soon as possible.
+2. The first action after startup must be to check the transaction journal
+    and send any remaining messages to NATS.
+3. A post-hook handler function must never be aborted before it persisted
+    it's messages to the transaction journal.
+4. modify requests need to be throtteled in the Pre-hook
+    to not overload the downstream tasks.
+
+Requirements one and two are not yet implemented
+in the first alpha version of this component.
+But it's already prepared in the `LDAPHandler.do_result()` function.
+Requirement three is implemented via a signal_handler
+that propagates an exit event to all threads and asyncio tasks
+and careful tuning of the python queue and socket polling timeout
+so that the exit event is evaluated with a reasonable frequency.
+(about once per second)
+Requirement four is implemented through a python `backpressure_queue`.
+The pre-hooks add an object to the `backpressure_queue` for ever LDAP write request
+and the post-hook removes an item after it has processed the request.
+While processing the request, the message is added to the `outgoing_queue`.
+That queue has a maximum size and blocks adding an item if it is reached.
+Items are removed from the `outgoing_queue`
+only after being successfully sent to NATS.
+This way, a slow/unresponsive NATS
+slows/stops the LDAP server from processing write requests.
+The pre-hook waits for a configurable timeout
+for a `backpressure_queue` seat to become available,
+before cancelling the LDAP Request with a custom error:
+
+```text
+RESULT
+code: 51
+matched: <DN>
+info: slapdsocklistener busy sending messages to the message queue\n
+```
+
+Code 51 = `LDAP_BUSY`
+
+### Classes and threads
+
+The LDIF-Producer acts as the UNIX Socket server
+while the back-sock LDAP overlay module is the client.
+
+The LdifProducerSlapdSockServer or rather it's parent classes
+create and own the UNIX socket and the `requests` python queue.
+It is instantiated in the main controller and executed
+as a blocking and long-running process
+in the socketserver main thread.
+It listens for TCP requests on the socket and puts them into the `requests` queue.
+The socketserver main thread also spawns worker threads
+that consume the `requests` queue.
+
+The logic for how to respond to back-sock hook requests
+is defined in the LDAPHandler class and it's parents.
+It is instantiated as a signleton,
+owns and instantiates the `backpressure_queue`
+and recieves a reference to the `outgoing_queue`
+from the main controller.
+
+## Performance Considerations
+
+Multiple slapd-sock worker threads can and should be configured
+to quickly respond to the LDAP-Server.
+
+But the `backpressure_queue` is currently hard-coded to 1
+This configuration means that many read requests
+can be handeled concurrently, but only one write-request
+can happen at a given time.
+This choice was made because we currently don't know
+how to preserve the message order
+with multiple in-flight write transactions.
+
+I expect that the bottleneck will be sending messages to NATS
+but don't know how to parallelize this operation
+without compromising the message order.
+
+## Multi-Master LDAP-Server support
+
+still under discussion.
+We expect that queue deduplication by the NATS server
+based on a unique transaction id
+will do the heavy lifting.

docs/ldif_producer_architecture.md

@@ -1,6 +1,5 @@
 # Test Organization
 
-
 ## TL;DR run the tests
 
 In a container:
@@ -11,51 +10,66 @@ docker compose up --build test
 
 Locally:
 
-```shell
-# Use "pipenv" to have the right environment
-pipenv sync -d
-pipenv run pytest
-
-# Get a shell
-pipenv shell
+```sh
+poetry install
+poetry shell
+pytest --cov=univention/provisioning/ --cov-report term-missing -v
 ```
 
-
 ## Target structure
 
 The target structure for testing shall eventually follow this pattern:
 
 ```
-└── tests  # top-level tests folder
-    ├── README.md  # explains test organization inside this folder
-    ├── unit  # name this unit_and_integration if keeping both here
-    ├── integration  # optional: omit if kept together with unit tests
+└── tests
+    ├── README.md
+    ├── unit
+    ├── integration
     └── e2e
 ```
 
 ### Unit tests
 
-The *unit* under test is in this repository the "container":
+Unit tests can run without any external dependencies.
 
-- `ldap-server` is the OpenLDAP server
+The unittests in this repository test the python code
+of the the LDIF-Producer slapd-socket server.
 
-- `ldap-notifier` is the Univention Directory Notifier
+the LDIF-Producer is a multithreaded application
+using the python threading library.
+Due to this fact, some unittests take around a second to execute
+or more explicitly to shut down
+because the exit signal can only be evaluated
+between polling intervals.
 
-Simple checks which focus on a single container are kept in the subdirectory
-`unit`.
+The multithreaded nature of the LDIF-Producer
+makes debugging test failures significantly harder.
+Tests can seem to be "hanging forever"
+when a worker thread throws an exception.
 
+The LDIF-Producer implements extensive logging to help in debugging.
+In a pytest run, you can activate the logs with the `-s` and `--full-trace`
+cli arguments.
 
-### Integration tests
+As a last resort, you can separate `stdout` from `stderr`
+you can sort the logs by process id to reveal different patterns.
+
+```sh
+docker logs dev-local-ldif-producer-1 > ldif-producer.log \
+&& docker logs dev-local-ldif-producer-1 2>> ldif-producer.log \
+&& vim ldif-producer.log
+```
 
-The current integration tests are in the folder `integration-test` in the root
-of this repository. The aim is to migrate those eventually into the folder
-`integration` as shown above in the overview.
+vim sort command:
+`:sort /\[\d \d*\]/`
 
-Tests which check the *integration* of multiple containers will be grouped into
-the folder `integration` as shown above in the overview.
+### Integration tests
 
-New tests are written as plain `pytest` based test cases.
+Integration tests are all test that depend on separately started containers.
+Those can be the `ldap-server`, `ldif-producer` and `nats`.
+Integration tests may need only one, some or all of those containers
 
+the tests in the `integration-tests` folder are assumed to be deprecated.
 
 ### End to end tests