fix(#9601): prototype duplicate prevention #9609

ChinHairSaintClair · 2024-11-04T12:52:57Z

Description

A prototype to prevent duplicate hierarchy contact siblings from being created as discussed with @jkuester and @mrjones-plip in a "technical working session". We expect a lot of feedback, changes, and further discussion before it's ready for approval.

To achieve this, we hook into duplicate detection strategies through "configuration", amend the promise fired on submit in the contacts-edit.component.ts file to run our additional check, and finally output the possible duplicate items to a duplicate_info section added to the enketo.component.html file.

Configuration:

_phdcChanges: { // Additional namespace
  // Specify your own contact_types here
  hierarchyDuplicatePrevention: Partial<{[key in 'person' | 'health_center']: Strategy;}>;
  // The Partial utility ensures that only the allowed keys (health_center, clinic, person, etc) are used, but none are mandatory.
};

Where the keys should match the contact_types listed in your base_settings.json file.

Currently two strategies are available, Levenshtein and NormalizedLevenshtein, with the ability to customize properties based on implementation needs.

Example implementation:

window._phdcChanges = {
  hierarchyDuplicatePrevention: {
    health_center: {
      ...Levenshtein,
      props: [
        {form_prop_path: `/data/health_center/name`, db_doc_ref: 'name'},
        {form_prop_path: '/data/health_center/external_id', db_doc_ref: 'external_id'}
      ],
      queryParams: {
        valuePaths: ['/data/health_center/is_user_flagged_duplicate', '/data/health_center/duplicate/action'],
        // eslint-disable-next-line eqeqeq
        query: (duplicate, action) => duplicate === 'yes' && action != null
      }
    }
  }
}

Where props are the definitions that should be used to evaluate how likely the current record is to its siblings. If no props value is provided, "name" will be used by default. E.g:

For a "location" we might want to match on street_number, street_name, and postal_code
For a "person" we might want to match on name, sex, and date_of_birth.

form_prop_path is the xml path to the interested value on the currently created/edited form. db_doc_ref would the property name of the sibling database document that the resolved value of the form_prop_path should be compared to. E.g:
Suppose form has a structure of: <data><clinic><name>Test</name></clinic></data>. The path would be: /data/clinic/name. The "clinic" sibling documents would have a property of name.

Finally, we have a use case where we need to conditionally fire the duplicate check based on if our CHW has confirmed a record to be a duplicate, after our backend has flagged the item as such. We use the queryParams's valuePaths to specify the xml path of the question, or questions, that would allow us to mark a record as a duplicate, submit it, and then merge or delete it downstream based on the specified action.

Misc:

We use the CHT provided medic-client/contacts_by_parent view to query for siblings on form init. The request gets processed in the background giving larger queries the opportunity to make progress before the result gets awaited in the form submit.

@kennsippell, since we've touched on the duplicate topic before, it would be great to get your thoughts as this as well.

#Issue
#9601

Code review checklist

Readable: Concise, well named, follows the style guide, documented if necessary.
Documented: Configuration and user documentation on cht-docs
Tested: Unit and/or e2e where appropriate
Internationalised: All user facing text
Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

fardarter · 2024-11-13T05:57:33Z

webapp/src/ts/main.ts

+      queryParams: {
+        valuePaths: ['/data/health_center/is_user_flagged_duplicate', '/data/health_center/duplicate/action'],
+        // eslint-disable-next-line eqeqeq
+        query: (duplicate, action) => duplicate === 'yes' && action != null


@jkuester @ChinHairSaintClair My feeling here is that this would need to end up with some kind of operator syntax support that lets the user do logic, maybe something like (though doesn't have to be) JsonLogic: https://jsonlogic.com/

The way I see it there are two point of config -- the excel files and the configuration json -- and probably neither should take serialised JS (?).

If we return the key => { key, value } as a tuple or object on key lookup, we could let the user express logic maybe like this (with the value found at the key tested against the value provided (with) using the operator):

let andOnly = { logic: [ [ // array bracket denotes a logical grouping { key1: { with: value, op: $in }, key2: { with: value, op: $contains }, key3: { with: value, op: $eq }, }, { key1: { with: value, op: $in }, key2: { with: value, op: $ne }, key3: { with: value, op: $startsWith }, }, ], ], }; let orWithNestedAnds = { logic: [ [ { key1: { with: value, op: $in }, }, { key2: { with: value, op: $eq }, }, ], // each grouping in its own bracket [ { key1: { with: value, op: $in }, }, { key2: { with: value, op: $in }, }, ], ], }; let orWithNestedOrWithNestedAnds = { logic: [ [ { key1: { with: value, op: $in }, }, { key2: { with: value, op: $in }, }, ], // OR [ [ // nested groupings acceptable { key1: { with: value, op: $ne }, }, // AND { key2: { with: value, op: $ne }, }, ], // OR [ { key1: { with: value, op: $contains }, }, // AND { key1: { with: value, op: $startsWith }, }, ], ], ], };

The other stuff seems easier to move to config.

fardarter · 2024-11-13T05:59:40Z

webapp/src/ts/polyfills.ts

@@ -88,7 +88,14 @@ declare global {
    CHTCore: any;
    angular: any;
    EnketoForm:any;
+    _phdcChanges: { // Additional namespace
+      // Specify your own contact_types here
+      hierarchyDuplicatePrevention: Partial<{[key in 'person' | 'health_center']: Strategy;}>;


@jkuester @ChinHairSaintClair Following on from my above comment:

The way I see it there are two point of config -- the excel files and the configuration json.

We probably couldn't expect the user to do config here. I agree the type safety is lovely and I'm not sure whether it's possible to generate a type from the config, but if not I don't see how we can keep the type safety and still maintain the existing configuration contract.

fardarter · 2024-11-13T06:02:59Z