improve performance when purifying tokens

[MegaphoneJon]

Overview

We think of token rendering as expensive. But 90% of the time spent in Civi\Token\TokenRow->render() is spent on a getFields() API call whose only purpose is to check if a field might have rich text so that we can purify it.

We can replace this with a call to \Civi\Api\Request->entityFields(). In my testing, when combined with the caching in #32210, this reduces the time spent in Civi/Token/TokenRow::fill() from 24 seconds to .28 seconds.

Before

Slow.

After

Fast.

Comments

My main concern is that getFields returns custom fields in API3 and not in API4. However, in my testing, the custom fields are already populated in $htmlTokens, so this has no effect.

[civibot]

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...

• If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
• A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
• A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
• If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
• Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
• A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
• To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
• For more information about contributing, see CONTRIBUTING.md.

Quick links for reviewers...

• ➡️ PR Demo site (can be rebuilt with the command test this please).
• 📖 Review standards
• 🗒️ Review template (brief or verbose)

➡️ Online demo of this PR 🔗

[MegaphoneJon]

PS - anyone else who runs XDebug that can replicate my numbers, I'd appreciate it! I assume there's nothing that odd about my setup that would cause such different results on getFields() calls but it'd be good to confirm. Sending a mailing to 1,000 contacts is a good way to gather this profile.

[eileenmcnaughton]

@MegaphoneJon oout of curiousity - is the apiv4 any different to v3?

[colemanw]

I'm confused about this. How the heck did it ever work before? Api3.getfields does not return "data_type". The output from the Api3 explorer looks like this:

        "activity_details": {
            "name": "details",
            "type": 32,
            "title": "Details",
            "description": "Details about the activity (agenda, notes, etc).",
            "rows": 8,
            "cols": 60,
            "usage": {
                "import": true,
                "export": true,
                "duplicate_matching": true,
                "token": false
            },
            "import": true,
            "where": "civicrm_activity.details",
            "export": true,
            "table_name": "civicrm_activity",
            "entity": "Activity",
            "bao": "CRM_Activity_BAO_Activity",
            "localizable": 0,
            "html": {
                "type": "RichTextEditor",
                "rows": 8,
                "cols": 60
            },
            "add": "1.1",
        }            

[seamuslee001]

@colemanw @MegaphoneJon It seemed to get returned for custom fields tho

This is doing a civicrm_api3('contact', 'getfields', ['api_action' => 'get']); on smaster FYI

        "custom_3": {
            "id": "3",
            "label": "Marriage Date",
            "headerPattern": "//",
            "title": "Marriage Date",
            "custom_field_id": "3",
            "groupTitle": "Constituent Information",
            "data_type": "Date",
            "name": "custom_3",
            "type": 4,
            "html_type": "Select Date",
            "default_value": null,
            "text_length": null,
            "options_per_line": null,
            "custom_group_id": "1",
            "extends": "Individual",
            "is_search_range": "0",
            "extends_entity_column_value": null,
            "extends_entity_column_id": "",
            "is_view": "0",
            "is_multiple": "0",
            "option_group_id": null,
            "date_format": "mm/dd/yy",
            "time_format": 0,
            "is_required": "0",
            "table_name": "civicrm_value_constituent_information_1",
            "column_name": "marriage_date_3",
            "serialize": 0,
            "where": "civicrm_value_constituent_information_1.marriage_date_3",
            "extends_table": "civicrm_contact",
            "search_table": "contact_a"
        },

[MegaphoneJon]

@eileenmcnaughton @seamuslee001 I address the v3/v4 and its custom fields in my original post. But now there are failing tests so hopefully I can address concerns that way.

[colemanw]

Now I'm wondering if this code does anything at all. If

1. This code only checks data_type if the field is not custom,
2. And Api3 only returns data_type if the field is custom,
3. Then this code is useless

[eileenmcnaughton]

looks like it started here #13283