feat(search): implement FST5 w/ sqlite for faster and better searching

[perfectra1n]

No description provided.

To fix the problem, ensure the replacement of < in highlightTag reliably creates the correct closing tag. For HTML tags like <b>, replacing the first < with </ is fine, but it is safer and clearer to use string manipulation that does not risk accidentally leaving multiple or malformed tags. The best approach is to construct the closing tag explicitly by inserting a / after the initial <. This can be achieved by checking if highlightTag starts with < and simply generating the closing tag as '<\/' + highlightTag.slice(1). Alternatively, if you want to perform a global replacement for future-proofing, use replace(/</g, '</'). In our context, to minimize code change and avoid accidental misbehavior, construct the closing tag explicitly.

All changes are within the file apps/server/src/services/search/fts_search.ts, specifically line 256.

---

The most effective way to address this problem, without altering the external behavior or adding extra dependencies, is to replace the script removal regex in a loop: repeatedly remove all <script> tags until none remain. This ensures that, even if one replacement exposes a new script tag (due to overlapping or nested tags), all script blocks are completely removed. We should use a do...while loop that keeps replacing script tags as long as they are present. This change should be applied in the stripHtmlTags function, specifically on the line that currently executes the single-pass .replace() for script tags (line 75). No further imports or method changes are needed, and no logic elsewhere is affected.

---

The best fix is to replace the custom regular expressions meant to strip script and style content with a well-tested HTML parser/sanitizer. For Node.js/TypeScript, the sanitize-html or cheerio package are both popular and robust. Since only shown one file (apps/server/src/migrations/0235__sqlite_native_search.ts), and changes are allowed only here, we'll use sanitize-html, configured to remove all script and style content and strip all tags, yielding only text content.

Modification steps:

1. Import sanitize-html at the top of the file.
2. Replace lines 74-92 in stripHtmlTags with a call to sanitize-html configured to remove all tags and output plain text. This handles script/style reliably.
3. The call to stripTags from utils will be redundant and can be removed, since sanitize-html replaces its functionality with better coverage.
4. HTML entity decoding ( , etc.) can still be applied after sanitization if absolutely needed, but sanitize-html already decodes entities unless configured otherwise.

Only lines within the shown file will be touched.

---

The best fix is to ensure that all occurrences of <style>...</style> blocks are fully stripped, even if multiple or nested/malformed instances occur (as attackers might trigger incomplete removal). To do this without changing existing functionality and with minimal impact, repeatedly apply the regex replacements for script/style blocks until the string no longer changes. This ensures that any re-exposed tags are also removed. The change should only affect the two regex replace calls for <script> and <style> in stripHtmlTags() in apps/server/src/migrations/0235__sqlite_native_search.ts. No extra dependencies are needed, only a small refactor of these lines into a loop.

---

To fix the double-unescaping bug, the replacements for HTML entities must be performed in the correct order: all replacements except & should occur first, and replace & last. In this code, that means re-ordering lines 83–87 so that .replace(/&/g, '&') is last, after the other entity decoding steps.

Make this change in the stripHtmlTags function in apps/server/src/migrations/0235__sqlite_native_search.ts:

• Move text = text.replace(/&/g, '&'); to after the replacements for " and '.
  No imports or definitions are required; only a change in order of these replacement lines. No additional libraries are necessary.

---

The best way to fix this is to ensure repeated removal is performed until all script/style tags are fully stripped, regardless of how input mutates during replacement. This can be achieved by using a loop to apply the .replace() until there are no more replacements. Alternatively, use a robust sanitization library such as sanitize-html, but since we're limited to shown code and not introducing major codebase changes, repeated replacement is the most direct and non-breaking fix. Concretely, modify the lines where <script> and <style> tags are removed to repeatedly apply the regex replacements until a fixpoint is reached (nothing more is removed). This must all be done within apps/server/src/services/search/sqlite_functions.ts inside the stripHtml method.

No new imports are needed. Only the removal logic for script/style content must change.

---

To fix this issue, reorder the entity unescaping code so that the ampersand entity (&) is decoded last, after all other decodings. This avoids the double unescape problem: if " is present, it remains " while other entities are decoded, until the final pass where & is decoded to &.

Specifically, in apps/server/src/services/search/sqlite_functions.ts, lines 418–424 contain a sequence of .replace() calls for decoding HTML entities. Move the line text = text.replace(/&/g, '&'); (currently line 420) so that it comes immediately after all other .replace() calls for <, >, ", ', ', and  . The order should be: unescape all non-ampersand entities first, then unescape ampersand.

No additional imports or logic are needed.

---

The best fix is to repeatedly apply the regular expressions to remove <script> and <style> tags until no further matches are found, ensuring that malicious input cannot survive through incomplete multi-character replacements. This will guarantee that all possible instances, including those emerging after each replacement, are eliminated. The fix should be limited to the processHtmlContent function in this file. No code outside that function will be edited. No new dependencies are strictly required for this fix, but if the project would prefer to use a well-tested HTML sanitization library (like sanitize-html), that would be even better; however, per instructions, I will stick to the repeated replacement method inside the code region provided.

---

The best way to fix this is to avoid using regex for HTML sanitization and instead use a well-known HTML parser or sanitization library. If you must use regex for a very specific use-case, at least make your regex robust against browser quirks. In this context, since you're already using a utility function called stripTags, try to utilize a well-established library such as sanitize-html or dompurify via Node, but per the instruction only use code you've been shown. Therefore, update the regex on line 119 to more robustly match closing <script> tags with optional whitespace and attributes (so it matches tags like </script > or </script foo="bar">). A good improvement is to match </script\b[^>]*> (case-insensitive).

Edit line 119 in apps/server/src/services/search/sqlite_search_utils.ts:

• Replace the regex <script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script> with <script\b[^<]*(?:(?!<\/script[\s>])<[^<]*)*<\/script[\s\S]*?>, which is more robust against browser quirks and will match closing </script>, </script >, and </script foo="bar">.
• Similarly, update the <style> removal regex on line 120.

No new imports or method definitions are required, unless you decide to use an external sanitization library (which would require a dependency), but we restrict ourselves to just improving the regex as per instructions.

General Fix:
Ensure that all <style> tags and their contents are completely removed, even if new matches arise after each replacement. One straightforward solution is to reapply the replacement repeatedly until no further changes occur.

Best Specific Fix:
Modify the code so that the .replace(<style...>) (and likewise the <script...> removal) are applied inside a loop that continues until the string no longer changes. This can be implemented with a do...while loop for each pattern.

Region to Change:
Lines 119–120 in processHtmlContent, inside apps/server/src/services/search/sqlite_search_utils.ts.

Required Methods/Imports:
No new methods or imports are needed; only local logic changes.

---

The underlying problem is the ordering of HTML entity replacements in the "unescape" (decode) section of the processHtmlContent function. To fix this, reorder the string replacement statements so that text.replace(/&/g, '&') is performed after all other entity replacements. This ensures that entities like " are converted first to " (in their intended single-escape form), and only then to (") when " is replaced. Only modify the order of these lines: lines 126 to 132, so that & is last.

No new imports or methods are needed, as the code is doing simple string replacements.

---