Skip to content

Feature/unicode superscripts #1674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sangwinc merged 3 commits into from
Feb 9, 2026
+42 −14
Merged

Feature/unicode superscripts #1674

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
12dbc46
Enable Unicode superscripts ², ³, ¹ input
Feb 6, 2026
b78ef11
Enable Unicode superscripts ², ³, ¹ input
Feb 6, 2026
8684a6f
Update tests to expect valid superscript input
Feb 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions stack/cas/parsingrules/180_char_based_superscripts.filter.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,18 @@ class stack_ast_filter_180_char_based_superscripts implements stack_cas_astfilte
// phpcs:ignore moodle.Commenting.MissingDocblock.Function
public function filter(MP_Node $ast, array &$errors, array &$answernotes, stack_cas_security $identifierrules): MP_Node {
if (self::$ssmap === null) {
self::$ssmap = json_decode(file_get_contents(__DIR__ . '/../../maximaparser/unicode/superscript-stack.json'), true);
// Option C: Allow only the most common superscripts: ², ³, ¹
// These are 2-byte UTF-8 characters and work without UTF-8mb4 requirement.
self::$ssmap = ['²' => '2', '³' => '3', '¹' => '1'];
}

$process = function($node) use (&$errors, &$answernotes) {
// Loop until no more changes are made.
// This ensures all superscripts are converted, even in expressions like x²+x².
$changed = true;
while ($changed) {
$changed = false;

$process = function($node) use (&$errors, &$answernotes, &$changed) {
if ($node instanceof MP_Identifier && !(isset($node->position['invalid']) && $node->position['invalid'])) {
// Iterate over the name to detect when we move from normal to superscript.
$norm = true;
Expand Down Expand Up @@ -109,6 +117,7 @@ public function filter(MP_Node $ast, array &$errors, array &$answernotes, stack_

if (count($parts) === 1) {
$node->parentnode->replace($node, $parts[0]);
$changed = true;
} else {
if (array_search('missing_stars', $answernotes) === false) {
$answernotes[] = 'missing_stars';
Expand All @@ -120,14 +129,16 @@ public function filter(MP_Node $ast, array &$errors, array &$answernotes, stack_
$a->position['insertstars'] = true;
}
$node->parentnode->replace($node, $a);
$changed = true;
}
return false;
}
}
return true;
};

$ast->callbackRecurse($process);
$ast->callbackRecurse($process);
}
return $ast;
}
}
4 changes: 4 additions & 0 deletions stack/input/inputbase.class.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -975,6 +975,10 @@ protected function validate_contents_filters($basesecurity) {
$filterstoapply[] = '101_no_floats';
}

// Filter 180 MUST run BEFORE Filter 150!
Copy link
Member

@aharjula aharjula Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order of filters is based purely on the numbers at the start of the filter name. 180 will never get executed before 150, so if one wants to do something before 150, that something needs to be named accordingly.

The array filterstoapply is not the execution order for them. It is just the set of filters that need to be included in the filter pipeline; the pipeline construction logic then sorts that set and initialises the filters with any options they might have, as well as checks any issues related to incompatible filters.

Interesting, though, that the filter existed but was not used in the actual logic. Only in the input fixtures tests? Guess someone got busy with something else, and this was forgotten?

Copy link
Member

@sangwinc sangwinc Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, though, that the filter existed but was not used in the actual logic. Only in the input fixtures tests? Guess someone got busy with something else, and this was forgotten?

Indeed @aharjula that's certainly probably me! Sorry, we can look into this.

// Filter 180: Intelligently converts Unicode superscripts to exponents (x² → x^2)
// Filter 150: Simple replacement of remaining Unicode letters (α → alpha, × → *)
$filterstoapply[] = '180_char_based_superscripts';
$filterstoapply[] = '150_replace_unicode_letters';

if (get_class($this) === 'stack_units_input' || get_class($this) === 'stack_numerical_input') {
Expand Down
5 changes: 5 additions & 0 deletions stack/maximaparser/corrective_parser.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,12 @@ public static function parse(string $string, array &$errors, array &$answernote,

// Check for invalid chars at this point as they may prove to be difficult to
// handle latter, also strings are safe already.
// Option C: Allow only the most common superscripts: ², ³, ¹
// These are 2-byte UTF-8 characters and work without UTF-8mb4 requirement.
$superscript = ['²' => '2', '³' => '3', '¹' => '1'];
Copy link
Member

@sangwinc sangwinc Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@christianp and I started a collection of patterns like this here.

https://github.com/sangwinc/unicode-math-normalization

We should link into this, rather than hard-wiring lists into the code.


$allowedcharsregex = '~[^' . preg_quote(
implode('', array_keys($superscript)) .
// @codingStandardsIgnoreStart
// We do really want a backtick here.
'0123456789,./\%#&{}[]()$@!"\'?`^~*_+qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM:;=><|: -', '~'
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/inputfixtures.class.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -520,8 +520,8 @@ function at a point \(f(x)\). Maybe a 'gocha' for the question author....",
['(x/y)/z', 'php_true', '(x/y)/z', 'cas_true', '\frac{\frac{x}{y}}{z}', '', ""],
['x/(y/z)', 'php_true', 'x/(y/z)', 'cas_true', '\frac{x}{\frac{y}{z}}', '', ""],
['x^y', 'php_true', 'x^y', 'cas_true', 'x^{y}', '', "Operations and functions with special TeX"],
["x\u{00b2}", 'php_false', '', 'cas_false', '', 'forbiddenChar', ""],
["x\u{00b2}*x\u{00b2}", 'php_false', '', 'cas_false', '', 'forbiddenChar', ""],
["x\u{00b2}", 'php_true', 'x^2', 'cas_true', 'x^{2}', 'superscriptchars', ""],
["x\u{00b2}*x\u{00b2}", 'php_true', 'x^2*x^2', 'cas_true', 'x^{2}\cdot x^{2}', 'superscriptchars', ""],
['x^(y+z)', 'php_true', 'x^(y+z)', 'cas_true', 'x^{y+z}', '', ""],
['x^(y/z)', 'php_true', 'x^(y/z)', 'cas_true', 'x^{\frac{y}{z}}', '', ""],
['x^f(x)', 'php_true', 'x^f(x)', 'cas_true', 'x^{f\left(x\right)}', '', ""],
Expand Down Expand Up @@ -625,7 +625,7 @@ function at a point \(f(x)\). Maybe a 'gocha' for the question author....",
['arsinh(x)', 'php_true', 'asinh(x)', 'cas_true', '{\rm sinh}^{-1}\left( x \right)', 'triginv', ""],
['sin^-1(x)', 'php_false', 'sin^-1(x)', 'cas_false', '', 'missing_stars | trigexp', ""],
['cos^2(x)', 'php_false', 'cos^2(x)', 'cas_false', '', 'missing_stars | trigexp', ""],
["sin\u{00b2}(x)", 'php_false', 'sin^2(x)', 'cas_false', '', 'forbiddenChar', ""],
["sin\u{00b2}(x)", 'php_false', 'sin^2(x)', 'cas_false', '', 'trigexp | superscriptchars | forbiddenVariable', ""],
['sin*2*x', 'php_false', 'sin*2*x', 'cas_false', '', 'forbiddenVariable', ""],
['sin[2*x]', 'php_false', 'sin[2*x]', 'cas_false', '', 'trigparens', ""],
['cosh(x)', 'php_true', 'cosh(x)', 'cas_true', '\cosh \left( x \right)', '', ""],
Expand Down
9 changes: 9 additions & 0 deletions tests/fixtures/test_strings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
[
"x²",
"x³",
"x¹",
"x²²",
"x²³",
"x²+x²",
"x²+x²+x²",
"x²y³",
"(x²)+(y²)",
"\"+\"(a,b)",
"\"1+1\"",
"\"Hello world\"",
Expand Down
15 changes: 7 additions & 8 deletions tests/input_algebraic_test.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2492,14 +2492,13 @@ public function test_validate_student_response_single_var_chars_unicode_superscr
$el->set_parameter('insertStars', 2);
$state = $el->validate_student_response(['sans1' => 'x²'], $options, 'x^2',
new stack_cas_security());
$this->assertEquals(stack_input::INVALID, $state->status);
// The rest needs to be updated once we know what the expected result is.
$this->assertEquals('forbiddenChar', $state->note);
$this->assertEquals('CAS commands may not contain the following characters: ².',
$state->errors);
$this->assertEquals('', $state->contentsmodified);
$this->assertEquals('<span class="stacksyntaxexample">x&sup2;</span>', $state->contentsdisplayed);
$this->assertEquals('', $state->lvars);
// Unicode superscripts are now converted to exponents by Filter 180.
$this->assertEquals(stack_input::VALID, $state->status);
$this->assertEquals('superscriptchars', $state->note);
$this->assertEquals('', $state->errors);
$this->assertEquals('x^2', $state->contentsmodified);
$this->assertEquals('\[ x^{2} \]', $state->contentsdisplayed);
$this->assertEquals('x', $state->lvars);
}

public function test_validate_student_response_km(): void {
Expand Down
Loading