Scripting 2.0: Lua Script Engine

[ProfLander]

These are the fundamental engine changes extracted from #240, minus the new features / extensions to existing ones / sweeping refactor of existing gamedata structure, and plus some additional refinement / compatibility work / cleanup.

The intent is to be its 'least controversial' subset, minimizing impact on existing gamedata, while reimplementing the script engine in Lua to reduce footprint, ease maintainability, and open the door to smoother and more extensive modding in the future.

Summary of Changes

• Moved the bulk of the following classes into Lua:

  • CScriptEngine
    • Previously handled:
      • Lua environment setup
      • Script autoloading
      • Script name resolution
      • Class registration
      • Common script execution
      • X-Ray Lua specializations
    • Now acts as a relatively thin wrapper around a standard Lua environment
      • Relevant functions have been split into Lua modules in gamedata
      • Engine-exposed APIs have been simplified and reimplemented in terms of Lua machinery
  • CScriptStorage
    • Previously a superclass of CScriptEngine
      • Used solely to categorize code; inheritance not used beyond this
      • Relevant remaining code moved into CScriptEngine
      • Reused name to implement a simple namespaced string map
        • Intended for data that has to persist across script engine instances
        • Currently used to cache I/O misses and reduce overhead of script autoload FS checks
  • CScriptProcess / CScriptThread
    • Mostly-unused coroutine scheduling system
    • Runs scripts as Lua coroutines ('threads') via a one-per-frame timeslicing system
    • Able to load scripts from game mode and level, but neither are populated in gamedata
    • Used primarily to dispatch console's run_script / run_string commands while in-game
    • Reimplemented on the Lua side
      • Uses Lua-side coroutine machinery
        • Cleaner implementation
        • Scheduling machinery now accessible to the Lua environment
      • Now tied to lifecycle of the script engine / Lua VM
        • Constructed automatically on startup
        • Updates driven by the same engine call sites
        • Automatic cleanup means long-running coroutines no longer crash out on save reload

• Stripped out legacy Lua Studio debugger integration

  • Unlikely that anyone uses Lua Studio to mod Anomaly or its engine in practice
    • Premium Lua-specific IDE that retails for $30 from a Web 1.0 site
    • Proprietary(?) GSC-developed Lua IDE
  • LuaDkmDebugger provides a free VS-native replacement that works out of the box with Anomaly
    • https://github.com/WheretIB/LuaDkmDebugger
    • Can be attached to an existing build given appropriate debug symbols
  • Removal of large inactive #ifdef blocks improves maintainability of related code
  • Significant reduction in supporting code footprint

• Generalized script environment boot process

  • CScriptEngine:
    • Performs minimal setup
    • Hands off to init.lua
  • init.lua:
    • Sets up a working print function
    • Establishes minimal working require and error-guarded loadstring
    • Hands off to boot module
  • boot module:
    • Disables dangerous Lua primitives
    • Configures extensible package.path machinery
    • Integrates X-Ray FS with package.loaders
    • Loads scripts specified in script.ltx
      • Lua port of the original CScriptEngine common scripts runner
  • script.ltx initializes:
    • xr/compiler
      • X-Ray script compiler
    • xr/lua
      • Baseline X-Ray Lua
    • amx
      • Modded Exes Lua extensions
    • xr/classes
      • Class registrator machinery
    • _G
      • Original Lua entrypoint
    • Modded Exes _G patches
    • DXML

• xr/compiler

  • Generalized script source loading
  • Overrides loadstring with per-script dispatch
  • Directed by file extensions, with header-comment syntax for in-script overrides
  • Uses basic .lua loader established in init.lua by default

• xr/lua

  • Extends xr/compiler with generalized X-Ray Lua specializations
    • Tied to .script extension
    • Original package environment behaviour
    • Auto-load scripts when their corresponding global is accessed
    • Expose loaded scripts directly in _G
  • Now a wrapper around standard Lua machinery
    • Automatic require + metatable indirection against package.loaded
    • Encapsulation allows for spec-compliant Lua 5.1 as base environment
      • Nestable directories in the scripts folder
      • Import / Export semantics via local, require, and return
        • i.e. Proper Modules
      • Fully Lua-native coroutine management
        • Allows implementation of abandoned features
          • e.g. wait and wait_game from vanilla _g.script
      • Unrestricted access to global namespace
      • Currently only exposed to core-code .lua scripts
        • Not currently unlocalizable
          • Different global semantic means existing impl would put all locals in _G
          • Achievable with some effort
          • Needs an alternate approach where each localis passed to the parent env
            (i.e. the init.lua passthrough compiler) for selective re-export
        • Existing .script scripts are sandboxed

• amx

  • Extends xr/lua with unlocalization support
  • Provides an entrypoint for future modded exes extensions

• xr/classes

  • Lua port of the original CScriptEngine class registration functionality

• Exposed recursion-friendly filesystem listings to Lua

  • Opens the door to integrating with DXML, axr_main, MCM, etc.

• New eval console command

  • Wraps run_string with the new working print primitive
  • Allows quick evaluation of script expressions
  • Variadic, as per print
  • Opens the door to Read-Eval-Print-Loop functionality for the console
    • Faster iteration and debugging for script developers

Compatibility

These changes are intended to be seamless for existing mods, and have been tested against Vanilla, my own 3D Ballistics debug modlist, and GAMMA.

Thus far, Vanilla and the 3D Ballistics list work as expected.

GAMMA exhibits ~4 load-time errors for existing mods, so further refinement of xr/lua is necessary before this can be considered complete.

[ProfLander]

There we go - let me know if this is still beyond scope for modded exes.

(Also, I note that the commit log mentions Lisp - that's a leftover from an early C implementation of Fennel, which is factored out in favor of strictly vanilla Lua-side compilation machinery by later commits.)

[themrdemonized]

Alright, let me know when you iron out all bugs

[ProfLander]

After some direct testing with GAMMA - which went fine, didn't realize the log errors were part of base 😅 - and rooting out a couple of unlocalization issues with assistance from plyaka.egor on Discord, this branch is now ready for public testing.

Equivalence Proofs

Gist - GAMMA Load Comparisons

Test Build

Mediafire - Anomaly Scripting 2.0 RC7

Mediafire - RC7 PDB Files

Testing Guidelines

Due to the necessity of testing against a variety of modlists, and the general complexity of X-Ray, avoiding false positives is critical.

As such, testers are requested to adhere to the following guidelines:

• Keep frequent saves for the sake of easy reproduction

• When encountering a crash or other bug with Scripting 2.0, save a copy of the resulting log

  • This is useful for pinpointing a potential incompatibility, but does not fully confirm one

• If possible, rewind to the previous save and try to reproduce the issue both with and without Scripting 2.0

  • Naturally, this only applies to in-game issues, as load-time crashes have no prior state
  • Perform the same actions in each, and save copies of both logs
    • This will make it easier to determine whether a given issue is an incompatibility, or outlier / false positive
    • There's room for error here, given the non-deterministic nature of Stalker gameplay, but do your best to repro if possible
  • If a given issue cannot be reproduced on a non-2.0 install, it's likely a Scripting 2.0 incompatibility that will need to be fixed
  • Ideally, a copy of the save prior to the issue would be provided, along with steps to reproduce the issue

• Report results on GitHub if possible, or in the addon-releases/[WIP] Anomaly Scripting 2.0 channel on the Anomaly Discord

[ProfLander]

Unconfirmed Issues

...

Known Issues

GAMMA

...

Individual Mods

...

Fixed Issues

GAMMA

Right click > Details in inventory results in an empty popup

• Crafting Uses populates as other items are inspected

calling override ui_itm_details.UIItemSheet.Reset
! xr/lua: error evaluating script:
 
[string "ui_itm_details"]:116: attempt to index local 'self' (a nil value)
 
stack traceback:
	[C]: in function 'xpcall'
	[string "xr/lua"]:110: in function 'ResetSuper'
	[string "zz_ui_itm_details_repair_bonuses"]:45: in function 'base_details_reset'
	[string "zzz_craft_use_in_tooltip_mcm"]:449: in function 'Reset'
	[string "ui_itm_details"]:50: in function 'start'
	[string "ui_itm_details"]:30: in function <[string "ui_itm_details"]:22>
	[string "ui_inventory"]:1291: in function 'ActionCustom'
	[string "ammo_maker"]:171: in function 'ActionCustom'
	[string "campfire_placeable"]:294: in function 'ActionCustom'
	[string "item_exo_device"]:416: in function 'ActionCustom'
	[string "wait"]:66: in function <[string "wait"]:59>
	[string "utils_ui"]:3367: in function <[string "utils_ui"]:3357>
Super function ui_itm_details.UIItemSheet.Reset called
* Register UI: UIItemSheet

• Was down to use of loadstring with the binary output of debug.dump; compiler preprocessing destroys the binary function handle, so needed xr/lua to pass raw loadstring through and wrap it with the modified environment after the fact

Stack overflow crash in copy_table when selected Rain Enhancers in SSS MCM

• Likely trying to copy env and getting stuck in the infinite self-reference

• Caused by stateful injected environment variables being enumerated by copy_table; now sanitized away behind metatable indexing

Individual Mods

Weird Task Framework 4.2

Crash on load due to debug.info usage

• xr/lua modules need to use @ .. script_name instead of namespace_name to preserve original debug.info src semantic

Dynamic Faction Relation Customizer

Crash upon killing an enemy

• Likely an unlocalizer edge-case

! [LUA]  0 : [C  ] __len
! [LUA]  1 : [Lua] ...data\scripts\dynamic_faction_relations_customizer.script(1003) : mod_check_for_reputation_change
! [LUA]  2 : [Lua] ...data\scripts\dynamic_faction_relations_customizer.script(3004) : calculate_relation_change
! [LUA]  3 : [Lua] ...data\scripts\dynamic_faction_relations_customizer.script(4674) : evaluate_npc_on_death
! [LUA]  4 : [Lua] ...data\scripts\dynamic_faction_relations_customizer.script(4721) : online_npc_on_observed_death
! [LUA]  5 : [Lua] ...bin/..\gamedata\scripts\dynamic_news_manager_dfrc.script(687) : 
! [LUA]  6 : [Lua] e:/stalker-anomaly/bin/..\gamedata\scripts\axr_main.script(278) : make_callback
! [LUA]  7 : [Lua] e:/stalker-anomaly/bin/..\gamedata\scripts\_g.script(119) : SendScriptCallback
! [LUA]  8 : [Lua] ...lker-anomaly/bin/..\gamedata\scripts\xr_motivator.script(396) : 
! [LUA] SCRIPT RUNTIME ERROR
! [LUA] ...data\scripts\dynamic_faction_relations_customizer.script:1003: attempt to get length of local 'rept_table' (a nil value)
! [SCRIPT ERROR]: ...data\scripts\dynamic_faction_relations_customizer.script:1003: attempt to get length of local 'rept_table' (a nil value)
! [LUA] SCRIPT RUNTIME ERROR
! [LUA] ...data\scripts\dynamic_faction_relations_customizer.script:1003: attempt to get length of local 'rept_table' (a nil value)
! [SCRIPT ERROR]: ...data\scripts\dynamic_faction_relations_customizer.script:1003: attempt to get length of local 'rept_table' (a nil value)

• Caused by broken unlocalization of local a, b cases

Banjajj's Undercover Communications

gameplay_disguise crash

• Unable to reproduce locally, further correspondence ongoing
• Fixed following recent unlocalizer changes

Useful Idiots

Error loading illish.*.lua scripts

• No crash reported, but log contains early errors
• Caused by use of / instead of idiomatic Lua . in module paths

Error accessing ui_mcm from inside illish.*.lua

• Unable to access _G members from importing environment
• Fixed by exposing .lua environment as a customizable global, maintaining inside amx/lua

Failure to index local loadouts in demonized_molotov_new_game_loadouts_mcm.script and drx_da_main_loadout_mcm

• Appears fixed as of RC7

[Xottab-DUTY]

I'd like to highlight one small detail:

Unlikely that anyone uses Lua Studio to mod Anomaly or its engine in practice

• Premium Lua-specific IDE that retails for $30 from a Web 1.0 site

That paid Lua Studio and X-Ray's Lua Studio are two different products.
IDE for Lua was developed by Dmitriy Iassenev (he said that in one of the interviews, IIRC) during his work on X-Ray 2.0 and he called it Lua Studio.
Separately from that, in a different part of the world, a company developed an IDE for Lua a they called it Lua Studio too.

[ProfLander]

@Xottab-DUTY Interesting, thanks for the pointer; I've updated the PR text appropriately.
Out of interest, do you happen to know if it was ever made public via SDK, source release, etc?

On another note - do you think these changes would be of interest to OXR once finalized?

I know it's early days yet, but figure it would be good to try and standardize around the parts that apply to non-Anomaly X-Ray if possible, since that would allow the resulting benefits to go beyond Anomaly's little corner of the ecosystem, and avoid further divergence from the more formal / active engine forks.

Beyond the naturally-tricky differences in C++, I've made sure to separate Anomaly's concerns where I can - xr/lua should be compliant with the semantics of old-school X-Ray Lua, assuming Anomaly's implementation of the dialect didn't change in course of its development.

The one exception at present being the handling of the game script process (previously discharged by CScriptProcess / CScriptThread) - presently that's hard-coded on the Lua side to run scripts from the single section of script.ltx at startup rather than using the active game mode, since Anomaly is SP-oriented and doesn't expose the corresponding string to Lua yet as far as I'm aware. Shouldn't be difficult to generalize, but is worth noting.

[Xottab-DUTY]

@Xottab-DUTY Interesting, thanks for the pointer; I've updated the PR text appropriately. Out of interest, do you happen to know if it was ever made public via SDK, source release, etc?

On another note - do you think these changes would be of interest to OXR once finalized?

I know it's early days yet, but figure it would be good to try and standardize around the parts that apply to non-Anomaly X-Ray if possible, since that would allow the resulting benefits to go beyond Anomaly's little corner of the ecosystem, and avoid further divergence from the more formal / active engine forks.

Beyond the naturally-tricky differences in C++, I've made sure to separate Anomaly's concerns where I can - xr/lua should be compliant with the semantics of old-school X-Ray Lua, assuming Anomaly's implementation of the dialect didn't change in course of its development.

The one exception at present being the handling of the game script process (previously discharged by CScriptProcess / CScriptThread) - presently that's hard-coded on the Lua side to run scripts from the single section of script.ltx at startup rather than using the active game mode, since Anomaly is SP-oriented and doesn't expose the corresponding string to Lua yet as far as I'm aware. Shouldn't be difficult to generalize, but is worth noting.

You can find part of the code related to Lua Studio in leaked X-Ray 2.0 code. But there's a backend part, which wasn't leaked. We should ask Dmitriy Iassenev about it, I guess :))
(at least for some encyclopedic knowledge)

Yes, we're interested in standardization!
There's few things to notice:
Our script engine is already refactored (changes aren't big like in this PR though, but it's still a bit of divergence).
We aim to be as compatible as possible with vanilla games (COP, CS, SOC), their mods and Call of Chernobyl particularly.
We still support multiplayer.

[ProfLander]

Progress update: After roughly a week of me testing in GAMMA and other users trying it with their own modlists, no verifiable Scripting 2.0 tracebacks have appeared in logs. I think it's as stable as it's going to get in terms of day-to-day play.

However, there's a catch; I tested it with buttplug-anomaly (still the foremost example of doing wacky native library FFI in Anomaly scripts...) and found a corner-case where buttplug.script tries to require buttplug.lua from its own custom path, but fails due to package.loaded["buttplug"] already being occupied by the .script.

Which is to say, I need to give the sandbox its own non-package.loaded, non-_G place to store loaded modules so they don't conflict. Once that's up and running, it should be ready to merge.

You can find part of the code related to Lua Studio in leaked X-Ray 2.0 code. But there's a backend part, which wasn't leaked. We should ask Dmitriy Iassenev about it, I guess :))
(at least for some encyclopedic knowledge)

Ah, well that's a relief in some sense; I didn't want to rip out code we could potentially use, but if there are missing pieces then the cleanup is for the best.

Yes, we're interested in standardization!
There's few things to notice:
Our script engine is already refactored (changes aren't big like in this PR though, but it's still a bit of divergence).
We aim to be as compatible as possible with vanilla games (COP, CS, SOC), their mods and Call of Chernobyl particularly.
We still support multiplayer.

Cool! A PR is probably a ways off, since I have a teetering tower of Anomaly woes to address in the meantime, but I'll endeavour to keep working with portability in mind. Any syntax / semantic breaks should be capturable in a prospective oxr/lua module, and fixing the level script entrypoint should go some way to supporting multiplayer. Though naturally, much testing would be needed - ample excuse for a fresh CoP playthrough!

On a related note, I've been eyeing up OXR's submodules for Anomaly integration following #264 (luabind and LuaJIT chief among them) and noticed b5ef6ba and 04084ec directly extending the lexer with new syntax. Such low-level language extensions - while not a problem, since we're too far behind to leverage most of the LJ improvements anyway - are a bit scary to bring over for now, since I've been pushing for a pure vanilla Lua baseline.

But, they're worth highlighting as an example of where this changeset could help, since one of its big motivators is being able to manipulate Lua's syntax and semantics (ex. to encode X-Ray's own hardcoded lang extensions) without needing to change any C/++. Fairly limited for now since Lua lacks a macro system, but I have my eye on MetaLua for achieving more elaborate transformations (ex. a macro-hygenic impl of our script unlocalizer) later down the line.

[LVutner]

peak modding

[Xottab-DUTY]

On a related note, I've been eyeing up OXR's submodules for Anomaly integration following #264 (luabind and LuaJIT chief among them) and noticed b5ef6ba and 04084ec directly extending the lexer with new syntax. Such low-level language extensions - while not a problem, since we're too far behind to leverage most of the LJ improvements anyway - are a bit scary to bring over for now, since I've been pushing for a pure vanilla Lua baseline.

Anomaly engine is using LuaJIT 2.0.4 since it was added in the Call of Chernobyl engine.
OpenXRay is based on LuaJIT 2.1 and it's main advantage over 2.0 version is it being fully 64-bit.
(note that it has 2.1 version for many years because of the switch to the rolling releases system. Our "2.1" is from 2021 and "2.1" from 2025 can have more breaking changes)
Our additions to the syntax are small, but I agree that updating LuaJIT may be potentially dangerous for compatibility with already written scripts.
But what's more problematic is our luabind – it is confirmed to break compatibility with scripts written for original luabind (used in all engines, including this repository) because it just behaves differently. So... Just don't use it.

But, they're worth highlighting as an example of where this changeset could help, since one of its big motivators is being able to manipulate Lua's syntax and semantics (ex. to encode X-Ray's own hardcoded lang extensions) without needing to change any C/++. Fairly limited for now since Lua lacks a macro system, but I have my eye on MetaLua for achieving more elaborate transformations (ex. a macro-hygenic impl of our script unlocalizer) later down the line.

Oh, introducing new language features by patching LuaJIT directly is bad because it adds additional burden of supporting new code and merging with the upstream LuaJIT. And what's more important it breaks compatibility with other engines that don't have these patches applied.
MetaLua, particularly, doesn't look bad because it is written in Lua itself :)

Also, did you considered Luau? Though, it's problematic, because some features that are actually used in Anomaly and addons are just removed from Luau for security reasons.