Support openmw file types

[magicaldave]

Partial resolution for MO2 #1609

Regarding points brought up in this issue:

While the vanilla engine DOES break when feeding it omwaddons, this is primarily caused by the naming convention used. Many omwaddons work perfectly fine in Morrowind.exe when renamed to ESP. Additionally, this is a very common practice on the part of users. TESCS and OpenMW-CS do not create plugins which are distinctive enough for MO2 to distinguish between the two using anything beyond the name. Of course, we know that the file extension is not at all a reliable means to distinguish between the two. Realistically, there is not one, because omwaddons are intended to be nearly the exact same. Additionally, there are proven cases of mods being created as ESP using TESCS and then renamed to omwaddon as well. Some mods have been created as omwaddon and published under the original format.

Also, these:
MOMW Patches
Oh No Stolen Reports
The simple fact of the matter is that MO2 can't take responsibility for this, and the user should understand that the mod they are using is not supported in vanilla. This is universally explained by the author in some capacity or another.

Additional references:
Ashlander Architect
Morrowland Uncapped Attributes

Due to OpenMW goals, supported file types are added in the global context and not the per-game one. Additionally, omwgame files are explicitly not supported by design as there simply are none and would be an entirely different game with its own instance and Nexus category. Since none exist, there's no way to begin an implementation of this. Omwgame should be supported at such a time as there is an omwgame to support.

Activatable BSA files are not a real feature anybody in the Morrowind community needs or wants. While, historically, the main use case for TES3 BSA files has been the Tamriel Rebuilt project, they no longer ship with BSAs. Creators tend not to use them because it complicates the process of creating mods, they don't support all file types used by TES3, MO2 does not support them, and people constantly mess up registering them.

Additionally, since OpenMW supports later titles (sort of) and their associated iteration of BSA files, this will cause additional compatibility problems with every game. You could feed New Vegas or Morrowind an SSE-era compressed BSA file and it will naturally not work. There's no real reason to put any effort into this, as it is just going to complicate the codebase for a feature literally nobody is asking for at this point in time.

This PR does not touch groundcover plugins. I do not have a thorough proposal for this at this time. However, I think it is best to avoid cluttering the main plugin list with additional fields that are morrowind-specific, so perhaps a right-click option Mark as Groundcover or something to that effect can be added which exporters hook into. I don't think there's really a nice way to handle that for both engines unless a check specifically for MGEXEgui.exe is added.

Edited for clarity & additional sources

[AnyOldName3]

I have mixed feelings about the general approach taken here, hence not doing this myself years ago. The most obvious is that there are a lot of good reasons for more of ESP management to be moved to plugins, like ESL only being a thing for the later games, and plenty of games supported via the basic_games plugins having something that kind of maps onto ESP files and which could be managed more directly.

While the vanilla engine DOES break when feeding it omwaddons, this is primarily caused by the naming convention used. Many omwaddons work perfectly fine in Morrowind.exe when renamed to ESP.

So far, but

omwaddons are intended to be nearly the exact same.

isn't exactly true. .omwaddons are similar at the moment, but the intention's always been to expand on the original format, so ones that kill the original engine will come into existence. .omwscripts is obviously not something the original engine can handle.

The simple fact of the matter is that MO2 can't take responsibility for this, and the user should understand that the mod they are using is not supported in vanilla.

That's not something that can be taken as a fact. As a mod manager, this is the kind of thing MO2 should deal with if it's at all feasible. OpenMW takes a hardline stance on things where a mod manager shouldn't, and it's much more reasonable for a mod manager to be full of helpful heuristic-based warnings that aren't necessarily always right, but give an idea of what should be investigated when things get weird, or save several clicks on average, as long as there's an option to ignore it when you know better.

As a bodge, it should be feasible to add an onAboutToRun callback to game_gamebryo to check the load order for OpenMW-specific content formats based on file extension, and if there are any, display a popup offering to disable them before continuing or to continue anyway, possibly with a checkbox to remember the answer. It just using file extensions is fine as mod managers live in a grey area where there's no right answer that always works, and they're a pretty good signal. They'll only get better as fewer people rename things to .esp to force them through MO2 once it knows about .omwaddon & friends.

Due to OpenMW goals, supported file types are added in the global context and not the per-game one.

OpenMW isn't aiming to support mods for Cyberpunk 2077, which is a game that people manage through MO2. Generally, we want to be migrating as much stuff as possible to be per-game. All the BGS games extend a common base class, so ideally, all of this would be handled in game_gambryo as that's the thing that maps onto OpenMW's goals.

BSA stuff

This PR doesn't do anything with BSAs and doesn't refer to any of the issues about BSAs, so this isn't relevant. #1609 is just about the OpenMW-specific content files.

This PR does not touch groundcover plugins. I do not have a thorough proposal for this at this time. However, I think it is best to avoid cluttering the main plugin list with additional fields that are morrowind-specific, so perhaps a right-click option Mark as Groundcover or something to that effect can be added which exporters hook into. I don't think there's really a nice way to handle that for both engines unless a check specifically for MGEXEgui.exe is added.

This is discussed in #1692. MGE XE keeps groundcover files in a separate INI, so we just need to make that INI one of the ones MO2 deals with and stick groundcover files in there

[magicaldave]

I really brought up the similarities between the two formats just to explain they're used interchangeably and the extension isn't necessarily reliable. The way I really should have phrased:

The simple fact of the matter is that MO2 can't take responsibility for this, and the user should understand that the mod they are using is not supported in vanilla.

Is maybe more like, "If the file extension specifically is not reliable then trying to distinguish between the two is so overly complex as to be a waste of time and not really work in any reliable way".

But it's definitely a more sane approach to expect that it be. We're already at a point where they can break due to either usage of Lua records.

I'm unfortunately not very well-equipped to work on MO2 at the moment, but I'll do what I can with it.

It seems like your first point is suggesting a larger refactor wherein the bethesda-specific code is removed from the main windows and migrated into game_gamebryo?

Actually from what I've seen of that repo I could probably close both of my PRs and simply apply the changes there, but there's at least three places that mention supporting plugins and two that mention supported directories, so it's a bit confusing as to what may work or not.

[AnyOldName3]

Is maybe more like, "If the file extension specifically is not reliable then trying to distinguish between the two is so overly complex as to be a waste of time and not really work in any reliable way".

As I said, mod managers always live in the grey area. Checks don't need to be perfect, just helpful on average, and possible to dismiss when they're potentially wrong. As heuristics go, the file extension is actually pretty good.

[MrGrymReaper]

Also since OpenMW 0.48 (https://openmw.org/2023/openmw-0-48-0-released/) there's an Lua scripting API, so this means there's now a definite requirement for it to have better support. This is due to its new scripting API and system not being supported in the Morrowind game when it is vanilla. Also there's been several several enhancements to shaders and/or lighting which isn't in the vanilla Morrowind.

These lighting enhancements were preceded by the new lighting system which was introduced for version 0.47 of OpenMW (https://openmw.org/2021/openmw-0-47-0-released/).

Anyway the OpenMW game is compiled as "openmw.exe" under Windows. Other people please chime in on how it is named on MacOS and Linux. The other file types can be handled with OpenMW having its own option in ModOrganizer thus when it encounters these above data files it can, be ready for them without too much worrying about the vanilla Morrowind as it won't be in the same section anymore.

[magicaldave]

The additions via the Lua API were a major driver in me originally creating this. That in itself really changes things, though.

Scripters are slowly getting into the habit of including arbitrary file types alongside their mods. I can point to cases which use YAML, JSON, and CSV files. This is just the beginning, though, as OpenMW allows modders byte-level access to any files located in the VFS at boot time. It could possibly be argued that the entire concept of valid file extensions is almost completely invalidated by this.
The file extension issue also extends in less pronounced form for loads of engine components - you can use Collada or OSGT models, any video type supported by ffmpeg, and also of course openmw's made-up filetypes such as omwaddon, omwscripts, and omwfx. Such mods are often likely to also include something that'd be more typically recognizable such as omwaddon/omwscripts, but it's not at all a requirement.

In *nixes openmw binary is build as openmw.x86_64 or openmw_launcher.x86_64.

Tend to agree the ideal approach is to treat openmw as its own game. That's not really a perfect solution, but, at such time as original openmw titles are coming out, they could also justifiably be given their own game option. I believe MO2's profiles option can also handle openmw's goals of supporting later Beth titles quite handily.

treating it as its own game would also (hopefully?) facilitate referring to the proper, useful, openmw.cfg, instead of the vestigial morrowind.ini OpenMW users can't do anything with.

[AnyOldName3]

Scripters are slowly getting into the habit of including arbitrary file types alongside their mods. I can point to cases which use YAML, JSON, and CSV files. This is just the beginning, though, as OpenMW allows modders byte-level access to any files located in the VFS at boot time. It could possibly be argued that the entire concept of valid file extensions is almost completely invalidated by this.
The file extension issue also extends in less pronounced form for loads of engine components - you can use Collada or OSGT models, any video type supported by ffmpeg, and also of course openmw's made-up filetypes such as omwaddon, omwscripts, and omwfx. Such mods are often likely to also include something that'd be more typically recognizable such as omwaddon/omwscripts, but it's not at all a requirement.

All this (or direct analogues) is true of mods for the later BGS games. It's why we've got lots of directories like Tools, SkyProc Patchers and NetScriptFramework included in the valid data checkers. It's not perfect, but MO2 is a mod manager, so everything's a heuristic anyway. Unlike the the stuff in this PR, this is something that is handled by the game-specific plugins instead of in the core of MO2, so a PR to add OpenMW-specific directories and file extensions to the lists here https://github.com/ModOrganizer2/modorganizer-game_gamebryo/blob/master/src/gamebryo/gamebryomoddatachecker.cpp#L10-L43 and here https://github.com/ModOrganizer2/modorganizer-game_gamebryo/blob/master/src/gamebryo/gamebryomoddatachecker.cpp#L52 would be helpful (as it'd get rid of the scary red warnings that mods don't look valid when in reality, they are) and wouldn't make any existing mess worse.

Tend to agree the ideal approach is to treat openmw as its own game. That's not really a perfect solution, but, at such time as original openmw titles are coming out, they could also justifiably be given their own game option. I believe MO2's profiles option can also handle openmw's goals of supporting later Beth titles quite handily.

I strongly disagree.

• We already have separate games that use the same engine (e.g. TTW uses the NV engine, Enderal uses the Skyrim engine etc.) so games and engines are already not a 1:1 mapping.
• It makes it a pain/impossible to share mods between the original engine and OpenMW, and wanting that (and an easy way to try identical mod lists in the two engines to compare behaviour) was the whole reason I resuscitated MO2's Python plugin API instead of the much easier job of adding GUI to manage data directories from OpenMW's launcher.
• This isn't something MO2's profile system can handle at all unless you copy your install for each base game into a mod directory (MO2 doesn't support arbitrary paths for individual mods like OpenMW does - they all have to share one parent directory). Even with that workaround, it'd be unpleasant, as all the mods for one instance are visible in all profiles, it's only whether or not they're enabled that's different. If you've got a few hundred mods for each game, it's not going to be good to have them all in one huge list together.
• The thing that can support this nicely, the instances system, lets you have a different game for each instance anyway.

So I don't think there are any advantages of OpenMW having its own game plugin, but do think there are disadvantages.

The good thing is that if I'm wrong, and everything relevant gets moved into plugins like I'm pushing for, then another plugin can be made that does treat OpenMW as its own game, and people can pick between them.

treating it as its own game would also (hopefully?) facilitate referring to the proper, useful, openmw.cfg, instead of the vestigial morrowind.ini OpenMW users can't do anything with.

I'd consider these two things to be orthogonal. The two existing exporters show that editing openmw.cfg doesn't necessitate it being a separate game, and Kezyma's one does it just as seamlessly as MO2 edits the original INI, including integrating with the profile-specific INI system (and also seamlessly enabling the broken archive invalidation BSA which doesn't even work for the original engine as MO2 doesn't know how to tell it to load the BSA).