Oolite is still on SDL 1.2

[alugarius]

I want SDL2!
I play Pioneer instead of Oolite because of this.

[sjnewbury]

I'm thinking I'll take look into porting to SDL2.

In #91 it was mentioned there are local changes to the in-tree binary libSDL-1.2, the Gentoo ebuild produces an Oolite binary linked to the system libSDL-1.2.so, maybe that explains the fullscreen vblank sync and performance issues with my build... nope it seems not. Is there any more information on what changes have been made to the in-tree version?

[AnotherCommander]

@sjnewbury Yes. All modifications to the standard libraries are listed here:
https://github.com/OoliteProject/oolite/blob/master/Doc/ExternalLibrariesSourceCodeChanges.txt

[sjnewbury]

I've made a start. Initially I'm just doing the bare minimum to get it working with the same functionality. Later new features and capabilities are possible such as multiple screens OpenGL3/4 core profiles etc.

I don't know if the Windows changes noted above are still necessary with SDL2. I'll create a pull request once I've tested it here on Linux, hopefully somebody with a Windows build setup can test it.

[sjnewbury]

Some thoughts while working on this:

Splash screen could/(should?) be a separate window - I'm leaving the code as is at this stage for minimal invasiveness.

A lot of special casing for Windows is probably not necessary with SDL2. I've removed some, but not all of this code. It will need testing. Probably most of it can be dropped.

There is a lot of code handling screen modes, I'm dropping it all on the floor. The API has improved for SDL2 here and I don't think any of it is needed. Unless I'm missing something...

"Surfaces" are a "compatible" feature with SDL2, to use GL you just need to create a GL context, so I've replaced "surface" with "glContext" thoughout, with the exception of the screenshot handling code, where I'm specifically creating a surface from the window for that function. Hopefully, that will let me get away with leaving all that code alone.

HiDPI support should work on all supported platforms now, but I can't test it. I'm trying not to break it!

[sjnewbury]

I left in the screen mode support largely as is for now. As I mentioned, I just want to alter as little as possible.

I've got the game running with "-nosplash". Obviously buggy, since without "-nosplash" the window fails to appear on screen although it shows up in the window list.

I've still got some of the code I added to replace the screen mode handling which I guess I will revert before creating the (provisional - for testing) pull request.

Good news, fullscreen is now fast and smooth for me.

[sjnewbury]

I am tempted to rip out and replace the splash screen code with a separate window.

[sjnewbury]

@AnotherCommander: Since the splash code has broken I've just ripped it out and left stubs. Perhaps I should copy the SDL directory to SDL2 before making the the pull request and make it a build time option? Preferences? It is working better for me than the SDL1 code in fullscreen. There is still too much code, on the other hand what I have now is working.

I'm also wondering about the fullscreen mode switching generally, do we really want to switch resolutions rather than always use native? I did try to implement the fullscreen mode switch using the viewSize like on Windows but I've now dropped that code for simplicity. It is however mode switching when leaving the game to the desktop (since task switching works with fullscreen SDL2, along with volume controls!) and then still uses that low resolution on return since it's then the desktop resolution!

[AnotherCommander]

@sjnewbury First of all, many thanks for the effort you are putting into this. It is much appreciated.

The SDL2 code should definitely be inserted initially as a compile-time option. Whatever changes you are proposing, we should still be able to build the game using the current SDL. As I am sure you are aware, the transition to SDL2 is not a simple task and is, in fact, very invasive, considering the stability we have managed to achieve to this point. It is changing the functionality of the game at low-level. As a consequence, some very heavy and detailed testing will be required and this could take long, even months, depending on the availability of people and test systems. So it is necessary that, until we are happy with the status of the SDL2 port, we keep the current SDL tree unning in a totally unaffected manner.

Regarding dropped features and stub functions: I would not agree with dropping existing features. The fullscreen mode changes may not seem of much use these days, but keep in mind that we are targeting also older machines and those may need the alternative modes for performance reasons. In any case, I would rather not have to pay the price of leaving behind existing features in favor of SDL2. If features do have to be dropped, then we'll have to have internal discussions about how we want to proceed. This is another reason why the existing SDL should remain alive until we are satisfied that the transition will be as transparent as possible.

While we are at that, please test also joystick input. This is very important, almost critical I would say. Joysticks, controllers etc. must continue to function exactly as before, on all platforms. Also, multi-monitor support will have to be tested as well to ensure that it works on Linux or, at the very least, that it does not break the Windows port.

Speaking of which, we'll have to see how to go about testing in Windows once the PR has been submitted. For the Windows port, all support libraries are built by ourselves; we do not use pre-built binaries and this is so that we are able to control any and all aspects of the game's code (we've already had to do plenty of editing into support libraries code, including gcc code in one extreme case - I can provide details if you want, but I'd rather forget about that torture :-)). So, SDL2 will have to be built for Windows, the Windows Dev Environment updated, then proceed with testing as time permits. I can tell you right now that this is going to take a while. I prefer putting time into the game's code rather than into its libraries and time is in short supply. I also don't have ability to test on a 64-bit system currently, so we'll have to see how we'll go about that too.

And last but not least, even the Limux builds we provide may have to change the way they are generated, if SDL2 moves on. I'll let the details of this to our Linux experts, but be aware that the Linux builds we provide are completely independnet of distros and libraries provided by such distros.

In summary, this is a subject that will probably not be resolved immmediately. We'll have to take time with it, test it thoroughly - and I mean that - and proceed only if we are all confident that the change is not causing more issues than it resolves and that the game is still as stable as we expect it to be. Looking forward to the first iteration of the PR when you have it ready.

Thanks again for your work on this.

[sjnewbury]

I'm going to create a pull request with what I have now.

Outstanding issues::

1. Splash screen doesn't work so I have removed the code for now except the switch/config settings.

2. Fullscreen mode switching isn't now supported by the Linux code. Hopefully, the Windows code still works.

3. Initially when going fullscreen there is odd flickering. Switching back and forth "fixes" it. Not sure why. It then works well.

Joystick, mouse, keyboard, sound all seem to be working.

[sjnewbury]

@AnotherCommander, thanks for the feedback. I'll split the SDL directory into an SDL2 directory with the changed files, and make it build time selectable before making the PR.

I'll see if I can get the splash screen going again as well, I was just spending too much time on it earlier today it's going to take a clean approach, I think.

[snipped]

The preferred SDL2 method is actually to always use the native resolution for display and use internal scaling from a lower resolution since not all SDL2 targets support mode switching, for example, native Wayland should be working as is. Android and iphone could work to.

[sjnewbury]

#234

[sjnewbury]

I just tested it under Weston. It works, it's buttery smooth, but it shows some issues with window resizing especially when switching to fullscreen.

[sjnewbury]

@AnotherCommander Many issues addressed. Mode switching works again. The code is back in place for the splash screen, but it isn't quite working due to a problem with the OpenGL bounds/offsets I think. The image often appears shifted down out of the view window. As I mentioned, I tested it under native Wayland (no X server at all), and it works fine. I think the flickering might well be related to the above mentioned OpenGL issues, and is probably due to my lack of experience/knowledge in that area, I'm working on it, but help would be really appreciated. This really clearly shows up when going up screen resolutions in fullscreen mode!

[sjnewbury]

One thing about the mode switching, SDL2 supports refresh rate, so that really needs to be supported properly, where currently it's only partial.

[sjnewbury]

Multi-monitor support is limited to only using the first display, that's where the mode list comes from. SDL2 isn't limited in that regard, and supports multiple windows and displays, it could in theory display a window for each Oolite game screen simultaneously and/or map them to multiple displays when fullscreen. That would be pretty cool IMHO.

[AnotherCommander]

I have managed to compile SDL2 2.0.5 for Windows. First attempt at compiling Oolite with it: ended early:

In file included from src/Core/Debug/OOJSConsole.m:42:0:
src/Core/OOOpenGLExtensionManager.h:225:30: error: 'glActiveTextureARB' redeclar
ed as different kind of symbol
In file included from src/Core/OOOpenGLOnly.h:58:0,
                 from src/Core/OOOpenGL.h:31,
                 from src/Core/Universe.h:28,
                 from src/Core/Scripting/OOJavaScriptEngine.h:27,
                 from src/Core/Scripting/OOJSEngineTimeManagement.h:31,
                 from src/Core/Debug/OOJSConsole.m:35:
C:/____DevelopmentEnvironment/gcc/Msys_x2/1.0/mingw/../devlibs/include/SDL2/SDL_
opengl.h:2019:23: note: previous declaration of 'glActiveTextureARB' was here
In file included from src/Core/Debug/OOJSConsole.m:42:0:
src/Core/OOOpenGLExtensionManager.h:229:35: error: 'glClientActiveTextureARB' re
declared as different kind of symbol
In file included from src/Core/OOOpenGLOnly.h:58:0,
                 from src/Core/OOOpenGL.h:31,
                 from src/Core/Universe.h:28,
                 from src/Core/Scripting/OOJavaScriptEngine.h:27,
                 from src/Core/Scripting/OOJSEngineTimeManagement.h:31,
                 from src/Core/Debug/OOJSConsole.m:35:
C:/____DevelopmentEnvironment/gcc/Msys_x2/1.0/mingw/../devlibs/include/SDL2/SDL_
opengl.h:2020:23: note: previous declaration of 'glClientActiveTextureARB' was h
ere
make[3]: *** [obj.win.spk/oolite.obj/OOJSConsole.m.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [internal-objc_program-all_] Error 2
make[1]: *** [oolite.all.objc-program.variables] Error 2
make: *** [internal-all] Error 2

I can already see that this is not going to be easy, like, at all...

[KonstantinosSykas]

@AnotherCommander ...and I thought MY compilation issues, in the Linux DevEnv, were THE problem! 😥

[sjnewbury]

@AnotherCommander Conflicting GL headers? I don't know how that differs for Windows.. No such errors here. @KonstantinosSykas did you have the same failure with Linux?

[sjnewbury]

I've not been working on this for a few days, been busy flying my Barracuda and hacking on OXPs instead! ;-) I'll see if I can work out what's happening above. My next plan for this week is to try using the Windows fullscreen handling code for the generic case just changing out the Windows calls for equivalent SDL2 calls.

[AnotherCommander]

Conflicting GL headers? I don't know how that differs for Windows.

It differs quite substantially because on Windows we have to declare every single function newer than OpenGL 1.1 as an extension. This involves plenty of hocus-pocus with function pointers, but it's a more or less standardized routine for OpenGL applications. It does mean that the handling is completely different on Windows, though. You can see how exactly this works in OOOpenGLExtensionManager.m inside the Windows-specific guards. I'll try to have a look at it if I can, but can't say when that will be.

[KonstantinosSykas]

Allow me to bring the discussion in here. The content is more general than just PullRequest-related.
I quote to connect with what was previously said in the Pull REquest thread.

KonstantinosSykas commented:

Hi. I'm maintaining the environment that produces the Linux distro independent Oolite installation packages.

The environment is an Ubuntu 8.04 (2.6.24-28 kernel) with a gcc v4.2.4. Pretty archaic, I know, but it has demonstrated to produce robust binaries with astonishing compatibility in the big ocean of Linux distros, making happy a lot of our followers having an old laptop with an old Linux distro, still managing to play Oolite decently!

Compiling libSDL2 on this environment breaks, complaining about missing mutlitouch device definitions.
Furthermore, warns about (among other things)

pragma GCC diagnostic push/pop (old gcc)

and

dereferencing type-punned pointer will break strict-aliasing rules

the second one leading to undefined behavior, from a quick search that I did.

I am not familiar with Gentoo, but do you get SDL2 source code that gets compiled in this enviroment or a latest binary from Gentoo repo? If it is source that you get, then can you share the configure and make logs?

sjnewbury commented:

@KonstantinosSykas Everything is compiled from source with Gentoo. Giving you my logs wouldn't be much help though since I'm mostly bleeding-edge, my compiler is one of the few "stable" components gcc-6.3.0, so by your standards, that's almost from the future! ;-)

That's a really old kernel!

I'm not sure what versions are present with Ubuntu 8, but the Gentoo ebuild might well help you out with minimum versions and configure flags since Gentoo does try to keep supporting fairly "legacy" setups and will build with few dependencies if they aren't needed:
https://gitweb.gentoo.org/repo/gentoo.git/tree/media-libs/libsdl2/libsdl2-2.0.5.ebuild

Regarding the second GCC warning, I wouldn't be surprised if the lack of warning with 4.2.4 doesn't mean the behaviour is any different!

KonstantinosSykas commented:

My concern is that, at the end of the day, to be included in a release, the libSDL2 will have to come out from that environment, unless

• the project team decides to set a baseline that will define the next gcc to be used, defining a backward compatibility level with older Linux distros and
• someone will stand up and decide to setup the new development environment to support that decision.

Hmmm... food for thought for all of us. 😉

sjnewbury commented • edited:

@KonstantinosSykas Since it's a build time option, unless and until use is made of the SDL2 capabilities perhaps having two builds would be the way to go? One for legacy systems and one for modern/alternative systems. I'm tempted to set up an Android build environment to see if I can make a build that works on my phone.

Of course this would mean it wouldn't be feasible to implement features like multiple Oolite screen -> display mapping.

This would actually mean a whole list of things, such as:

• draw a redline and define what we will call "modern/alternative systems". (What will be the oldest kernel to support? As a consequence, what will be the oldest GCC to use? etc.)
• I am putting this in a separate bullet, because of it's importance; because, at the end of the day the purpose is to have Oolite players, easily entering the Oolite reality pausing RL. How will they be impacted and what will they gain at the end of this journey?
• build at least two new development environments (one for Windows and one for Linux; "at least two" because, we currently have two for Linux, as there was never enough time to find out how to automate the Linux32 binary build from the 64bit environment).
• If we are to modernize, then let's do it properly; all dependencies should be brought forward to the stable version dated "back to the future" baseline/redline (confusing?... wait, the best is yet to follow...).
• The new libraries will most certainly come with new dependencies (we experienced this while resolving [Windows] In-game packs manager can not download OXP over HTTPS #205). This means that the minimum set of libraries packed with Oolite installer, must be redefined (i.e. checked with at least a couple of distros for both 32bit & 64bit architectures, from each big Linux family such as RPM, Debian, Arch, Slackware, as well as some major and popular independents, Gentoo, openSUSE, Fedora etc.
• 8 more binaries to support until the transition (5 releases as usual, 4 releases as not usual, 5 nightly builds as usual and 4 nightly builds as not usual)
• extensive testing (bleeding, considering everything will be new) in terms of duration and amount of community testers, to ensure stability with the new libraries ecosystem; it gives me chills to only think of handling defects in system where everything is new.
• someone will come up with the time to allocate on building the new infrastructure

These are just the first things that I think of and I am mostly speaking for Linux. 😓

[KonstantinosSykas]

Concerning the GL headers compilation issue...

@sjnewbury That point is not reached yet.
SDL2 compilation is failing in the Oolite for Linux Development Environment. 😞 😫

[sjnewbury]

@KonstantinosSykas Yes that's another issue... I'll see if I can create a VM and get SDL2 build in the "Oolite for Linux Development Environment". I''m quite used to getting things to compile against their will! ;-)

@AnotherCommander Perhaps SDL2 takes care of all that for Windows now? I understand that the Windows support is much better than with SDL1, regarding cross-platform feature mapping and support. Windows was always and afterthought for SDL1.

I'm afraid I got a bit sidetracked into learning how to write OXPs this week. I'll get back to this soon...

[AnotherCommander]

@sjnewbury Can you please clarify a bit for me what you mean by "all that" in your previous response? I am not entirely sure what exactly you are referring to.

In any case, SDL1 on the Windows version does fine most of the things we need and whatever it doesn't do that we want, we do by use of direct WinAPI calls. Not sure what will be needed to get an SDL2 port up to the same level we are at now, but we will need to achieve at least a 1-to-1 parity with the SDL1 build status before anything gets accepted. Having to do additional reading up on SDL2 just to get it on par with the current build does not help the situation much.

As for SDL2 Windows support status, I have been doing some investigation on the compile issue reported earlier (I ended up spending time on non-game stuff after all :-( )and, if the problem is what I think it is, then I can say that SDL2 has still some road to cover on Windows before we can call its support adequate. I'll do some more tests to be sure and get back to you on that. But so far, I am not completely convinced that SDL2 (at least on Windows) is mature enough to be adopted.

[sjnewbury]

@AnotherCommander Sorry about being unclear, I was referring to handling of OpenGL headers in a cross-platform manner. I'm afraid I haven't used SDL at all on Windows, my Windows development experience is almost non-existent in actual fact.

From the horses mouth:

https://wiki.libsdl.org/FAQWindows

https://wiki.libsdl.org/Installation

SDL2 implements a lot of the functionality that is currently handled in Oolite by direct Windows API calls, how well it works, and whether it works on all supported Windows versions I'm afraid I do not know.

[sjnewbury]

SDL1 only incidentally worked [was made to work] on Windows it was originally made to port Windows games to Linux and MacOS 9 which is why all the Window API calls are needed.

[AnotherCommander]

Well, that was interesting...

I have concluded with my investigation regarding the Windows SDL_opengl.h compile issue. At the moment, the Windows SDL2 build attempt goes past the point where it was failing earlier but still fails later on when trying to compile MyOpenGLView.m. But I think this time it has more to do with porting details between SDL1 and SDL2 rather than actual SDL2 bugs. But before I go on with the results of the investigation, two quick comments to set the record straight:

1. I fully believe that if SDL2 gets working to a satisfactory level for both Win and Lin platforms, it will result in much more portable code and will allow us to get rid of a lot of platform-specific ifdefs, making life easier for everyone. That will definitely be a big plus on its favor and that is before we even consider the increased capabilities of SDL2 and the flexibility it offers.
2. SDL2 on Windows compiles and installs successfully. I have had no issues building the library and had also the chance to test its functionality both with the tests included in the source distribution, and, more importantly, by using it in place of the SDL2.dll distributed with other projects running on top of SDL2, like Naev. Therefore, I know for sure that what I have built is fully functional and works as intended.

With that said, let's go back to our specific build problem. As it turns out, it is a (quite serious IMO) bug in the SDL2 headers. The problem is that the OpenGL function declarations for glActiveTextureARB and glActiveClientTextureARB are not guarded in SDL_opengl.h. In SDL1, the block of GL_ARB_multitexture declarations containing those two functions is guarded with an #ifdef GL_GLEXT_PROTOTYPES preprocessor statement, which is entirely missing in the equivalent SDL2 ones. At some point during SDL2 development those declarations, together with many others, were moved to another header named SDL_opengl_glext.h, which is included at some point by SDL_opengl.h. However, on August 18th 2014, there was a commit that tried to avoid having to use system OpenGL headers, that added a bunch of declarations that were already declared inside SDL_opengl_glext.h and not only did it add them, but it also added them unconditionally (see https://hg.libsdl.org/SDL/log/a0327860b8fb/include/SDL_opengl.h, commit titled "Don't use the system OpenGL headers, ever."). And it added them unconditionally, BEFORE attempting to include SDL_opengl_glext.h. This is why we end up having redeclarations. I have not tried to test this, but I believe that any of the many other non-guarded similar declarations in SDL_opengl.h will cause a similar issue and I am really surprised that in all these years the bug has been in there, there seems to have been only one complaint (at least one that I could find) in the SDL forums. Link: https://forums.libsdl.org/viewtopic.php?p=52283.

So, after seeing this kind of bug remaining unfixed for three years now in a supposedly stable version, I have some doubts about how suitable SDL2 currently is for Oolite on Windows in its current shape and form. I have really spent a lot of time here for something that looks like an SDL2 bug and am once again debugging libraries, which is exactly what I don't want to do. I wonder what else awaits us if we hit those kind of walls so early in the process. Of course there is always the chance that I may have misunderstood something - at which point I'd be glad to be corrected - but I keep asking myself whether the effort and time required is justified with respect to the expected benefits. And I have not even touched any of the issues @KonstantinosSykas mentioned about Linux itself either. I'd like to try to continue the attempt to build the game under SDL2 now that I have modified the SDL2 headers and went past that hurdle just to see how it runs, if anything, but I am also starting to run out of patience. I'm afraid that if I hit one more issue like that, I may just let it go and not try anymore. Will post more once I have the chance to do more on this (hint: not soon).

[Luraktinus]

Any news on this?

[AnotherCommander]

No great news on this. PR #234 has not been updated for a while and we do not have 1:1 parity with the SDL 1.2 build yet. Regarding the Windows port, I did manage to get an executable at some point, but only after disabling most of the Windows-specific window management code, thus sacrificing multi monitor (and proper fullscreen) support. The result was definitely not satisfactory and there werre also some serious issues which seemed related to threads management, but I was so frustrated at that point that I decided to not be bothered any longer. I have deleted my local copy of the SDL2 attempts since.

[sjnewbury]

I will get back to looking at this again, unfortunately I bought Elite Dangerous for my PS4 and there's only so much time...