raylib callbacks system

[raysan5]

I open this discussion to organize a bit my ideas about function callbacks in raylib and also allow anyone to comment.

I tried to avoid callbacks usage in raylib since the beginning, I find callbacks complex to understand and use for beginners and they only introduce a new level of complexity and insecurity to the API that most users probably don't need.

Advance users are expected to use raylib directly from sources and tweak the library for their needs, that's the professional way to deal with open source libraries when you are developing a custom product. For those users, raylib offers a set of defines to enable/disable several features but also change some internal configurations. For example, memory allocation/free can be customized just re-defining RL_MALLOC/RL_FREE macros.

One callback was introduced some versions ago (TraceLogCallback) to redirect log messages at runtime and now I'm considering introducing several more callbacks. Actually, I'm writing this post to organize my ideas around that.

Callbacks considered belong to 3 categories:

1. Log messages (traceLog)
2. Memory management (memAlloc, memRealloc, memFree)
3. File access (loadFileData, saveFileData, loadFileText, saveFileText)

Proposed implementation:

// Callback to for log messages
typedef void (*TraceLogCallback)(int logType, const char *text, va_list args);

// Callbacks for memory management (alloc/free)
typedef void *(*MemAllocCallback)(int size);
typedef void *(*MemReallocCallback)(void *ptr, int size);
typedef void (*MemFreeCallback)(void *ptr);

// Callbacks for file access
typedef unsigned char* (*LoadFileDataCallback)(const char* fileName, unsigned int* bytesRead);
typedef void (*SaveFileDataCallback)(const char *fileName, void *data, unsigned int bytesToWrite);
typedef char *(*LoadFileTextCallback)(const char* fileName);
typedef void (*SaveFileTextCallback)(const char *fileName, char *text);

Every category implies some concerns.

1. Log messages (traceLog)

Actually this callback has already been available for some time, I didn't know of anyone using it or if it has ever been used.

raylib has two config.h flags to enable/disable tracelogs:

• SUPPORT_TRACELOG: Enables/Disables trace log system.
• SUPPORT_TRACELOG_DEBUG: If SUPPORT_TRACELOG, enables/disables LOG_DEBUG level traces.

raylib also defines two macros for custom tracelog functions (require raylib recompilation):

• TRACELOG(level, ...): If not defined by user, it maps to utils->TraceLog() function.
• TRACELOGD(level, ...): If not defined by user, it maps to utils->TraceLog() function.

I think trace log system is quite flexible if dealing with raylib sources, allowing custom tracelog but, by default, it could not be hook at runtime, that's why a callback was added.

• TraceLogCallback type: Internally defines traceLog variable.

SetTraceLogCallback() allows setting a custom TraceLogCallback at runtime.

traceLog callback is checked when calling TraceLog() function mapped by TRACELOG()/TRACELOGD() macros. So, raylib source code configuration will prevail over traceLog runtime callback. Some points to think about here:

• Does a regular raylib user need to hook trace log system? Possible real situation?
• Do advance users prefer setting a callback over the code tweaking?
• Should the callback depend on the raylib sources configuration or not?
• TraceLogCallback must follow a fixed signature, what about if users wants a custom tracing implementation? TRACELOG macro supports that option...
• What about security risks? Could allowing hooking a custom function be risky in some way?

2. Memory management (memAlloc, memRealloc, memFree)

raylib has been redesigned to improve memory management for some versions now. Actually, some custom functions were even added not long ago to expose raylib internal memory allocators, in case users need them.

raylib defines several macros for custom memory allocators (require raylib recompilation):

• RL_MALLOC(sz): If not defined by user, it maps to system function malloc(sz).
• RL_CALLOC(n,sz): If not defined by user, it maps to system function calloc(n,sz).
• RL_REALLOC(ptr,sz): If not defined by user, it maps to system function realloc(ptr,sz).
• RL_FREE(ptr): If not defined by user, it maps to system function free(ptr).

Users can customize memory allocators just redefining those macros and recompiling raylib.

Lately some functions were also added to expose the value of those macros:

• MemAlloc(): It internally calls RL_MALLOC().
• MemRealloc(): It internally calls RL_REALLOC().
• MemFree(): It internally calls RL_FREE.

At this moment I'm considering adding some additional callbacks:

• MemAllocCallback type: Internally defines memAlloc callback variable.
• MemReallocCallback type: Internally defines memRealloc callback variable.
• MemFreeCallback type: Internally defines memFree callback variable.

Some points to think about here:

• Should MemAlloc() actually check if memAlloc is defined and use it?
• Should memAlloc replace RL_MALLOC(), which one should have priority?
• Should memAlloc be allowed to be changed in the middle of a program execution??? Without going into details yet, that does not sound very nice to me...
• Should memAlloc be hook before raylib initialization (like other libraries do)?
• MemAllocCallback must follow a fixed signature, what about if users wants a custom implementation? RL_MALLOC macro supports that option...
• What about security risks? We are dealing with custom memory accessors...

3. File access (loadFileData, saveFileData, loadFileText, saveFileText)

raylib file access system has also been redesigned in raylib in the last versions to load data from memory, centralizing all file access into 4 functions in the utils module, so, almost all file loading/saving in raylib ends up using:

• LoadFileData(): Loads file data into memory and returns data buffer + byte size.
• LoadFileText(): Load file as text into memory and returns text data buffer + byte size.
• SaveFileData(): Save data buffer into a binary file.
• SaveFileText(): Save text data buffer into a text file.

This category has no macros to allow replacing those functions, so, callback implementation could be very useful. Probably the most common usage scenario is loading/saving data from a custom file-system (i.e. a ZIP file, memory, network...).

The callbacks considered for this case are:

• LoadFileDataCallback type: Internally defines loadFileData callback variable.
• SaveFileDataCallback type: Internally defines saveFileData callback variable.
• LoadFileTextCallback type: Internally defines loadFileText callback variable.
• SaveFileTextCallback type: Internally defines saveFileText callback variable.

Those callback objects are checked inside every relative function (i.e. loadFileData checked on LoadFileData() function) and, in case the callback has been defined, the callback is executed before the default implementation.

Actually, one improvement over it could be creating a flag (SUPPORT_STANDARD_FILEIO) to avoid the default implementation on compilation, that way, it would be possible to just omit stdio library requirement (i.e. using a custom clib implementation). b084552

Again, several points pop into my mind to think about:

• Should file io callbacks be set before initialization? or can be set at any time during program execution?
• LoadFileDataCallback must follow a fixed signature, what about if users wants a custom implementation?
• What about security concerns? We are dealing with custom file access!

Ok, so, that's it, now it's time to think about it and try to find a balanced solution.

[Elkantor]

I'm not a big fan of callbacks, even if I clearly see the different sort of advantages they offer (no states, less conditions in the codebase...).
The real problem with callbacks is that it makes the code not really readible in my experience.
Plus, in a multithreading environment (so here it's not something beginners will experience, it's more for advanced users), debugging with callbacks is a real pain...
And finally it's less trivial to implement bindings for other languages.

[JeffM2501]

I think your comments assume that the callbacks will be widespread and mandatory. In this case nether will be true. They are optional callbacks that CAN be installed, and only because they do offer advantages. Memory callbacks are sometimes necessary on very memory restricted platforms and you can't rely on the the compiler to handle malloc/free correctly (such as cases where you need to use a memory poll). File IO callbacks are the only way to hook into an archive system (making raylib read from like a zip file with all assets). Tracelog callbacks may be the only way to get logging info in a native windows application. All of these solve a specific need and are not just there for fun.

[JeffM2501]

I think these callbacks are very important as people try to use raylib to do more and more things. They are going to be the only solutions to some of the problems people face (like reading assets from an archive). I direly need these for some things I have planned.

I totally agree that they can be confusing to beginners. I suggest that they be exposed in a sperate header, such as raylib_callbacks.h, that is not included in raylib.h by default. This will make it much harder for beginners to "stumble onto it" and become frustrated. It would also clearly mark it as a more advanced totally optional feature. The important thing is that they are 100% optional.

As for security risks, I don't see this as being any different from the security risk in the user's own code. They will have to install the callbacks at compile time, and if they expose something they should not have, then they should have tested it, just like anything else they put in the code they wrote.

[jmorel33]

These suggestions are awesome as it opens up a whole set of possibilities.

If this is implemented I will use most of it.

[vb-mich]

• Does a regular raylib user need to hook trace log system? Possible real situation?

I don't think a regular raylib user would need a trace log callback system. The only possible real application I can think of is if you need to redirect logs on specific platforms.

• Do advance users prefer setting a callback over the code tweaking?

As an advanced user myself I say yes.

• Should the callback depend on the raylib sources configuration or not?

Yes, makes me feel more safe.

• TraceLogCallback must follow a fixed signature, what about if users wants a custom tracing implementation? TRACELOG macro supports that option...

Imho in this case the flexibility gain is not worth the simplicity loss.

• What about security risks? Could allowing hooking a custom function be risky in some way?

None, as far as I concern

• Should MemAlloc() actually check if memAlloc is defined and use it?
• Should memAlloc replace RL_MALLOC(), which one should have priority?
• Should memAlloc be allowed to be changed in the middle of a program execution??? Without going into details yet, that does not sound very nice to me...
• Should memAlloc be hook before raylib initialization (like other libraries do)?
• MemAllocCallback must follow a fixed signature, what about if users wants a custom implementation? RL_MALLOC macro supports that option...
• What about security risks? We are dealing with custom memory accessors...

Memory callbacks could be used only on very specific cases by very advanced users, even tho I'd personally benefit from this (I develop on limited memory embedded boards) I believe it's a bit out of the scope for a game/multimedia library. Anyway, I'd be nice to have.

• Should file io callbacks be set before initialization? or can be set at any time during program execution?

I'd prefer the second option.

• LoadFileDataCallback must follow a fixed signature, what about if users wants a custom implementation?

If a user wants a custom implementation then I assume that user do not need a pre-made callback system in the first place. I prefer to keep it simple.

• What about security concerns? We are dealing with custom file access!

Simplicity and freedom come at a cost: responsibility.

[raysan5]

Thank you very much for your detailed review!

[leoagomes]

The main positive I'd see with a callback system as proposed, would be the ability to add, for example, PhysFS to raylib's file loading mechanisms, but that would surely be a more advanced use case.

[chrisdill]

I think a config style struct for callbacks is worth considering.
This would only need 2 functions to get and set it which makes the api less confusing for new users.

The default state can still be set up via macros allowing both use cases to work together.
This approach is also already used with most external dependencies as well.

Example usage:

// Naming is hard...
ConfigState state = GetConfigState();
state.traceLog = &MyTraceLogCallback;
SetConfigState(state);

InitWindow(1024, 600, "Test window");
// ...

[raysan5]

Personally, I prefer to keep the callbacks as separate functions for hook.

[jamesl-github]

I love callbacks
callbacks are great
that's why I prefer GLFW to SDL

[gilzoide]

To be honest, I see little point in adding these runtime callbacks when you can easily implement them on top of the compile-time macros like RL_MALLOC and friends.
Also, if raylib itself is not going to call the callbacks, like RL_MALLOC is called, but MemAlloc is only defined for users to call, there is even less point in doing it in the core library.

I checked that this is not the case with LoadFileData and friends, but maybe it would be even better to use the same design around all of the configurable functions and add RL_LOAD_FILE_DATA and friends macros that default to calling them.
Having a Load*FromMemory-like variant for every Load* function is another good option for avoiding IO callbacks and permitting custom IO to be performed as needed.

Of course, I am assuming people using callbacks would already be compiling raylib alongside their project.
If raylib was mainly supposed to be linked as a shared library this would not be the case, but even then it's easy to build a version of raylib where the macros call runtime defined callbacks.

---

After all that said, though, it may be nice to have an implementation for it, because people would be more comfortable reading other people's code.
Enabling the callback system could also be configurable at compile-time and possibly disabled by default.

* must follow a fixed signature

Even as macros, they still have to follow a non-strict signature for the code to compile, so no worries there.

What about security risks?

Unless one carelessly puts a pointer for anything other than a function, or one of wrong signature, as callback, no worries as well.
Compilers will give errors for those, unless there is an explicitly cast.

If one loads a shared library as a plugin and install some symbol of it as a callback, on the other hand, things can go very wrong.
But this would be a super rare case where the responsibility for setting up security measures is on the user anyway.

[JeffM2501]

Having a Load*FromMemory-like variant for every Load* function is another good option for avoiding IO callbacks and permitting custom IO to be performed as needed.

That won't work for loaders that read additional files, like obj when it needs to pull in a material file. Trying to make "from memory" versions is what started the request for file IO callbacks.

[gilzoide]

Oh my, you're right. I mostly do 2D stuff, so I always forget about these cases, sorry =/

As I can see, tinyobj would have to be updated for this to work, although should be easy enough.

[JeffM2501]

TinyObjLoader has to be updated anyway, it has bugs in it that prevent multi-material files from loading in Raylib 3.5.
The current version of TinyObjLoader itself has a file read callback for the exact same reason that raylib needs one.

typedef void (*file_reader_callback)(const char *filename, int is_mtl, const char *obj_filename, char **buf, size_t *len);

Raylib would just hook into that and call the existing file read functions that would check the callback before calling the existing native reader if no callback existed.

[raysan5]

Just for completion, here some of my final conclusions.

1. Log messages (traceLog)

Some people is already using it and it seems to be useful for some situations. It was already added so it won't be removed. Just note that source code config has preference over the callback, so, if SUPPORT_TRACELOG is not enabled, the callback will never be called.

2. Memory management (memAlloc, memRealloc, memFree)

I'm not adding this ones. Advance users requiring custom memory allocators can directly edit the sources and redefine the macros.

3. File access (loadFileData, saveFileData, loadFileText, saveFileText)

They are just a must to hook an external file-system, so, they will be added.

[konsumer]

I don't mean to necro this discussion, but I was recently working on a physfs-based custom-filesystem. The current hooks work great for read/write files, but there are still a few functions that have no way to hook (even with macros, as far as I can tell.)

FileExists
DirectoryExists
GetFileLength
ChangeDirectory
IsPathFile
LoadDirectoryFiles
LoadDirectoryFilesEx
GetFileModTime

I wouldn't mind overriding these with a macro or a callback-thing, but as it is, I had to add a suffix to the names. In this setup, I am trying to make an API that looks just like raylib, so I would prefer if they all used the original names.