Notes on external vs. cexternal

[samiam95124]

Notes on external vs. cexternal

First, cexternal turns out to be a very bad idea that may end up getting implemented anyways. Why? Because cexternal is a keyword that occurs long after the parameters have been parsed. Thus it comes down to front end vs. back end. That is, can the C language conversions be performed entirely in the back end, or do they need front end support? The jury is still out on that one. The alternative is to use something like "clang" as a keyword before the procedure or function, or perhaps just after it.

Wrappers for external

Having almost completed the services.pas wrapper for Petit-Ami's services.c, its obvious that performing that creating a wrapper in assembly is non-trivial. Examples include string conversions, both to C and from C, data structure conversions, etc. pgen could help with many or even all of these conversions, and it is tempting.

However, the reality is that nobody out side of people working on wrappers for Petit-Ami are going to do that work (ie., me). Virtually all C module adaption work is going to be done using cexternal (or whatever the facility ends up as). Thus individual, hand coded assembly conversions are not that bad a way to go.

Why not cexternal for Petit-Ami?

I'll give that a firm "maybe". The first domino that would fall is that overloads aren't implemented in C, so the Pascaline interface would have to change to exclude them. The Pascaline interface for the libraries was created back when the libraries themselves were written in Pascaline, so it was never an issue. An example of this is strings.pas, which makes heavy use of Pascaline constructs and could never be coded in C.

A retospective: wrappers everywhere?

In IP Pascal, circa 1995, wrappers were rare, for the simple fact that all of the code IP Pascal relied on was also written in Pascal. This was taken to extremes. The paslib.pas and syslib.pas, the Pascal language support and the base OS interface, respectively, were written before the compiler was operational. This meant that those modules had to be debugged on a compiler that had not proven to have working output.

So none of that needed wrappers. The other modules, like services and the rest of Petit-Ami, were also written in Pascal. And so no wrappers were needed. The wrappers that actually were made, like the general Windows API functions (yes, IP Pascal could use the Windows API) were hand encoded in assembly, which was itself a fair amount of work.

This resulted in two efforts. The first was the creation of a "super macro" that could automatically generate wrappers by invoking a series of submacros on each parameter type to convert it. The second was a program, ch2ph, which translated C header files automatically into Pascal header files. At the same time, it would generate an assembly file containing wrapper macros for each Windows API function.

Although I believe that Pascal-P6 does this job more simply by its better integration with the C language, several of the lessons of IP Pascal still apply. The first is that macros are used in the current Pascal-P6 wrapper files to reduce the complexity. The second is that the ch2ph file converter is certainly going to be refactored for Pascal-P6. The open question is: if the wrapper generator is really required anymore for Pascal-P6. Its possible that, with cexternal, a ch2ph converted header file can be directly used by Pascal-P6.

That would, of course, reduce the amount of work in wrapper files to zero, or close to it.

Wrappers in Pascaline

Another idea is to write the wrappers for Petit-Ami in Pascaline. With cexternal, Pascaline can access C routines, but without coining or overloads. However, Pascaline itself can implement that on it's externally facing interface, then translate the overloaded interface into C call form.

[samiam95124]

Plan B

Ok, having a day to think about this, here is what I came up with and actually implemented.

Wrapper conversions are made simpler by expressing them in a language instead of assembly (duh!). What language? Well, wrapper conversions can get dirty (type wise). And the whole idea here is to isolate type insecurities to the C language. C handles dirty better than Pascaline, so the correct language for the wrapper code is C[2].

So what happens? The present calling system, which is the "compatible call system", so called because it is only kinda compatible with System V AMD64 ABI[1]. It suffers from several issues, which I have hopes will be corrected in later versions[3]:

1. It only handles 6 parameters in registers, or 3 if they are double wide (two registers).
2. If more than 6 parameters exist, they will be in reversed (so called "Pascal") ordering on stack.
3. Pascal-P6 stacks all parameters, not just > 6. This can be mitigated by dumping the first 1..6 parameters off stack.
4. It needs 16 byte alignment before entering C code.
5. It need module coining (services.*).

Ok, so there is still a services_wrapper.asm. For each call, all of these factors are compensated for. That is, the calls from Pascaline are converted to fully C compatible calls. With the new (future) calling convention, these would reduce to zero or close to it.

Next, the C routines are called in services_support.c. The code there performs all of the parameter conversions to and from the Petit-Ami services.c routines.

Now note that in some cases, putting the C call into proper C format is all that is needed. These are simple passthough calls. They can be directly called to services.c, or could be called to passthoughs in the wrapper just for neatness sake. I think that mainly relies on seeing what code is produced, ie., how much code gcc puts in a simple passthrough.

[1] yes, it is a joke, I make them on occasion.
[2] So C does it better than Pascal, and assembly language does it better than both. Lower level means greater expanse of application. That's not why high level languages are important.
[3] The CCS unites all of the different machine ABIs and the interpreter together. Its kinda genius. A system doesn't have to actually be correct to be good.

[samiam95124]

gcc passthrough functions

I did the following experiment:

void externalfunc(void);

void passthrough(void)

{

    externalfunc();

}

void main(void)

{

}

Compiled with:

gcc -O3 -S test.c

Gave the output code:

        .file   "test.c"
        .text
        .p2align 4
        .globl  passthrough
        .type   passthrough, @function
passthrough:
.LFB0:
        .cfi_startproc
        endbr64
        jmp     externalfunc@PLT
        .cfi_endproc
.LFE0:
        .size   passthrough, .-passthrough
        .section        .text.startup,"ax",@progbits
        .p2align 4
        .globl  main
        .type   main, @function
main:
.LFB1:
        .cfi_startproc
        endbr64
        ret
        .cfi_endproc
.LFE1:
        .size   main, .-main
        .ident  "GCC: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0"
        .section        .note.GNU-stack,"",@progbits
        .section        .note.gnu.property,"a"
        .align 8
        .long   1f - 0f
        .long   4f - 1f
        .long   5
0:
        .string "GNU"
1:
        .align 8
        .long   0xc0000002
        .long   3f - 2f
2:
        .long   0x3
3:
        .align 8
4:

ie, the passthrough turned into a jump, which is exactly the minimum behavior I would expect. Thus using dummy functions for passthroughs is a reasonable solution.

This was also found if the passthrough had parameters:

void externalfunc(int x);

void passthrough(int x)

{

    externalfunc(x);

}

void main(void)

{

}

        .file   "test.c"
        .text
        .p2align 4
        .globl  passthrough
        .type   passthrough, @function
passthrough:
.LFB0:
        .cfi_startproc
        endbr64
        jmp     externalfunc@PLT
        .cfi_endproc
.LFE0:
        .size   passthrough, .-passthrough
        .section        .text.startup,"ax",@progbits
        .p2align 4
        .globl  main
        .type   main, @function
main:
.LFB1:
        .cfi_startproc
        endbr64
        ret
        .cfi_endproc
.LFE1:
        .size   main, .-main
        .ident  "GCC: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0"
        .section        .note.GNU-stack,"",@progbits
        .section        .note.gnu.property,"a"
        .align 8
        .long   1f - 0f
        .long   4f - 1f
        .long   5
0:
        .string "GNU"
1:
        .align 8
        .long   0xc0000002
        .long   3f - 2f
2:
        .long   0x3
3:
        .align 8

4:

C compiler optimization is actually pretty stunning.

[samiam95124]

Then what does the .asm layer do?

Good question. It does two things:

1. It connects up coined module names (which don't exist in C) to their base name equivalents.
2. It connects up various overloads to handlers.

[samiam95124]

Stacking interfaces

So the layout of services.c is:

services.h - The Petit-Ami header file.
services.c - The Petit-Ami implementation file
services_wrapper.asm - The Pascaline wrapper front end.
services_wrapper.c - The C wrapper code.
services_support.c - Routines to convert types to/from Pascaline in C.

The services_wrapper.asm uses the trick that .asm files won't be erased by a clean like the normal .s files will[1]. Don't services_wrapper.asm and services_wrapper.c converge on a single .o? Naw, the makefile produces services_wrapper_asm.o. A dirty trick, but it works.

services_support.c will soon be on the move, name wise. The reason is that the routines it contains are applicable to all wrappers. Thus probably wrapper_support or somesuch.

How does this mess get straightened out?

Well, compilers aren't known for being simple programs. However, these things get resolved by accumulation into archives/libraries, which are collections of objects.

[1] Although, naming an assembly file .s is a sure sign of mental instability.

[samiam95124]

And maybe... Nothing

Ok, stay with me. Here are the cases:

1. A module user generates coined module calls to externals.
2. A module is the external module, and has coined module procedures and functions.
3. A module contains external calls, and contains no useful code (its a header).
4. A module contains mixed code and externals.

IF, the module were to contain cexternals, we have established that the call is retrograde. That is, it must preserve all parameters and treat them specially for use to call C modules.

Thus, a module containing externals ATM (At The Moment) is a throwaway module by definition, except for case 4. Thus I would assert that cexternal and external can be effectively equal. Yes, external is used to make it easier to create header files (which for the Pascaline case are modules stripped of their body code). Even in the case of such a header, code is generated, just not useful code, again, with the exception of case 4.

What happens if external becomes a call to C (language modules)?

1. C becomes the official external language of Pascal-P6 (note, not Pascaline, but Pascal-P6).
2. Mixed language modules become possible (there's that case 4). An external call placed in the middle of Pascaline code becomes a call to external C code (effectively a thunk created on the spot).

The thing you have to read carefully in the above is that external is never really retained in Pascal module to module calls. It is only a throwaway as far as Pascaline is concerned.

Effects of external

With the assumption of external as being and instruction to generate C external calls, all of the conversions to C appropriate parameters become active. That is, things like:

1. Automatic conversion of in parameter strings and pstrings to zero terminated C strings.
2. Automatic convertion of out parameter strings to padded strings.
3. Unification of pointers, arrays and even integers.

Good luck explaining that

The remaining issue with that is that it is very hard to explain. "my Pascaline header file contains C calls?". This argues for inclusion of cexternal as an alias for external just to make people feel better. If you follow me at all you would know that such arguments don't impress me much. But it deserves thought[1].

[1] The end purpose of Pascal-P6 as a code generator was always to be effectively married to C (quite unlike IP Pascal). It uses C for its support libraries, the wrappers for C written in it are in C, and this proposal would make it trivial to call C routines in it. The argument is "to embrace C completely" (as the effective low level language of Pascaline).

[samiam95124]

And another way

One proposal is that:

external generates module coining.
cexternal does not.

But both convert parameters to C form. Whats the point?

cexternal both converts parameters to C and allows direct calling of C functions.

external converts to C parameters but DOES NOT allow direct calling of C. Nor does it allow direct calling of Pascaline, since Pascaline cannot use C converted parameters. The use of this is it automates the building of C wrapper functions. These must by necessity go to assembly language. At that time, they can either just get passed on (passthrough), or have further conversion performed on the parameters. They could even go back to C written wrappers as happens now.

Difference between external and cexternal

In this discussion, the difference comes down to coining. Since C can generate coins ("wrapper_routine") this can be handled in C, but it generally indicates that you want to redirect the call to a wrapper function vs. a direct call function. And the external and cexternal directives can be intermixed.