An old friend: scanner.pas

[samiam95124]

An old friend: scanner.pas

This module, part of pc.pas (in the ./pc directory), has a long history. Its basically the scanner module from IP Pascal, created in about 1985. Its the oldest code in that compiler, which dates to about 1997. It was written long before it became a full Pascal compiler.

In the late 1990s, I was looking to build a facility to translate C header files to Pascaline header files. The first attempt at this was a pattern matching and substitution program, also written in Pascal. It was a fairly large program, and it was a disaster. It never really worked well. The net result of that was my being convinced that only a full C compiler front end could do the job. Thus begin the second version of ch2ph.

The first thing I needed for that was a scanner for C. I wanted to repurpose the IP Pascal scanner module for that. I improved the modularity of it, and put the symbol and reserved word tolkens together with the code to form a scanner. Two more modules completed the C front end set. The first was a macro package. That deserves a bit of explanation. A scanner is a fairly language independent module, and mainly needs a table change and a few other minor modifications. The macro processor for C was originally an independent program called cpp or (C PreProcessor), which still exists, and in fact was used in Pascal-P5[1]. Most compilers after the original one on Unix merged the macro processor with the C compiler front end.

Macro does two things in ch2ph. It performs substitutions, but it also looks at the macros being defined and evaluates if the definition contained can be reduced to simple constant declarations. The reason this is necessary is that in C, define statements are often used to create constants like:

#define GUIDE_TO_GALAXY 42

Since the goal is to create Pascal headers from C headers, any macro defined constant that can be a simple constant can also be part of a C header defintion.

Above the macro module, exists the parser module. The C parser is what I call a "follower", that is, it simply follows the syntax of a C program and checks it for correctness. At places like variable declarations or function declarations, it upcalls ch2ph program functions to allow it to emit Pascal functions definitions that mimic the C header definitions.

The C parse code still exists, in the IP Pascal repo.

Swiss army scanner

The scanner module from that project was moved to pc, A scanner thus can be a universal tool, used to create beautifier tools, cross referencers, etc. And oh, by the way, it can be used in an automated build system like pc. Its used by pc to tour the source code, looking for uses/joins statements, module types, determine if the module has code in it, etc.

Modernizing the scanner

The pc version of the scanner module was looking a bit dated when I overhauled pc for Pascal-P6 use. The hash tables it used were automatically generated, but the output from that had to be cut and pasted into the scanner.pas module. This didn't need to be, and I converted the system to generate full modules containing each table and the definitions they contained. Thus they could just be used by scanner without any cut and paste.

Onward and upward

I have often thought of using scanner as a base for a prettyprinter. Yes, prettyprinters exist for Pascal, and I have tried them. They... how shall I put this... suck. I think its that same pattern matching issue mentioned above. The program has to understand the structure of it's source language to work properly, not just look for keywords. Beyond that is, of course, a cross referencing program, the modern equivalent of which exist in programs like doxygen.

And ch2ph?

ch2ph was fairly IP Pascal specific, but is certainly on the radar for adaption to Pascal-P6. In fact, the research I did to create ch2ph convinced me that a c2p facillity is possible. Yes, you read that correctly, a C language to Pascal language converter. Obviously there are caveats there. I'll be writing more about that in the future.

[1] No I didn't write it, it was part of the original Unix tooling set.