Could not parse properly a header file: spectro from Avantes

[seb5g]

Hi Nate,
I'm back with another library I'd like to add to instrumental but again I'm stuck with the header file parsing...
You'll find attached here a zip with the header file and my attempt in _build_avantes. I produced some hooks to clean the header but got some issues during parsing of binary operators. In particular from line 215 where it adds the value of a constant. But the parser considers USER_ID_LEN as an ID constant and not an integer one... Could not figure out the reason.

Some typedefs couldn't be parsed also after that (I tweaked the code to pass the previous issue with USER_ID_LEN but this is ugly).

Please help :-)

_avantes.zip

[natezb]

Can you provide the error traceback? I haven't had a chance to get the code running on my system.

Generally speaking, I think the issue is that this header uses const variables for its constant rather than #define macros like most other headers do.

[seb5g]

Here is the traceback:

:5: unsupported expression: expected a simple numeric constant
:5: unsupported expression: expected a simple numeric constant
cannot parse "SpectrumCalibrationType m_IntensityCalib;"
<cdef source string>:3:3: before: SpectrumCalibrationType
:5: unsupported expression: expected a simple numeric constant
:3: unsupported expression: expected a simple numeric constant
cannot parse "SDCardType m_SDCard;"
<cdef source string>:6:3: before: SDCardType
:3: unsupported expression: expected a simple numeric constant
:5: unsupported expression: expected a simple numeric constant
Traceback (most recent call last):
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\cffi\api.py", line 183, in _typeof
    result = self._parsed_types[cdecl]
KeyError: 'DetectorType'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\cffi\api.py", line 186, in _typeof
    result = self._typeof_locked(cdecl)
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\cffi\api.py", line 171, in _typeof_locked
    type = self._parser.parse_type(cdecl)
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\cffi\cparser.py", line 552, in parse_type
    return self.parse_type_and_quals(cdecl)[0]
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\cffi\cparser.py", line 559, in parse_type_and_quals
    raise CDefError("unknown identifier '%s'" % (exprnode.name,))
cffi.CDefError: unknown identifier 'DetectorType'
python-BaseException

I applied a custom hook:

def rm_dllapi_hook(tokens):
    return remove_pattern(tokens, ['extern', '"C"', '__declspec', '(',  'dllexport', ')'])

because it was having issues with the declaration of the functions using the token DLL_API defined as:

#define DLL_API extern "C" __declspec (dllexport)

Following what you said I tried to create a token_hook to replace const by #define using :

def replace_const(tokens):
    return modify_pattern(tokens, [('d', 'const'), ('a', '#define')])

but it seems the #define is not added ... and the parser then generates another error:

Traceback (most recent call last):
  File "c:\users\weber\labo\programmes python\git_others\nicelib\nicelib\process.py", line 1372, in generate
    chunk_tree = self.parse(csource_chunk)
  File "c:\users\weber\labo\programmes python\git_others\nicelib\nicelib\process.py", line 1486, in parse
    return self.parser.cparser.parse(input=text, lexer=self.parser.clex, debug=0)
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\pycparser\ply\yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\pycparser\ply\yacc.py", line 1118, in parseopt_notrack
    p.callable(pslice)
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\pycparser\c_parser.py", line 564, in p_pp_directive
    self._parse_error('Directives not supported yet',
  File "C:\Users\weber\AppData\Roaming\Python\Python38\site-packages\pycparser\plyparser.py", line 67, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: c:\programdata\avantes\as5216x64-dll_2.3\examples\qtdemos\qtdemo_simple_demo\as5216.h:1170:1: Directives not supported yet
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "c:\users\weber\labo\programmes python\git_others\nicelib\nicelib\process.py", line 1385, in generate
    raise plyparser.ParseError(msg)
pycparser.plyparser.ParseError: c:\programdata\avantes\as5216x64-dll_2.3\examples\qtdemos\qtdemo_simple_demo\as5216.h:1170:1: Directives not supported yet
When parsing chunk:
<<<
#line 1170 "c:\programdata\avantes\as5216x64-dll_2.3\examples\qtdemos\qtdemo_simple_demo\as5216.h"
#
#line 39 "c:\programdata\avantes\as5216x64-dll_2.3\examples\qtdemos\qtdemo_simple_demo\as5216.h"
uint8     USER_ID_LEN             = 64;
>>>
If you a developer wrapping a lib, you may need to clean up the header source using hooks. See the documentation on processing headers for more information.
python-BaseException

I therefore have three questions:

• is it fine how I remove the DLL_API token ?
• how to use the 'a' parameter in the modify_pattern function (as it seems mine didn't work
• should I just replace the const directive by #define ?

thx for the help
Seb

[seb5g]

Hi Nate, could you take some time to have a look at this please?
Regards

[natezb]

I would suggest taking a look at the NiceLib docs, particularly this and following sections.

For this sort of thing, it helps to have an understanding of C and the C preprocessor so I'd highly recommend reading up on them to be more comfortable.

Here's a synopsis of how NiceLib preprocesses header files (as I remember it):

• Load file text
• Use custom NiceLib Lexer to tokenize this text
• Use custom NiceLib Parser to parse and preprocess the token stream in a way meant to emulate the C preprocessor. Among other things, this does macro substitution. One output of the parsing process is a new preprocessed token stream.
• Run token hooks on to modify or augment this token stream
• Group the tokens into minimal "chunks" that are parseable by a C parser
• Use pycparser to parse each chunk in turn
• Apply the ast hooks to each chunk
• Combine all chunks together into one tree and run FFICleaner on it to evaluate numeric expressions among other things
• Use this final tree to generate the "cleaned" header source which is returned

You can examine the source of process.py for more details.

As for your questions:

1. Removing DLL_API should not be a problem. I would remove it
2. There are a few issues here. "#define" is considered to be two tokens by the lexer, so you need to separate them: [... ('a', '#'), ('a', 'define')]. This is not enough though, because a macro #define will not include the type (e.g. uint8), equals sign, or semicolon. You can test and develop your token hooks by playing with the lexer in the REPL, e.g. from nicelib.process import lexer; tokens = lexer.lex('MY TEST STRING'); list(my_token_hook(tokens)). However, I don't believe this approach will work anyway because token hooks are applied after initial preprocessing, meaning the #define may have no effect. (I'm not 100% sure on this one, as CFFI may have some basic macro handling).
3. I'm not 100% sure what you're asking here. Simply replacing const with #define will not work. You can always hand-modify the header rather than using hooks if you're comfortable distributing it. For small headers without existing built-in hooks this can be an easier solution.

I'm not convinced the existing hook system can directly support your use case. I would either manually edit the header file or study process.py to see if there may be a reasonable way to handle what you need.