Problem Regarding Evaluation

[chrisliu298]

Hi,

Thank you for developing this fantastic dataset for code generation!

I had a quick question about the evaluation script (compute_metric.py). In my experience, it takes about 17 hours complete, even with the provided example generation. When I tried switching to parallel evaluation, the process finished much faster (in seconds), but all the results returned as zero. During evaluation, I used the original code an generated solutions provided in this repo.

Is this expected behavior, or am I missing something? I would appreciate any guidance you could provide.

[sssszh]

@chrisliu298 Hi, have you solved this problem? I’ve encountered the same issue.