Skip to content

Ensure disabled tasks return full tuple & fix public‐dataset filter#23

Open
LudovicTuncay wants to merge 2 commits intojimbozhang:mainfrom
LudovicTuncay:fix-scoring
Open

Ensure disabled tasks return full tuple & fix public‐dataset filter#23
LudovicTuncay wants to merge 2 commits intojimbozhang:mainfrom
LudovicTuncay:fix-scoring

Conversation

@LudovicTuncay
Copy link
Contributor

This PR addresses two related bugs in task scoring:

  1. Disabled‐task return shape

    • Previously, when config.disabled == True, worker() returned a 3-tuple
      (formal_name, (0,0), (0,0)). Downstream code always expected a 4-tuple
      (formal_name, mlp_score, knn_score, private_flag), causing an index-out-
      of-range error when building the DataFrame.
    • Fix: Return (formal_name, (0,0), (0,0), config.private) so the shape is
      consistent in all cases.
  2. Public‐dataset filter logic

    • The “public” weighted‐average was filtering on v[-1] == True, but v[-1]
      is actually the private flag. That made “public” averages include only private tasks.
    • Fix: Change the condition to if v[-1] == False so that only non-private (public)
      tasks are aggregated.

Both fixes are split into two commits for clarity. All existing tasks (and any new ones) should now pass without index errors, and the “public” averages reflect the intended data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant