Quickstart

To fit a standard scikit-learn classifier, or any classifier with either .fit or .train and .predict or .test methods, using error_consistency.consistency.ErrorConsistencyKFoldHoldout

.. testcode::

   import numpy as np
   from error_consistency.consistency import ErrorConsistencyKFoldHoldout as ErrorConsistency
   from sklearn.neighbors import KNeighborsClassifier as KNN

   x = np.random.uniform(0, 1, size=[100, 5])
   y = np.random.randint(0, 3, size=[100])
   x_test = np.random.uniform(0, 1, size=[20, 5])
   y_test = np.random.randint(0, 3, size=[20])

   knn_args = dict(n_neighbors=5, n_jobs=1)
   errcon = ErrorConsistency(KNN, x, y, n_splits=5, model_args=knn_args)
   results = errcon.evaluate(
      x_test,
      y_test,
      repetitions=10,
      show_progress=True,
      parallel_reps=True,
      loo_parallel=False,
      turbo=True
   )

   # 10 reps * 5 splits = 50 errors sets
   print(len(results.consistencies))  # 50*(50-1) // 2

.. testoutput::
   :hide:

   1225

To evaluate the error consistency of a set of predictions on a test set with error_consistency.functional.error_consistencies:

.. testcode::

   import numpy as np
   from error_consistency.functional import error_consistencies
   from sklearn.neighbors import KNeighborsClassifier as KNN

   # random training set
   N_TRAIN = 10
   x_trains = [np.random.uniform(0, 1, size=[100, 5]) for _ in range(N_TRAIN)]
   y_trains = [np.random.randint(0, 3, size=[100]) for _ in range(N_TRAIN)]
   x_test = np.random.uniform(0, 1, size=[50, 5])
   y_test = np.random.randint(0, 3, size=[50])
   y_preds = [KNN(5).fit(x, y).predict(x_test) for x, y in zip(x_trains, y_trains)]

   # only grab consistencies and matrix
   consistencies, matrix = error_consistencies(y_preds, y_test)[0:2]
   print(matrix.shape)  # (10, 10)

.. testoutput::
   :hide:

   (10, 10)