Store model info + feature weights in s3 – or database

[ybressler]

All model info & model meta-data should be stored either in an s3 or a database.

[ybressler]

@rashnil-git, here's an example of model info being generated each time the model is re-trained:

{
  'model_name': 'RandomForestClassifier',
  'model_training_info': {
    'n_samples_train': 20,
    'n_samples_test': 7,
    'SMOTE': True
  },
  'model_metrics': {
    'accuracy': 0.7142857142857143,
    'balanced_accuracy': 0.75,
    'log_loss': 9.868221827117338,
    'precision_score': 1.0,
    'recall_score': 0.5,
    'f1_score': 0.6666666666666666,
    'granular_prediction_metrics': {
      'TP': 2,
      'TN': 3,
      'FP': 0,
      'FN': 2
    },
    'pred_spread_of_test_data': [0.050345191401305944, 0.06316810541539636, 0.0004412144013537511, 0.09368154875740009, 0.1517320695730138, 0.03541553527312033, 0.12563187182614421]
  },
  'feature_importance': {
    'surveyid': 0.0,
    'diagnosis_tested': 0.0,
    'symptoms_cough': 0.03,
    'symptoms_body_pain': 0.08,
    'symptoms_headache': 0.04,
    'symptoms_sore_throat': 0.04,
    'symptoms_fever': 0.04,
    'symptoms_runny_nose': 0.06,
    'symptoms_sinus_pressure': 0.01,
    'symptoms_diarrhea': 0.05,
    'symptoms_chills': 0.05,
    'symptoms_fatigue': 0.12,
    'symptoms_reduced_smell_taste': 0.06,
    'symptoms_asthma': 0.01,
    'symptoms_rash': 0.01,
    'symptoms_shortness_breath': 0.03,
    'symptoms_sneezing': 0.0,
    'symptoms_chest_pain': 0.08,
    'symptoms_nausea': 0.0,
    'age_11-30': 0.05,
    'age_31-50': 0.09,
    'age_51-70': 0.02,
    'age_54': 0.0,
    'age_nan': 0.0,
    'country_Canada': 0.03,
    'country_United States of America': 0.02,
    'country_nan': 0.0,
    'gender_Female': 0.0,
    'gender_female': 0.06,
    'gender_male': 0.02,
    'gender_nan': 0.0,
    'reportsource_report_diagnosis': 0.0,
    'reportsource_nan': 0.0
  }
}

How / where do you suggest we store these metrics? A relational db would work – I could set something up a MySQL db in RDS. (Unless you want to keep the databases uniform in DynamoDb.)

[rashnil-git]

Let's use the same database so it is easy to maintain. We can create a new Table in dynamo BD and store this json payload.

Let's generate a model ID (can be a hash value) when we are training and building a new model along with model creation date and other metadata so that they can be stored in dynamo db as keys for the json payload.

[ybressler]

Okay! Can you share a link to documentation so I can read up on it?

[rashnil-git]

I will create an API that will store this. I will let you know when its ready. I will share the API Keys. You will need to send a POST request to this new api end point and it will store in dynamo DB. Please share the revised schema which has modelID and creation timestamp.