Distillation/Pruning of BERT LM for faster response times

[vdpappu]

No description provided.

[vdpappu]

@shivamakhauri04 please post the current updates

[shivamakhauri04]

Have written 60-70 percent of the training pipeline. Since there are certain uncertainties with the network architecture, I am going forward with setting the parent network to be a bert classifier network, as the only few kinds of literature available on ml distillation suggest the same.. Hence, I am taking the parent network to be the nsp context window classifier I had developed earlier (in the process, will see if I can do some optimization there too). Will train on a general dataset for validating the sanity of the pipeline, once fully set.