In Knowledge distillation: A good teacher is patient and consistent, Beyer et al. investigate various existing setups for performing knowledge distillation and show that all of them lead to ...

Abstract: Deep neural networks (DNNs) have always been a popular base model in many image classification tasks. However, some recent works suggest that there are some man made images will easily lead ...

This repository demonstrates how to train a small language model to perform a specific task by learning from examples generated by a larger, more capable model. The specific task here is generating ...