Pretraining Algorithms for Natural Language Processing

HKU-TCL Joint Research Centre for AI Workshop - Pretraining Algorithms for Natural Language Processing

Speaker: Dr Lingpeng KONG, Assistant Professor, Department of Computer Science, Faculty of Engineering, HKU

In this talk, Dr Kong will focus on the algorithmic side of pretraining by investigating two potential solutions to the drawbacks of the sLM objective. In the first half of the talk, he will discuss their newly proposed understanding the objective function of pretraining from the mutual information maximization perspective. Their new framework allows them to construct pretraining tasks around more complex views of the sentence (e.g., phrases or semantic parses) beyond solely the word-level. In the second half of the talk, he will focus on how his research team incorporate syntactic bias from a generative model (RNNG) into the pretraining stage by constructing a baseline employing distillation approach.