Controllable Generation from Pre-trained Language Models via Inverse Prompting (Paper Summary)

#nlp #languagemodel #transformers
Open-ended Text Generation systems usually suffer from problems like Low relevance and out-of-context generations. This paper proposes an easy, intuitive yet effective method for Controllable Text Generation using Pre-trained Transformers Language Models.

⏩ Abstract: Large-scale pre-trained language models have demonstrated strong capabilities of generating realistic text. However, it remains challenging to control the generation results. Previous approaches such as prompting are far from sufficient, which limits the usage of language models. To tackle this challenge, we propose an innovative method, inverse prompting, to better control text generation. The core idea of inverse prompting is to use generated text to inversely predict the prompt during beam search, which enhances the relevance between the prompt and the generated text and provides better controllability. Empirically, we pre-train a large-scale Chinese language model to perform a systematic study using human evaluation on the tasks of open-domain poem generation and open-domain long-form question answering. Our results show that our proposed method substantially outperforms the baselines and that our generation quality is close to human performance on some of the tasks.

Please feel free to share out the content and subscribe to my channel :)

⏩ OUTLINE:
0:00 - Abstract and Background
02:24 - Baseline: Prompting and Beam Search
04:12 - Inverse Prompting
05:45 - Examples for Inverse Prompting
07:19 - Implementation
07:41 - Open-domain Long-form Question Answering
09:15 - Open-domain Poem Generation
09:50 - Self Training for Poem Generation

⏩ Paper Title: Controllable Generation from Pre-trained Language Models via Inverse Prompting
⏩ Author: Xu Zou, Da Yin, Qingyang Zhong, Ming Ding, Zhilin Yang, Jie Tang
⏩ Organisation: Department of Computer Science and Technology, Tsinghua University, Beijing Academy of Artificial Intelligence, Recurrent AI Ltd.

**********************************************
If you want to support me financially which is totally optional and voluntary ❤️

**********************************************
*********************************************

Tools I use for making videos :)

#techviz #datascienceguy #nlproc #research #machinelearning
About Me:
I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3 years in the field of Data Science and Machine Learning with a particular focus on Natural Language Processing (NLP).