Large generative pretrained transformers (e.g., GPT-3.5, ChatGPT) have shown surprisingly good performance in language modeling and have been successfully productized in long-form code generation (i.e., Codex for Co-pilot). However, for long-form natural language generation (e.g., writing an email for an office worker), the performance of large language model (LLM) is currently far from perfect because it doesn't have specific domain knowledge and user-specific information that are both critically needed to write appropriate content in the context.

To address the challenge to support the long-form natural language generation feature, we propose to equip the LLM with a novel knowledge patch to allow the LLM to digest necessary knowledge at runtime without parameter tuning. Our proposed knowledge patch is a non-parametric module that can store any knowledge we want LLM to digest runtime. It can be majorly built offline efficiently (e.g., it can be built based on the content of user's previous documents and mails) and easy to maintain (e.g., add new knowledge or remove old knowledge). Like patches to operating systems, our proposed knowledge patch can not only keep the LLM always updated, but also enable the model to become a personalized writing assistant that can really write to meet the user's need. We believe the LLM with our proposed knowledge patch will be the key modeling technology to provide long-form generation.

Natural Language Processing

• Ph.D. students majoring in computer science, electronic engineering, or related areas

• Self-motivated and passionate in research

• Solid coding skills

• Experienced in Natural Language Processing