Enabling Broader Learning Through Simplified and Personalized Summaries

Text simplification consists of re-writing a text with an objective of making it accessible to a larger audience of readers, for instance by reducing the length and complexity of sentences, and reducing the use of rare words.

Unlike summarization where progress has been rapid with the emergence of large datasets in many textual domains, text simplification remains challenging, as datasets are small and of poor quality due to the difficulty of aligning texts at different readability levels.

We propose to circumvent the data limitation by approaching text simplification as an unsupervised  task and show that this method is promising both on automated metrics as well as through a user-study testing ease of reading via comprehension questions.

An Example Simplification

Can you tell which is: (1) the original (complex) text, (2) the manual simplification from a Newsela educator (the reference corpus), (3) the output of our unsupervised model.

Spoiler: they are in order, left is the original text, center is the Newsela manual rewrite, and right is the output of the unsupervised model.

Researchers

• Philippe Laban, UC Berkeley, Graduate Student
• Marti Hearst, UC Berkeley, Professor and PI
• Tobias Schnabel, Microsoft Research
• Paul Bennett, Microsoft Research, PI

Overview

We propose to build a reward function that can assess the quality of a simplified paragraph, without relying on training data. Once we establish the reward function, we use a Reinforcement-Learning-based algorithm to train a text generator to optimize this reward.

We craft our reward to consist of three components that must be balanced simultaneously to produce high-quality simplifications:

• A Coverage model: the simplified text must cover the same facts and nuances of the original text.
• A Fluency model: the simplified text must be valid grammatically, and use standard English expressions
• A Simplicity model:the text should be simpler according to a Readability Metric, such as the Flesch-Kincaid measure.

Because all three rewards must be maximized jointly, our final reward is the product of these three metrics.

Automated Results

We train a model (GPT-2 Medium) using our unsupervised method, as well as another using supervision on the Newsela dataset, a popular dataset for news simplification.

We evaluate whether the models are able to lower the expected grade-level required to read 100 held-out paragraphs according to the Lexile measure, a gold-standard used in most school textbooks. We compare it to gold-standard manual re-writes from the Newsela corpus where an educator manually re-wrote a paragraph for a target Lexile number.

User Study Results

We ran a user study with the aim to go beyond automated offline evaluation and measure the potential usefulness of text simplification.

We select several short news articles each re-written by hand and algorithmically. A user reads a single version of the document, and must complete a comprehension quiz  consisting of 5 multiple choice questions.

We measure the average time it takes participants to get all answers right, as well as the average number of re-submissions of the quiz.

Our preliminary, statistically non-significant results suggest that simplified texts tend to lead to faster quiz completion times, and fewer resubmissions on average, with the Unsupervised model leading to a decrease in time spent of roughly 29%.

Contact

• phillab@berkeley.edu

Update August 2021:

• We submitted our work to ACL2021 and it was accepted! Check the full paper here: https://aclanthology.org/2021.acl-long.498/
• All the code and trained models are available on Github: https://github.com/tingofurro/keep_it_simple
• We started a follow-up project focused on detecting inconsistencies in summarization and simplification. We submitted our work, and it is still under review for now. Stay tuned for updates!