Reading/Web/Content Discovery Experiments/Simple Article Summaries
|
Simple Article Summaries
|
the Dopamine article on English WikipediaWe would like to provide article summaries, which would simplify the content of the articles. This will make content more readable and accessible, and thus easier to discover and learn from. This part of the project focuses only on displaying the summaries. A future experiment will study ways of editing and adjusting this content.
Previous research and experiments
[edit]Previous research
[edit]Currently, much of the encyclopedic quality content is long-form and thus difficult to parse quickly. In addition, it is written at a reading level much higher than that of the average adult. Projects that simplify content, such as Basque Txikipedia, are designed to address some of these issues. They do this by having editors manually create simpler versions of articles. However, these projects have so far had very limited success - they are only available in a few languages and have been difficult to scale. In addition, they ask editors to rewrite content that they have already written. This can feel very repetitive.
In our previous research (Content Simplification), we have identified two needs:
- The need for readers to quickly get an overview of a given article or page
- The need for this overview to be written in language the reader can understand
Wikimania 2024 session
[edit]At Wikimania 2024, Wikimedians discussed ways that AI/machine-generated remixing of the already created content can be used to make Wikipedia more accessible and easier to learn from. The session focused on ways to give editors control over the structure and content of these new features.
Userlytics experiment
[edit]We ran an experiment presenting article summaries to web readers. We wanted to determine whether article summaries have the potential to increase reader retention, as proxied by clickthrough rate and usage patterns. We were checking whether the proposed concept was clear to the users and whether the feature was usable.
We ran Dopamine. We analyzed videos and screen recordings of the sessions. Participants found simple summaries easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed. See more at Usability study on Simple Summaries.
Browser extension experiment
[edit]The second experiment added the new feature into the pre-existing browser extension. The goal was to determine whether summaries are worth future investment for more costly experimentation. This time, we gathered evidence for the following hypotheses:
- Users are interested in the feature as presented (as measured through initial engagement with summary feature)
- Results: out of all pageviews where summaries were displayed, 8.09% of pageviews had a summary opened. The pages containing the summary received 10,198 impressions and 825 total clicks to open the summary.
- Users report positive experiences with the user experience of the feature (as measured by the answers to the "was this useful?" question)
- Results: 101 responses were received in our survey. 75.2% of all clicks selected "yes", 24.8% of all clicks selected "no".
The above summary is based on data collected from the English Wikipedia (enwiki) desktop platform between December 4 and 17, 2024. Only users who had enabled the extension could view the summary, and not all pages included one. See phab:T381253 for more details
Next steps
[edit]Goals for further experimentation
[edit]Based on the results above, we believe this feature has the potential to significantly benefit readers. We have identified two hypotheses that will help guide and direct this work in the future that can focus both on the technical and moderation aspects of our next steps.
- Evaluate impact of a proposed summary feature on key user metrics on a Wikimedia wiki
- Evaluate community interest and sentiment towards the summary feature, and its range across different language communities
- Explore community involvement in the feature, specifically focusing on the need and approach towards summary moderation
- Explore requirements for improvements and applications of the model used for summary generation
Mobile summary pilot (Hypothesis 1)
[edit]If we introduce a pre-generated summary feature as an opt-in feature on a the mobile site of a production wiki, we will be able to measure a clickthrough rate greater than 4%, ensure no negative effects to session length, pageviews, or internal referrals, and use this data to decide how and if we will further scale the summary feature.
Moderation consultations (Hypothesis 2)
[edit]If we approach summary moderation design collaboratively with communities — through surveys and other on-wiki discussions, we will be able to determine the minimal viable moderation workflow required for initial scaling of the feature and clarify whether moderation should be community-led, automated (at the prompt level), or some combination of both.