How Neural Networks Detect and Interpret Wordplay: New Insights from HSE Researchers

© iStock

An international team including researchers from the HSE Faculty of Computer Science has presented KoWit-24, an annotated dataset of 2,700 Russian-language Kommersant news headlines containing wordplay. The dataset enables an assessment of how artificial intelligence detects and interprets wordplay. Experiments with five large language models show that even advanced systems still make mistakes, and that interpreting wordplay is more challenging for them than detecting it. The results were presented at the RANLP conference; the paper is available on Arxiv.org, and the dataset and the code for reproducing the experiments are available on GitHub.

Wordplay refers to deliberate use of language that violates linguistic norms in order to attract attention, entertain, or amuse the reader. It is common in Russian news headlines and can take various forms. For example, the headline ‘Osobo bumazhnye persony’ plays on the phrase ‘Osobo vazhnye persony’ (Russian for ‘very important persons’). The word vazhnye (‘important’) is replaced with bumazhnye (‘paper-related’), which rhymes with the original and shifts the meaning toward the topic of paper production. Another example is ‘Kod naklikal,’ the headline of an article about open-source code. It closely resembles ‘kot naplakal,’ an idiom meaning ‘very little,’ thereby creating a humorous ambiguity. 

For human readers, such wordplay in headlines is immediately apparent and requires no explanation. However, large language models such as ChatGPT or GigaChat Max are often at a loss, struggling not only to detect the wordplay but even more so to explain the joke. One reason for this difficulty is the limited humour datasets on which LLMs are trained. In most cases, humour in these datasets is represented by canned internet jokes explicitly labelled as ‘jokes,’ which is insufficient for the models to learn why something is funny. In addition, such datasets contain almost no annotation—there are no machine- or human-readable layers of description indicating whether wordplay is present, what type of technique is used, what the headline refers to, and so on.

Researchers from the HSE Faculty of Computer Science, in collaboration with colleagues from IT:U—Interdisciplinary Transformation University Austria—and independent researchers, have created KoWit-24, a dataset dedicated to wordplay. It comprises 2,700 headlines from the Russian business daily Kommersant published between January 2021 and December 2023, along with contextual information: each headline is accompanied by a short description of the news story (the lead) and a summary. For each instance of wordplay, the authors manually annotated the type of technique, identified the anchors—the words that trigger the wordplay—and, where possible, linked the original expressions to relevant Wikipedia articles.

The authors adopted linguist Alan Scott Partington’s definition of wordplay, according to which wordplay occurs when the same expression can be interpreted in at least two ways and this effect is intentional. Wordplay can arise in several ways. One case involves ambiguity inherent in a word or its sound. For example, in the headline ‘Volgu ne mogut zastavit’ tech’ bystree,’ the word Volgu (Volga) refers both to the river and to a federal highway with the same name. Another case involves a slight modification of a well-known phrase or title, in which the author alters the wording while relying on the reader to recognise the original and complete the joke. For instance, ‘Missiya sokratima’ alludes to ‘Missiya nevypolnima,’ the Russian title of the film Mission: Impossible, while the headline itself suggests that a diplomatic mission can be downsized.

The researchers also distinguished ‘nonce words’—coined for a single occasion—and oxymorons, which combine two contradictory meanings. This approach not only allowed them to collect and describe examples but also to compare the performance of different language models.

After annotation, the authors tested the dataset on five LLMs: GPT-4o, YandexGPT-4, GigaChat Lite, GigaChat Max, and Mistral NeMo. Each model was provided with a headline and the corresponding news lead and asked to perform two tasks: first, to determine whether the headline contained wordplay, and second, to interpret it by identifying the original phrase or reference. The researchers compared the effects of two types of prompts: a simple prompt asking whether the headline contained wordplay, and an extended prompt providing a definition along with examples of different wordplay types. The extended prompt improved performance on the detection task for three of the five models, while GPT-4o demonstrated the strongest performance in both detection and interpretation. For all models, interpreting the source of the joke proved significantly more difficult than simply detecting the presence of wordplay.

Pavel Braslavski

‘KoWit-24 addresses two key limitations of earlier datasets: it provides context for each headline and includes multi-level annotation. This transforms a collection of examples into a full-fledged “testbed” for AI. It now allows for an objective comparison of models—whether a model can detect wordplay, identify the anchor, and correctly recall the original phrase or reference. Such verifiable metrics not only allow for a more accurate evaluation of current systems but also support their intentional improvement through selection of prompts, training examples, and fact-checking strategies. In the future, we plan to investigate whether this dataset can be used to enhance humour generation,’ says Pavel Braslavski, Associate Professor at the HSE Faculty of Computer Science and co-author of the paper.

In addition, the dataset establishes a common and transparent standard for evaluation, as researchers use the same data and experimental scripts. This reduces variability in the results and helps develop models that better understand natural language, rather than merely following the logical structure of the text.