At the same time, the philosophy communities (except for a few notable philosophical hamlets in Munich, Amsterdam, or Oxbridge) are falling behind as they struggle to come to terms with mostly technical research for which they have little training. For this reason, I’ll try to list a few hot topics from 2026 that I find particularly promising – to help philosophers on the one hand catch up with some of the buzz, and AI and ML researchers and practitioners on the other hand enrich their work with perspectives from philosophy. The following list is by far incomplete and should only serve to create some appetite in both camps for an increased dialogue. I hope some of my ideas can inspire the readership. They are on purpose often rather broadly defined and less focused on narrow topics of philosophy of machine learning, which exists already but is often stuck in statistical theory of outdated ML methods (sorry!).

Autoresearch

Andrej Karpathy has published a new project on GitHub, which he has termed Autoresearch. Within the last few weeks, this project has received more than 70k stars and has become one of the most popular GitHub contributions. In a nutshell, it’s a framework guiding an agent to constantly improve the code of a research project by applying smart changes and verifying their merit through fixed quality criteria. Changes are kept when they improve the quality. This is a form of local search (also called hill climbing) that the agent performs: it climbs within the landscape of the chosen criteria to higher and, in that, better metric values. A broader approach to AI in research has been proposed by Sakana’s AI Scientist: they implement an agent that traverses a tree in search not only of better code, but also of an idea as well as different experiments, to arrive at a complete scientific paper. Similarly, other strategies explore the usage of AI in science through global search methods: consider AlphaEvolve or ShinkaEvolve, which harness evolutionary approaches (i.e., the generation of multiple candidate solutions that are iteratively sampled, crossed, and re-evaluated) to make foundation models come up with improved algorithms.

These methods have been developed and demonstrably used to enhance scientific discovery. They work particularly well when verification mechanisms exist, and they are part of the reason why fields like mathematics will be vulnerable to AI automation, as they further enhance foundation models’ abilities to find ways (e.g., proofs) toward a verifiable solution. This is a trend that should not only excite or worry mathematicians like Daniel Litt, who has recently revised his view on AI’s ability to automate math, but also philosophers (of science).

Empirically, the AI autoresearch methods will not only fundamentally change scientific practices and, with that, assumptions about how scientific communities work (I’m thinking here of Thomas Kuhn and the sociology of science, which would need a big update in a few years), but could also be informed by many of the insights that the philosophical community has gathered about science. To limit this excursion to one example: I believe that the ML community in general has often little clue about what it actually would like to investigate. Does it formulate and check hypotheses about the world, AI, the model, what the model is supposed to model (e.g., language, reasoning, intelligence), or something else? Similarly, the AI co-scientists and agents that will now populate our labs’ repositories are largely confused if they don’t receive proper instructions and guidance. For this reason, I can imagine that ML and AI researchers could profit from philosophical theories about what properties and forms a scientific model can have. If, likewise, the AI scientists and agents knew what they are supposed to discover as well as to model in more detail, they would surely be better off. Some might wonder why I make this point, as it seems rather far-fetched since we might just ask the agent to improve a given algorithm, but I’m trying to think here in more long-term perspectives and in terms of more ambitious approaches such as Sakana’s AI Scientist. Just consider how rich the list of possible scientific models from Stanford’s Encyclopedia of Philosophy is: “Probing models, phenomenological models, computational models, developmental models, explanatory models, impoverished models, testing models, idealized models, theoretical models, scale models, heuristic models, caricature models, exploratory models, didactic models, fantasy models, minimal models, toy models, imaginary models, mathematical models, mechanistic models, substitute models, iconic models, formal models, analogue models.” Can this not somehow be better incorporated into what a scientific agent is supposed to achieve in the future?

Neural Computers

With the increasing power of foundation models, more ideas for their applications emerge. One of these ideas is to let neural networks fully simulate the computer as we know it. Mingchen Zhuge et al., under the guidance of Jürgen Schmidhuber, have just published a new paper that experiments with multimodal models to replicate a simple command-line terminal as well as more complex GUIs. Everything the user does in terms of clicking or writing is forwarded as input (i.e., tokens) to the model right away. While initial experiments are still brittle, the authors have outlined a few criteria that the neural computer must ideally meet. One of them is, of course, Turing completeness: the power to express and run computations. In some ironic way, the authors conceptualize a neural computer that starts as a neural tabula rasa only to learn the automata and rules that turn a machine into a computer.

This is much harder than it might appear at first glance. Just think of the difficulties that LLMs have with counting. And it is also here where I believe that researchers on neural computers could profit from philosophy in the future. My intuition is that a neural computer would need to begin by learning a few fundamental principles before it could be used as a user-input-and-model-output terminal or GUI. Consider François Chollet’s ARC-AGI considerations, which I have detailed in a different blog entry. Inspired by cognitive scientists Spelke and Kinzle, Chollet argues that an intelligent system would most likely be bootstrapped from four core cognitive systems, one of which is the system that allows the intelligence to grasp numbers and ordering principles. Basically, half of Wittgenstein’s Philosophical Investigations is about the puzzling difficulty of understanding how a child learns to count, and with that, how rule following more generally can be taught. My hope would be that research on neural computers could be inspired more by these fundamental philosophical conundrums, for which different theories have also emerged. It remains an open question whether everything can simply be learned empirically, i.e., whether neural nets would just need enough data to become Turing machines (this is in some sense the topic of Cameron Buckner’s From Deep Learning to Rational Machines: What the History of Philosophy Can Teach Us about the Future of Artificial Intelligence; although several chapters become rather hand-wavy later in the book), but research in AI is hitting these hard walls more often than ever before – walls which we have encountered before in philosophy.

AI Safety: Agents in the Wild

A new workshop at ICLR is called Agents in the Wild, which addresses the increasing safety needs for AI agents that become more and more powerful, as they will not only improve our code but also execute their own decisions with real-world impacts. While the trolley dilemma from autonomous driving is a dead horse that I don’t want to beat further, the demand for secure deployments is bigger than ever. These agents will come, and we will need different strategies to mitigate their risks. I believe some of these strategies will be technical, others will be normative. In this area, philosophers from AI ethics have already made important contributions, and I think it is the field where philosophy has so far had its greatest point of contact with AI. However, many philosophers are still shy to engage in true engineering. Agents would need a particular ethical worldview that could steer their behavior. Constitutional AI approaches like the one from Anthropic have been driven by non-philosophers, yet it is these kinds of approaches that are used to guide Claude. ML and AI engineers should demand more from the philosophical research community than abstract meta-discussions about different ethical perspectives or cultural analyses of how AI is perceived in society.

Multimodal Reasoning Continued

Reasoning models will increasingly engage with more than text and logical problems and focus more on the physical world. This will require different forms of reasoning, and training models to better navigate the world through mostly imagery is already happening. But can neural networks truly understand something like colors, which are for us a fundamental property of our lived world and an important signal, in some sense, to navigate the world, too? I have repeatedly thought about qualia when I was engaging with multimodal reasoning. Qualia are at the heart of the mind-body problem. They refer to subjective experiences of colors or feelings. It is open whether they fully reside in the mind alone or are also part of the external world. The most famous thought experiment on qualia has been proposed by Frank Jackson: he invites us to imagine Mary, the color scientist, who has been confined her whole life to a black-and-white space. However, she’s gathered all the knowledge about colors from physics, biology, and other sources of information through black-and-white books and TV. In this fashion, Mary has become an absolute color authority. Eventually, Mary is let free, and for the first time she sees red and green colors in a garden full of flowers. Suddenly, Mary has experienced colors in some other way; she’s learned something new about the world. How could that be? This is what qualia are about. Mary the color scientist, confined in her black-and-white world with access to all the knowledge about colors, sounds a lot like an LLM to me. In AI, we should therefore wonder what it is like when a model is further fed with different modalities, which are currently vectorized through image tokens, and what effects that could have on the model. Again, I think there are a few lessons that could be learned from philosophy.

Counterfactual Reasoning

Counterfactual reasoning is particularly important in moral reasoning. It is the ability to imagine different possible worlds and make decisions considering alternative scenarios. While LLMs output text that sometimes seems to engage in such reasoning, I haven’t seen much research in AI in this direction so far. However, I believe that builders of AI who want to create more capable moral systems would need to further incentivize counterfactual reasoning in foundation models. In fact, counterfactual reasoning is also important in order to learn what the difference between customs and norms is, or which ethical core principles would hold across all possible worlds. Basically, it is one of the most complex, creative, and distinctive forms of cognition that humans excel at. The topic is further entwined with debates around rule following and rules more generally, something I’ve already mentioned with regard to learning how to count. I believe counterfactual reasoning is a field in philosophy that is worth understanding, and one day we’ll see new rounds of reasoning models that are tuned on this.

Mechanistic Interpretability Continued

Mechanistic interpretability has become an important strain in ML research, where different methods have been developed to shed light on Transformer models in particular. However, from a philosophical perspective, the field still seems rather chaotic, because rarely do most of the authors understand the difference between explanandum, explanans, and explanatory links, nor do they have a more focused agenda about what they would actually like to explain and interpret. These points have been made in Dear XAI Community, We Need to Talk!. Why mechanistic interpretability needs philosophy has been further pointed out by Iwan Williams et al.. The authors argue that “mechanistic interpretability needs philosophy: not as an afterthought, but as an ongoing partner in clarifying its concepts, refining its methods, and assessing the epistemic and ethical stakes of interpreting AI systems.” For example, mechanistic interpretability has often focused on one aspect, one level of mechanism, to explain a phenomenon; however, philosophers of science have shown that systems rarely have a unique and natural order of decomposition. In practice, what level of mechanism should be used and how their entanglement is interpreted would often depend on the pragmatic goals of the research agenda.

Not Hot Yet: AI Sentience & Memory and Identity

Finally, I’d like to briefly mention two more topics which are not hot yet but where I see great potential for an increasing cross-disciplinary engagement. The first is AI sentience, that is, the investigation of to what degree a so-far non-living entity can eventually develop sentience, and whether we ever have the chance of making this observation in the first place. I’ve seen little research in this direction. The best and deepest consideration of this topic that I know has not been presented by a trained philosopher, but by a computer scientist who has now rather become a philosopher: Berkeley’s Edward Lee. I can highly recommend watching his lecture Will Embodied AI Become Sentient? where you will learn about zero-knowledge proofs along the way (fun!). In short, Lee’s answer is that we’d have trouble ever knowing about sentience in another being (no surprise for the philosophically literate readers), but he’s found an intriguing way around the epistemic scepticism to make him more hopeful that we could still have something of a proof for sentience (with zero knowledge!). Finally, another hot topic in AI is the development of systems with better memories, since the vanilla neural nets that we’ve dealt with so far have no memory module as such. Ultimately, I can imagine that a memory would not only usher in new cognitive abilities (it is linked to consciousness) but might also be the foundation for something like an identity. The intrigued ML researcher is hereby invited to read Derek Parfit’s Reasons and Persons (Parfit is the most famous philosopher who has dealt with the question of how we entertain an identity over time).

Why choose this approach? With a small investment, you can work directly with core LLM libraries and experiment with open-source models. For example, I own a MacBookAir; however, essential libraries such as unsloth don’t support Apple chips, nor are many other libraries optimized for them. In my experience, Google Colab is less reliable when it comes to GPU availability (even for paid options). It’s also limited to notebooks, which restricts flexibility. Using a remote instance with GPUs is also helpful if you want to prepare your scripts before submitting heavy jobs on a high performance cluster. Finally, setting this up is a fun way to gain some hands-on ML Ops experience.

Create account on vast.ai and add credit

After having created an account on vast.ai, charge your credit with 5€-10€.

Fig. 1: Account page on vast.ai

Rent instance

You can rent an instance immediately by clicking on rent. There are a few technical details to keep in mind to ensure the model runs smoothly. For this example, I want to inference from and fine-tune a Qwen/Qwen2.5-7B-Instruct model, using unsloth. unsloth is a convenient Python library for fine-tuning models; it serves as a wrapper around the PEFT (parameter efficient fine-tuning) package, which is part of the PyTorch and HuggingFace universe.

The instance from vast.ai should therefore have an NVIDIA GPU and a fitting Linux distribution.

I recommend the following specifications:

• GPU: RTX 30XX, RTX A40XX or other RTX; if more compute is needed an A100. The GPUs should have more than >24GB (fine-tuning a 7B model is possible with unsloth; the transformers library would rather require 80GB for the same task)
• RAM: Usually 2xVRAM is best, but at least 16GB should suffice
• HDD: 50-100GB
• CPU: whatever comes with the instance
• Bandwith: At least 100 Mb/s, since you’ll need to download large libraries and models.

Each instance can use a different template. For LLM experiments, it’s easiest to start with the PyTorch template, which comes with CUDA, cuDNN, and many essential libraries pre-installed. This template runs inside an unprivileged Docker container.

Fig. 2: Template on vast.ai

A few tips for selecting the right instance:

• Check that the instance’s resources (CPUs, RAM) aren’t already heavily used. Some servers share hardware, and it’s possible to find instances on vast.ai where CPU cores are busy.
• I’ve had problems with several instances (e.g., they might come with an older CUDA version); just delete them and search for a different one.
• As soon as the instance is being stopped, the system continues to exist and vast.ai will charge some money everyday for storage costs (not much though).
• Once you stop an instance, someone else can rent its GPU. You won’t be able to restart your instance until the other user finishes, which can take hours or even days.

Connect to your instance

You’ll connect to the instance via ssh. The example below uses macOS with zsh, but the setup is similar on other systems — mostly the file paths differ.

Open your terminal and go to your hidden .ssh directory:

cd ~/.ssh

If the directory doesn’t exist, create it in your home folder and set the correct permissions:

mkdir -p ~/.ssh
chmod 700 ~/.ssh

Next we want to create a key, which we will call “vast_ai”, by running:

ssh-keygen -t ed25519 -f ~/.ssh/vast_ai -C "Name or E-Mail address"

from /.ssh.

You’ll be asked to assign a password.

The key pair consists of a public and a private file. We want to open:

cat vast_ai.pub

and copy the publicly accessible content. We insert the content on our vast.ai account page and save it.

Fig. 3: Adding ssh key to vast.ai

Next, we navigate to our instance. I’m using the following for this example:

Fig. 4: Instance on vast.ai

If we click on the small key symbol (our ssh key should appear, if not we add the ssh key here again and click on ADD SSH KEY) then the section direct ssh connect appears. vast.ai will show us which port we can use to connect to the instance.

Copy the provided SSH command from direct ssh connect, add your key name with the flag -i, and run it in your terminal:

ssh -i ~/.ssh/vast_ai -p 10532 root@194.26.196.132 -L 8080:localhost:8080

Enter the password you set for the key, and you’re in! vast.ai will automatically start a tmux session for our instance.

Check GPU availability

At this point, we can check if the GPU is accessible with

nvidia-smi

We should see something like the following:

Fig. 5: nvidia-smi showing the GPU

If not, for example, if there’s an issue with the driver, the easiest way might be just to delete the instance and look for another one.

Activate mamba environment

Luckily, the instance comes with condaand mamba pre-installed. For this reason, we can simply run the following to create a mamba environment which we can call “test_env”:

mamba create -n test_env python=3.12

After the installation, initialize the shell for the environment:

eval "$(mamba shell hook --shell bash)"

will do it and then we can activate it:

mamba activate test_env

Connect to your instance with VS Code

To make life much easier, we can also connect to our instance via VS Code. To do this, open your local terminal, create an .ssh/config file if it doesn’t already exist, and set the correct permissions.

cd ~/.ssh
touch config
chmod 700 config

We’ll have to add (e.g., with nano) the following information, depending on the instance respectively:

Host vast_ai_instance
    HostName <IP-of-instance>
    IdentityFile ~/.ssh/vast_ai    
    Port <Port-of-instance>
    User root
    LocalForward 8080 localhost:8080

Next, we can open VS Code and try to connect to the remote host. The name from the config should appear as an option, when we click Connect to Host....

Fig. 6: ssh to remote host with VS Code

Run python scripts

At this point, we can use VS Code to create new python scripts on our instance. If we go back to our instance terminal, we can install the following packages:

pip install --upgrade unsloth transformers accelerate bitsandbytes

With these libraries installed, we can write a simple Python script (e.g., test.py) that automatically downloads an LLM and runs an inference on a prompt.

from unsloth import FastLanguageModel

model_name = "Qwen/Qwen2.5-7B-Instruct"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = model_name,
    load_in_4bit = True, # Load in 4-bit precision to save memory (optional)
    device_map = "auto", # Auto place model on GPU if available
)

prompt = "Explain what a high performance cluster is."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)


print("Generating response...")
outputs = model.generate(
    **inputs,
    max_new_tokens = 200,
    temperature = 0.7,
    do_sample = True,
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("\n=== Model Response ===")
print(response)

Simply, run the script in your mamba environment with:

python3 test.py

In my case, the model was saved under /workspace/.hf_home/hub from where we can also remove it, if we want to make space.

Fine-tune model

From here on, you can fine-tune your model with unsloth, using one of their many notebooks as an inspiration: https://docs.unsloth.ai/get-started/unsloth-notebooks

You may also connect to your git repository to save code or data for later use.

1 Introduction

Artificial intelligence has made remarkable progress in domains such as language understanding or pattern recognition. Yet, the quest to develop systems that demonstrate the kind of flexible, general reasoning observed in humans remains unresolved. The Abstract Reasoning Corpus (ARC-AGI) [11] was designed to probe this gap, presenting tasks that resist memorization and instead reward adaptive reasoning and generalization. These puzzles, simple for most humans yet challenging for current AI models, provide a lens through which to examine whether AI can capture the cognitive priors underlying human intelligence.

This article investigates how different reasoning paradigms – symbolic program synthesis, transductive neural predictions, and multimodal extensions – can be combined to tackle the ARC-AGI challenge. Recent advances have demonstrated the complementarity of inductive and transductive approaches [28], suggesting that no single style of reasoning suffices for the diversity of problems embodied in ARC. Building on these insights, I argue that expanding beyond purely linguistic reasoning toward multimodal architectures may prove critical since ARC puzzles are inherently visual.

2 The Abstract Reasoning Corpus (ARC) challenge and related work

This section presents the ARC-AGI challenge and discusses leading approaches to solving it through a combination of program synthesis and transductive predictions.

2.1 The Abstract Reasoning Corpus (ARC) challenge

The ARC challenge, introduced by Chollet [11], is designed as a diagnostic tool to evaluate the gap between human and artificial intelligence. It consists of independent tasks that test core cognitive abilities considered fundamental to human intelligence. These tasks draw on the work of developmental psychologists Spelke and Kinzler [44], who propose that human cognition is grounded in four basic systems that allow: (a) discerning objects and their mechanics, (b) understanding goal-directed agents, (c) grasping numbers and ordering principles, and (d) navigating spatial and geometrical environments. These systems, also described as core knowledge [44], enable us to perform fundamental tasks in the world. If an AI model succeeds on ARC tasks that require knowledge or skills derived from one or more of these systems, it would provide evidence that the model may share similar underlying cognitive structures with humans.

The ARC challenge (later renamed ARC-AGI-1 [12]) has translated these ideas about core knowledge into 2D visual puzzles (see Figure 1). The benchmark comprises 1,000 of them where each has a minimum of two demonstration pairs with a test input that is supposed to be solved based on the demonstration. In other words, the ARC-AGI-1 is a few-shot benchmark. Pairs are made up of two elements: an input grid and an output grid. The input grid is a rectangle that can vary in size, but never larger than 30 rows by 30 columns, with each cell taking on one of ten possible colors. The output grid is expected to be completely deducible on the basis of the properties of the input grid. Two guesses are allowed to pass each ARC test.

Figure 1: Example of an ARC 2D puzzle with demonstration pairs and another test input to be solved. The figure is from [12].

Most importantly, the benchmark is designed to resist memorization and instead reward strong generalization skills. Each task is unique and follows its own logic, yet still allows participants to infer the underlying procedure needed to solve it. To limit brute-force learning, ARC-AGI-1 is kept deliberately small. It consists of 1,000 tasks divided into four splits: (1) a public training set of 400 examples, (2) a public evaluation set of 400 similar but more challenging tasks, (3) a semi-private evaluation set of 100 tasks, and (4) a fully private evaluation set of 100 tasks to prevent leakage. The challenge also imposes computational constraints: competitive submissions must run in a single Kaggle notebook, with a maximum of 12 hours on one P100 GPU.

According to Chollet [11], ARC-AGI-1 puzzles are generally easy for humans. Nevertheless, LeGris et al. [27] report that, on average, humans solve only 61.6% of the tasks in the public evaluation set, with the best individuals reaching 97.8% accuracy. By comparison, the winning approach [18] of the ARC Prize from 2024 achieved 71.6% accuracy on the public evaluation set and 53.5% on ARC tests from the private evaluation set. This raises the question whether we have already achieved AI that masters core knowledge as well as humans. I will discuss it with respect to the new and upcoming benchmarks ARC-AGI-2 and ARC-AGI-3 in section 4.

The latest iteration of the ARC-AGI-1 challenge, in the form of the ARC Prize, from 2024, has been accompanied by several promising developments [12]: (a) large language model (LLM) based program synthesis to generate programs solving the ARC tasks [19, 28], (b) test-time training (TTT) for transductive models [3, 18], readjusting the model’s weights on the spot before making predictions, and (c) combining program synthesis with transductive models [28].

2.2 Combining induction and transduction for abstract reasoning

The winning paper from Li et al. [28] of the ARC Prize 2024 combines program synthesis (which the authors call ’induction’) with a transductive model using TTT in one ensemble. For induction as well as transduction, Li et al. [28] convert the ARC tasks into strings using one token per grid cell with newlines to indicate the end of rows (consider Figure 2). Both approaches were realized by fine-tuning a Llama 8B-Instruct model [14] on an enlarged dataset of ARC-like tasks (400k examples). Li et al. [28] generated this dataset with GPT4o-mini [24] based on natural language descriptions and manually written seed programs for 100 ARC-AGI-1 tasks. Li et al. [28] further collected training examples from sources such as Hodel [22].

The synthesis of the programs is achieved by letting Llama-8B-Instruct generate Python programs that can map an ARC input to the desired output solution. In contrast to previous synthetic approaches for ARC [9, 2], Li et al. [28] do not create a domain-specific language (DSL) for this purpose, but stick to Python. During the fine-tuning of the LLM, generated programs are compared with correct ones from their synthetic ARC dataset, where they added programs to each example.

The transductive Llama-8B-Instruct model is also fine-tuned to directly generate solution sequences for ARC tasks. To improve prediction, Li et al. [28] apply TTT. Following the approach of Sun et al. [45], this strategy introduces an additional self-supervised training step before each prediction. Specifically, the input is augmented (e.g., through rotations or color permutations), and the model is tasked with recovering the original version. If it fails, its weights are updated accordingly. Interestingly, this process not only improves the model’s ability to handle such reversions, but also encourages it to learn auxiliary features that transfer to the downstream task – in this case, producing the correct output for a given ARC test input.

By combining program synthesis with transduction via TTT and applying reranking to predict the most likely output across multiple augmentations, Li et al. [28] report a 56.75% accuracy for their ensemble approach on the public ARC-AGI-1 evaluation set. One of their key findings is that induction and transduction “are strongly complementary, even when trained on the same problems” [28]. The authors stipulate that inductive program synthesis might resemble a deliberate approach and transduction an intuitive, where both leverage different forms of reasoning, comparable to system-2 and system-1 reasoning as introduced by Kahneman [25]. Their finding suggests that even within the ARC’s core knowledge tasks the problems are of such different nature that they also require different forms of reasoning.

Although the results reported by Li et al. [28] are impressive, their approach relies heavily on large amounts of synthetic data designed to mimic ARC-AGI-1, raising concerns about potential overfitting. In addition, the use of an ensemble introduces operational complexity by requiring the coordination of two separate models. A unified architecture would ideally mitigate these issues by reducing fragmentation and simplifying deployment.

3 Motivating multimodal reasoning to solve ARC-AGI

The complementary nature of inductive and transductive reasoning underscores the diversity of reasoning strategies that can be deployed for tasks like ARC. Some ARC tasks can be effectively solved with deterministic, symbolic algorithms (i.e., Python programs), whereas others are more amenable to transductive predictions (i.e., patterns arising from neural activations). This points to the value of integrating two distinct reasoning paradigms: symbolic and neural approaches to AI. Yet, I propose broadening this perspective even further: reasoning may manifest in additional forms across different modalities, such as vision or other sensory perception. This consideration is particularly relevant because the ARC-AGI-1 puzzles are presented in a two-dimensional (2D) visual format.

Therefore, this section emphasizes that reasoning does not necessarily depend on language. It also examines the difference between 2D and 1D representations of ARC puzzles and how these may affect the performance of LLMs. Finally, it introduces multimodal large language models (MLLMs) as potential models for tackling ARC challenges.

3.1 Evidence for non-linguistic reasoning

Do we think in language? Fedorenko and Varley [15] review evidence from cognitive neuroscience addressing this long-standing question, drawing on findings from functional magnetic resonance imaging (fMRI) studies and from research on individuals with global aphasia. They argue that reasoning in domains such as arithmetic, music processing, spatial navigation, executive functions, and theory of mind does not depend on language. Specifically, fMRI evidence shows that brain regions involved in linguistic processing are not necessarily active when humans engage in arithmetic, logical, or functional reasoning, process music, or recognize others’ internal states. Furthermore, people with severe damage to language-related brain regions can still perform tasks in these domains. In sum, Fedorenko and Varley [15] conclude that various forms of thought and reasoning are possible without relying on language or the cognitive brain areas underlying language processing.

Ainooson et al. [2] compare the ARC-AGI-1 challenge with Raven’s Progressive Matrices (RPM) which is an intelligence test that touches on similar abilities and tasks as ARC. Studies with neurodivergent individuals who excel in visual imagery showed that these individuals scored differently – and often more strongly – on the test [43]. This led to the assumption that the tests can be solved with mental representations that are not necessarily linguistic, but retinotopic, that is, visually grounded [37]. Further imagery-based methods were developed that successfully helped solving the RPM tests with visual reasoning alone [31]. These experiments provide evidence that reasoning as well as reasoning about core knowledge tasks can be language independent.

In other words, the evidence from cognitive science and psychology suggest that thinking and reasoning can be performed in different modal domains. As humans can reason visually without language, ARC solvers may need non-linguistic pathways as well. This does not mean that reasoning is entirely non-linguistic, since much of our thinking is indeed processed through language – as is evident from inner monologues and the many textual representations of reasoning problems. Even so, it is important not to conflate the content of reasoning with the underlying cognitive processes.

Theories of mind debate whether mental states have linguistic properties. This idea is most prominently proposed by the language of thought hypothesis from Fodor [16] and Fodor and Pylyshyn [17]. According to this hypothesis, mental states are “symbolic structures, which typically have semantically evaluable constituents, and […] mental processes are rule-governed manipulations of them that are sensitive to their constituent structure” [38]. In other words, if we truly ’think in language’ our thought processes would be governed by a syntax-like structure, and the elements of our thought would be semantic-like objects. Opposing views, for example, from the Connectionist strain [30, 41, 42], argue that mental states originate from activation patterns – also referred to as ’nodes’ – within a neural network. Mental processes spread these patterns. However, the ’nodes’ have no semantic characteristics; for example, they do not refer to entities in this world. Activations are neither governed by strict grammar-like rules.

In summary, reasoning can take different forms, such as transductive or algorithmic, and can deal with various modalities, like language or visual input. Given this, it is reasonable to assume that any AI system capable of mastering core human knowledge across diverse contexts – especially within the ARC-AGI-1 framework with which we are concerned – would also be able to reason using these different forms and modalities.

3.2 Architectural bias: from 2D to 1D

Xu et al. [47] conduct a failure analysis of GPT-4’s performance on ARC puzzles. Specifically, they argue that the model’s reliance on textual encodings and its inherently sequential processing of ARC’s 2D input-output grids impede effective problem solving. Their argument is supported by three empirical observations. First, when ARC objects are linearized into sequences of strings (as is done by the majority of approaches to ARC, like [18, 28] – consider Figure 2), GPT-4 exhibits higher accuracy on tasks involving horizontal object movements than on otherwise analogous tasks requiring vertical movements, indicating a bias introduced by the LLM’s sequential processing. Second, translating ARC tasks into a DSL (with standardized object descriptions and coordinates) that explicitly foregrounds object-level structures approximately doubles GPT-4’s accuracy. Third, Xu et al. [47] introduce 1D-ARC, a simplified variant of ARC in which puzzles are reduced to one-dimensional tasks designed to capture the same core knowledge priors. In combination with the DSL, GPT-4 achieves near-perfect performance on 1D-ARC.

Figure 2: Example of an ARC 2D grid that is translated into 1D sequential representations where each grid cell is either represented with a text string (left) or a number (right). New lines are indicated with “\n”.

The experiments by Xu et al. [47] show that the current Transformer architecture configuration [46], which underpins GPT-4 and most other LLMs applied to ARC [18, 28], is biased toward sequential tasks. They further demonstrate that performance improves significantly when the puzzles’ visual representations are converted into quasi-linguistic forms. This suggests that models not constrained to 1D sequential data, and capable of processing visual inputs, would be well suited for solving puzzles from ARC-AGI-1.

3.3 Multimodal reasoning models

Puget [39] develop a 2D Transformer model to solve ARC-AGI-1 puzzles. The model’s architecture is similar to GPT-2 [40], but it takes 2D grids as input rather than 1D sequences. This is achieved through a 2D attention mechanism, which attends to all cells in the same row of a given grid cell as well as to all cells in the same column. Puget [39] report that using an attention mechanism attending to all cells reduced performance in experiments. By permuting colors and applying rotations and transpositions to ARC-AGI-1 puzzles, they generate 335 million samples to train the model. Combined with TTT, their 42M-parameter model achieves 17% on the public evaluation set.

While Puget [39] focused on a single modality, that is, visual input, their results illustrate the promise of addressing the 1D sequential bias. Yet human-level reasoning involves more than visual 2D pattern recognition; it requires integration with abilities such as counting or reasoning about goal-directed objects. This motivates multimodal architectures, which can combine visual processing with other cognitive capacities.

Most MLLMs, including Flamingo [4], PaLM-E [13], LLaVA [29], GPT-4o [24], and Claude3-Opus/Sonnet [5], accept both visual and textual inputs and generate textual outputs. Rather than adopting specialized 2D attention, they linearize visual inputs into sequences of tokens, enabling integration with language modeling. Nevertheless the aligned training from visual and textual inputs turns them into powerful models for cross-modal tasks. However, early attempts to apply them to ARC were unsuccessful: Xu et al. [47] found that GPT-4V [35] produced no meaningful results. Reasons for this failure could be shallow tokenization of the pixel-level structure needed for ARC tasks, a mismatch between the training distribution of GPT-4V and ARC-AGI-1, or the model’s over-reliance on language. More recently, private submissions of MLLMs (e.g., OpenAI’s o3 [36]) have reported strong performance, achieving 53% on the semi-private evaluation set [26].

It is interesting to note that research on multimodal models for ARC remains limited. I hypothesize that this partly due to computational constraints in the ARC Prize challenge, which restricts participants to a single 12-hour Kaggle runtime on one P100 GPU. Scaling studies [1] suggest that synergies across modalities in Transformer-based architectures emerge only at sizes of ~30B parameters or more. This implies that current competition settings may under-resource multimodal approaches, even though they may ultimately prove the most suitable paradigm for ARC.

4 Future research directions in multimodal reasoning

Ultimately, we want models that reason with different modal inputs and based on different strategies, transductively and algorithmically. This section discusses possible research approaches to bring us closer to this vision and to solving ARC puzzles.

4.1 Solving ARC-AGI-1 with multimodal models

To explore whether multimodal capabilities can improve a model’s predictive performance on ARC puzzles, one could adopt a strategy similar to the combination of induction and transduction proposed by Li et al. [28]. Rather than ensembling an inductive and a transductive model, this approach would ensemble a purely linguistic LLM with a MLLM that processes both text and vision inputs. If the hypothesis of this article is correct – that visual reasoning differs from linguistic reasoning and can support more accurate predictions of ARC grids – then comparing an LLM with a VLM should also yield complementary results. While prior work with GPT-4V [47] has not shown success, a comparison involving more recent models such as the LLMs Qwen2.5 [48] or Mistral Large 2 [32] and their multimodal counterparts, Qwen2.5-VL [7] or Pixtral Large [33], would be valuable. Eventually, the question would be if a multimodal VLM alone could suffice to solve ARC puzzles.

This line of research could be strengthened by incorporating interpretability methods to better understand how models realize reasoning on ARC. Drawing inspiration from cognitive studies such as Fedorenko and Varley [15], which examine reasoning in humans, one could apply neurolinguistic-style probing techniques to LLMs [20, 21]. For instance, a subset of hidden states from a model – alongside randomly generated states – could be used to train a simple classifier on a downstream task derived from an ARC puzzle. If the hidden states encode task-relevant information, the classifier should outperform those trained on random states. An especially compelling question is whether VLMs mobilize different hidden states than their purely linguistic counterparts (LLMs) in such experiments.

A different approach that could strengthen visual reasoning for ARC could be realized with Sketchpad [23]. Sketchpad is a framework that helps MLLMs, such as GPT-4o, with additional tool calls to dissect, enhance, or mark images and plot additional figures. For example, a sliding window for a visual input can be used to help the model focus on parts of an image. While this approach increases the GPT-4o abilities on vision tasks by 8.6% [23], it is conceivable that similar tools could help an MLLM with ARC. Again, it would be interesting if such vision focused model with visual tools would perform differently than its linguistic counterpart.

Finally, a long-term project should focus on developing an architecture in the spirit of Puget [39] that effectively integrates both 2D and 1D inputs and pay appropriate attention to visual aspects. Although many current MLLMs have an alignment between visual and linguistic tokens, visual aspects are still not properly considered in many models’ context according to recent studies [8]. Therefore, existing VLMs may be further improved by techniques such as Dynamic Attention ReAllocation [10], which help models give proper weight to visual cues that are often overlooked in favor of textual patterns.

4.2 ARC-AGI-2 and ARC-AGI-3

With respect to the strong performance of the approaches [18, 28, 36] in the 2024 ARC-AGI-1 challenge, Chollet et al. [12] highlight several shortcomings. They argue that the public evaluation set of 100 tasks has effectively become a training signal for participants, diminishing its novelty and encouraging overfitting. Moreover, they report that 49% of the evaluation tasks could be solved by brute-force program search, which they regard as inadequate for measuring the kind of generalization humans demonstrate when solving ARC puzzles from demonstration pairs.

To address these issues while maintaining the overall structure of ARC-AGI-1, Chollet et al. [12] plan to launch ARC-AGI-2 with new puzzles for the 2025 ARC Prize. They also outline ARC-AGI-3, scheduled for 2026, which will feature similarly structured tasks but with interactive elements, such as moving bars [6]. Transforming ARC into an interactive benchmark foreshadows a paradigm shift that could further strengthen MLLMs, as these models are naturally well suited to handle interactive inputs. In particular, models trained within more dynamic, realistic environments – such as EmbodiedGPT [34] – represent promising candidates for interactive reasoning to tackle ARC-AGI-3.

Beyond technical concerns, a deeper question arises about ARC’s validity as a benchmark for human-like reasoning. The framework is inspired by core knowledge theories, yet its 2D grid puzzles may measure abstract symbol manipulation rather than genuinely grounded reasoning about objects, agents, or space. Moreover, ARC’s anthropocentric orientation risks equating human-like performance with intelligence, overlooking the possibility that alternative, non-human strategies could be equally valid forms of reasoning. As such, while ARC-AGI-2 and -3 introduce new tasks to mitigate overfitting, they may still inherit structural limitations from ARC’s original design.

Conclusion

Solving the ARC-AGI challenge requires moving beyond single paradigms. Symbolic program synthesis and transductive prediction demonstrate clear complementarities, but their current reliance on sequential, language-based encodings limits their ability to fully capture the visual and structural nature of ARC puzzles. This article has argued that multimodal reasoning offers a path forward: by integrating visual, algorithmic, and transductive strategies within unified models, we can move closer to the kind of flexible, general intelligence that ARC is designed to probe.

Looking ahead, multimodal approaches should be systematically evaluated against purely linguistic models, not only in terms of accuracy but also in terms of error profiles, interpretability, and robustness across different puzzle types. Future ARC iterations such as ARC-AGI-2 and ARC-AGI-3 will provide fertile ground for such experiments, especially as they introduce new puzzles and interactive elements that align naturally with embodied and multimodal reasoning.

Bibliography

[1] Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. (2023). Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning.

[2] Ainooson, J., Sanyal, D., Michelson, J., Yang, Y., and Kunda, M. (2023). A neurodiversity-inspired solver for the abstraction reasoning corpus (arc) using visual imagery and program synthesis.

[3] Akyürek, E., Damani, M., Zweiger, A., Qiu, L., Guo, H., Pari, J., Kim, Y., and Andreas, J. (2024). The surprising effectiveness of test-time training for few-shot learning. arXiv preprint, 2505.07859.

[4] Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. (2022). Flamingo: a visual language model for few-shot learning. arXiv preprint, abs/2204.14198.

[5] Anthropic (2024). The claude 3 model family: Opus, sonnet, haiku.

[6] ARC Prize Foundation (2025). Arc-agi-3: Interactive reasoning benchmark. https://arcprize.org/arc-agi/3/. ARC-AGI-3 is currently in development: early preview limited to six games (three public and three scheduled for release in August 2025); development began in early 2025 and full launch expected in 2026.

[7] Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. (2025). Qwen2.5-vl technical report. arXiv preprint, 2502.13923.

[8] Baldassini, F. B., Shukor, M., Cord, M., Soulier, L., and Piwowarski, B. (2024). What makes multimodal in-context learning work? 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1539–1550.

[9] Butt, N., Manczak, B., Wiggers, A. J., Rainone, C., Zhang, D. W., Defferrard, M., and Cohen, T. (2024). Codeit: Self-improving language models with prioritized hindsight replay. arXiv preprint, 2402.04858.

[10] Chen, S., Liu, J., Han, Z., Xia, Y., Cremers, D., Torr, P. H. S., Tresp, V., and Gu, J. (2025). True multimodal in-context learning needs attention to the visual context. arXiv preprint, 2507.15807.

[11] Chollet, F. (2019). On the measure of intelligence. arXiv preprint, 1911.01547.

[12] Chollet, F., Knoop, M., Kamradt, G., and Landers, B. (2024). Arc prize 2024: Technical report. arXiv preprint, 2412.04604.

[13] Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. H., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. R. (2023). Palm-e: An embodied multimodal language model. In International Conference on Machine Learning.

[14] Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A. S., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., tian Cantón Ferrer, C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E. A., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J.-Q., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., neth Heafield, K.-., Stone, K. R., El-Arini, K., Iyer, K., Malik, K., ley Chiu, K., Bhalla, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M. H. M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., lay Bashlykov, N., Bogoychev, N., Chatterji, N. S., Duchenne, O., cCelebi, O., Alrassy, P., Zhang, P., Li, P., Vasi´ c, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., nie Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., hana Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S. C., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., ginie Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., ney Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A. K., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, P.-Y. B., Loyd, B., de Paola, B., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, S.-W., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzm’an, F., Kanayet, F. J., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Thattai, G., Herman, G., Sizov, G. G., Zhang, G., Lakshminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Molybog, I., Tufanov, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., KamHou, U., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg, L., Lavender, A., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., ish Bansal, M., Santhanam, N., Parks, N., White, N., ata Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., dro Rittner, P., Bontrager, P., Roux, P., Dollár, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S. K., Cho, S.-B., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V. A., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Wang, Y., Hao, Y., Qian, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. (2024). The llama 3 herd of models. arXiv preprint, 2407.21783.

[15] Fedorenko, E. and Varley, R. A. (2016). Language and thought are not the same thing: evidence from neuroimaging and neurological patients. Annals of the New York Academy of Sciences, 1369.

[16] Fodor, J. (1975). The Language of Thought. Harvard University Press.

[17] Fodor, J. A. and Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1):3–71.

[18] Franzen, D., Disselhoff, J., and Hartmann, D. (2025). Product of experts with llms: Boosting performance on arc is a matter of perspective. arXiv preprint.

[19] Greenblatt, R. (2024). Submission for arc prize. https://www.kaggle.com/code/rgreenblatt/rg- basic-ported-submission?scriptVersionId=184981551.

[20] He, L., Chen, P., Nie, E., Li, Y., and Brennan, J. R. (2024). Decoding probing: Revealing internal linguistic structures in neural language models using minimal pairs. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N., editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4488–4497, Torino, Italia. ELRA and ICCL.

[21] He, L., Nie, E., Schmid, H., Schuetze, H., Mesgarani, N., and Brennan, J. (2025). Large language models as neurolinguistic subjects: Discrepancy between performance and competence. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T., editors, Findings of the Association for Computational Linguistics

[22] Hodel, M. (2024). Rearc. https://github.com/michaelhodel/re-arc.

[23] Hu, Y., Shi, W., Fu, X., Roth, D., Ostendorf, M., Zettlemoyer, L., Smith, N. A., and Krishna, R. (2024). Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C., editors, Advances in Neural Information Processing Systems, volume 37, pages 139348–139379. Curran Associates, Inc.

[24] Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., Mkadry, A., Baker-Whitcomb, A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A., Kirillov, A., Christakis, A., Conneau, A., Kamali, A., Jabri, A., Moyer, A., Tam, A., Crookes, A., Tootoochian, A., Tootoonchian, A., Kumar, A., Vallone, A., Karpathy, A., Braunstein, A., Cann, A., Codispoti, A., Galu, A., Kondrich, A., Tulloch, A., drey Mishchenko, A., Baek, A., Jiang, A., toine Pelisse, A., Woodford, A., Gosalia, A., Dhar, A., Pantuliano, A., Nayak, A., Oliver, A., Zoph, B., Ghorbani, B., Leimberger, B., Rossen, B., Sokolowsky, B., Wang, B., Zweig, B., Hoover, B., Samic, B., McGrew, B., Spero, B., Giertler, B., Cheng, B., Lightcap, B., Walkin, B., Quinn, B., Guarraci, B., Hsu, B., Kellogg, B., Eastman, B., Lugaresi, C., Wainwright, C. L., Bassin, C., Hudson, C., Chu, C., Nelson, C., Li, C., Shern, C. J., Conger, C., Barette, C., Voss, C., Ding, C., Lu, C., Zhang, C., Beaumont, C., Hallacy, C., Koch, C., Gibson, C., Kim, C., Choi, C., McLeavey, C., Hesse, C., Fischer, C., Winter, C., Czarnecki, C., Jarvis, C., Wei, C., Koumouzelis, C., Sherburn, D., Kappler, D., Levin, D., Levy, D., Carr, D., Farhi, D., Mély, D., Robinson, D., Sasaki, D., Jin, D., Valladares, D., Tsipras, D., Li, D., Nguyen, P. D., Findlay, D., Oiwoh, E., Wong, E., Asdar, E., Proehl, E., Yang, E., Antonow, E., Kramer, E., Peterson, E., Sigler, E., Wallace, E., Brevdo, E., Mays, E., Khorasani, F., Such, F. P., Raso, F., Zhang, F., von Lohmann, F., Sulit, F., Goh, G., Oden, G., Salmon, G., Starace, G., Brockman, G., Salman, H., Bao, H.-B., Hu, H., Wong, H., Wang, H., Schmidt, H., Whitney, H., woo Jun, H., Kirchner, H., de Oliveira Pinto, H. P., Ren, ˙ H., Chang, H., Chung, H. W., Kivlichan, I., O’Connell, I., Osband, I., Silber, I., Sohl, I., Ibrahim Cihangir Okuyucu, Lan, I., Kostrikov, I., Sutskever, I., Kanitscheider, I., Gulrajani, I., Coxon, J., Menick, J., Pachocki, J. W., Aung, J., Betker, J., Crooks, J., Lennon, J., Kiros, J. R., Leike, J., Park, J., Kwon, J., Phang, J., Teplitz, J., Wei, J., Wolfe, J., Chen, J., Harris, J., Varavva, J., Lee, J. G., Shieh, J., Lin, J., Yu, J., Weng, J., Tang, J., Yu, J., Jang, J., Candela, J. Q., Beutler, J., Landers, J., Parish, J., Heidecke, J., Schulman, J., Lachman, J., McKay, J., Uesato, J., Ward, J., Kim, J. W., Huizinga, J., Sitkin, J., Kraaijeveld, J., Gross, J., Kaplan, J., Snyder, J., Achiam, J., Jiao, J., Lee, J., Zhuang, J., Harriman, J., Fricke, K., Hayashi, K., Singhal, K., Shi, K., Karthik, K., Wood, K., Rimbach, K., Hsu, K., Nguyen, K., Gu-Lemberg, K., Button, K., Liu, K., Howe, K., Muthukumar, K., Luther, K., Ahmad, L., Kai, L., Itow, L., Workman, L., Pathak, L., Chen, L., Jing, L., Guy, L., Fedus, L., Zhou, L., Mamitsuka, L., Weng, L., McCallum, L., Held, L., Long, O., Feuvrier, L., Zhang, L., Kondraciuk, L., Kaiser, L., Hewitt, L., Metz, L., Doshi, L., Aflak, M., Simens, M., laine Boyd, M., Thompson, M., Dukhan, M., Chen, M., Gray, M., Hudnall, M., Zhang, M., Aljubeh, M., teusz Litwin, M., Zeng, M., Johnson, M., Shetty, M., Gupta, M., Shah, M., Yatbaz, M. A., Yang, M., Zhong, M., Glaese, M., Chen, M., Janner, M., Lampe, M., Petrov, M., Wu, M., Wang, M., Fradin, M., Pokrass, M., Castro, M., Castro, M., Pavlov, M., Brundage, M., Wang, M., Khan, M., Murati, M., Bavarian, M., Lin, M., Yesildal, M., Soto, N., Gimelshein, N., talie Cone, N., Staudacher, N., Summers, N., LaFontaine, N., Chowdhury, N., Ryder, N., Stathas, N., Turley, N., Tezak, N. A., Felix, N., Kudige, N., Keskar, N. S., Deutsch, N., Bundick, N., Puckett, N., Nachum, O., Okelola, O., Boiko, O., Murk, O., Jaffe, O., Watkins, O., Godement, O., Campbell-Moore, O., Chao, P., McMillan, P., Belov, P., Su, P., Bak, P., Bakkum, P., Deng, P., Dolan, P., Hoeschele, P., Welinder, P., Tillet, P., Pronin, P., Tillet, P., Dhariwal, P., ing Yuan, Q., Dias, R., Lim, R., Arora, R., Troll, R., Lin, R., Lopes, R. G., Puri, R., Miyara, R., Leike, R. H., Gaubert, R., Zamani, R., Wang, R., Donnelly, R., Honsby, R., Smith, R., Sahai, R., Ramchandani, R., Huet, R., Carmichael, R., Zellers, R., Chen, R., Chen, R., Nigmatullin, R. R., Cheu, R., Jain, S., Altman, S., Schoenholz, S., Toizer, S., Miserendino, S., Agarwal, S., Culver, S., Ethersmith, S., Gray, S., Grove, S., Metzger, S., Hermani, S., Jain, S., Zhao, S., Wu, S., Jomoto, S., Wu, S., Xia, S., Phene, S., Papay, S., Narayanan, S., Coffey, S., Lee, S., Hall, S., Balaji, S., Broda, T., Stramer, T., Xu, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Cunninghman, T., Degry, T., Dimson, T., Raoux, T., Shadwell, T., Zheng, T., Underwood, T., Markov, T., Sherbakov, T., Rubin, T., Stasi, T., Kaftan, T., Heywood, T., Peterson, T., Walters, T., Eloundou, T., Qi, V., Moeller, V., Monaco, V., Kuo, V., Fomenko, V., Chang, W., Zheng, W., Zhou, W., Manassra, W., Sheu, W., Zaremba, W., Patil, Y., Qian, Y., Kim, Y., Cheng, Y., Zhang, Y., He, Y., Zhang, Y., Jin, Y., Dai, Y., and Malkov, Y. (2024). Gpt-4o system card. arXiv preprint, 2410.21276.

[25] Kahneman, D. (2012). Thinking, fast and slow. Penguin, London.

[26] Kamradt, G. (2025). Analyzing o3 and o4-mini with arc-agi. https://arcprize.org/blog/analyzing-o3-with-arc-agi.

[27] LeGris, S., Vong, W. K., Lake, B. M., and Gureckis, T. M. (2025). A comprehensive behavioral dataset for the abstraction and reasoning corpus. Scientific Data, 12.

[28] Li, W.-D., Hu, K., Larsen, C., Wu, Y., Alford, S., Woo, C., Dunn, S. M., Tang, H., Naim, M., Nguyen, D., Zheng, W.-L., Tavares, Z., Pu, Y., and Ellis, K. (2024). Combining induction and transduction for abstract reasoning. arXiv preprint, 2411.02272.

[29] Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023). Visual instruction tuning. arXiv preprint, 2304.08485.

[30] McCulloch, W. S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4):115–133.

[31] McGreggor, K., Kunda, M., and Goel, A. K. (2014). Fractals and ravens. Artif. Intell., 215:1–23.

[32] Mistral AI (2024a). Large enough: Mistral large 2. https://mistral.ai/news/mistral-large-2407.

[33] Mistral AI (2024b). Pixtral large. https://mistral.ai/news/pixtral-large.

[34] Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P. (2023). Embodiedgpt: Vision-language pre-training via embodied chain of thought. arXiv preprint, 2305.15021.

[35] OpenAI (2023). Gpt-4v(ision) system card.

[36] OpenAI (2025). Openai o3 and o4-mini system card.

[37] Pearson, J. and Kosslyn, S. M. (2015). The heterogeneity of mental representation: Ending the imagery debate. Proceedings of the National Academy of Sciences, 112:10089 – 10092.

[38] Pitt, D. (2022). Mental Representation. In Zalta, E. N. and Nodelman, U., editors, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2022 edition.

[39] Puget, J.-F. (2024). A 2d ngpt model for arc prize. https://github.com/jfpuget/ARC-AGI- Challenge-2024/blob/main/arc.pdf. Presented as a solution to the ARC Prize 2024 competition on Kaggle.

[40] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.

[41] Rumelhart, D. E. (1997). The architecture of mind: A connectionist approach. In Mind Design II: Philosophy, Psychology, and Artificial Intelligence. The MIT Press.

[42] Rumelhart, D. E., McClelland, J. L., and Group, P. R. (1986). Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations. The MIT Press.

[43] Soulières, I., Dawson, M., Samson, F., Barbeau, E. B., Sahyoun, C. P., Strangman, G. E., Zeffiro, T. A., and Mottron, L. (2009). Enhanced visual processing contributes to matrix reasoning in autism. Human Brain Mapping, 30.

[44] Spelke, E. S. and Kinzler, K. D. (2007). Core knowledge. Developmental science, 10 1:89–96.

[45] Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. (2019). Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning.

[46] Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Neural Information Processing Systems.

[47] Xu, Y. W., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. (2023). Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations. arXiv preprint, 2305.18354.

[48] Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y.-C., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., Qiu, Z., Quan, S., and Wang, Z. (2024). Qwen2.5 technical report. arXiv preprint, 2412.15115.

1 Introduction

Drawing and sketching are cognitive tools that humans use not only to express and communicate thoughts, but also to generate new ones [6]. For this matter, we would like to equip any intelligent system with the same ability to improve and help it communicate its reasoning. First steps in this direction have been proposed within the field of Multimodal Chain-of-Thought (MCoT) where reasoning steps are enriched with data from different modalities, such as visuals. Therefore, future research on sketching should advance the design of MCoT reasoning strategies. Improving Multimodal Large Language Models (MLLMs) that perform such cross-modal reasoning is also relevant.

The present text outlines the motivation why sketching should be incorporated into CoT in Section 2. Several related works that use images and even sketches to aid models in reasoning are introduced in Section 3. Finally, Section 4 proposes directions for future research: a new dataset combining sketches with reasoning chains, advancements of unified MLLMs for MCoT, in particular with diffusion models, as well as the usage of reinforcement learning for existing MCoT approaches, such as rewards with Reinforcement Learning with Verifiable Rewards (RLVR) [9] and Group Relative Policy Optimization (GRPO) [28], both for textual and visual reasoning.

2 Motivation to incorporate drawing capabilities into AI

Humans express and communicate ideas visually through drawing and sketching, which is a quick and loose form of drawing. Drawing is a representation of thought, but also an activity that can support ongoing cognition [6]. Drawing and sketching precede writing: The first documented drawings date back as far as 64,000 years [10]. For that reason, Fan et al. [6] argue that drawing is one of the most enduring and versatile cognitive tools from which humans have benefited.

One explanation for the power of drawing and sketching can be derived from cognitive enhancement and offloading strategies. According to Morrison and Richmond [21], technologies are used as external memories, facilitating other tasks by freeing up memory. Similarly, Osiurak et al. [24] show that tools such as maps can extend human’s cognitive abilities.

Given the relevance of drawing and sketching for human thought, expression, and communication, we would want to equip any AI with the capability to also use this tool to advance and share its own ideas. Sketching can not only be a window into how AI models process information, but it is fair to assume that it can also support their reasoning.

Reasoning in large language models (LLMs) has been greatly improved with in-context learning (ICL) [20] and Chain-of-Thought (CoT) techniques [23, 34]. ICL helps models with additional information added to the input to find appropriate responses for a given task. With CoT, the contextual information is specifically extended by a simulation of human reasoning steps, where a task is divided into subtasks for which intermediate solutions are given so that the model can derive its final answer from them. This can be achieved by eliciting reasoning through prompting, as with ’think step-by-step’ prompts (Zero-Shot-CoT [15]), or by providing the model with an explicit reasoning demonstration (also called a rationale) for a given problem (Few-Shot-CoT [34]).

CoT has been extended with multimodal information [32, 33] where models receive more than text to guide them toward a correct answer. This information can consist of visual, auditory, or spatio-temporal data. Sketches would be additional visual information. They could also help models to offload complex tasks and retain intermediate memories, for example, of subtasks. Therefore, an implementation of the capability to sketch in order to enhance models’ reasoning abilities should expand existing research in MCoT. A detailed account of MCoT is given in Appendix A.

3 Related work

Several recent approaches explore MCoT reasoning, though most do not fully integrate sketch generation into the reasoning process.

Zhang et al. [42] propose a two-stage framework for multiple-choice reasoning for text and image inputs where a FLAN-AlpacaBase model [30, 44] first produces a rationale, then derives the answer. Fusing text and image features from the input improves performance, but the system cannot generate new visual content. This limits applicability to reasoning scenarios that benefit from active visual exploration, such as diagram construction in geometry or mechanical design tasks.

Meng et al. [19] extend CoT by having an LLM produce symbolic sketch-like diagrams (e.g., with SVG), rendered into images and re-encoded for reasoning. Their ’think image by image’ approach helps, for example, with geometric tasks. However, this gain comes at the cost of operational complexity: the pipeline depends on separate LLMs, rendering engines, and encoders, creating latency and integration challenges. Unified MLLMs avoid such fragmentation and may better support generalization by learning a shared latent space for both text and sketches.

In contrast to the previous two approaches, Liao et al. [16] fine-tune unified MLLMs (SEED-LLaMA [7] and SEED-X [8]) on their ImageGen-CoT dataset. Reasoning steps of their models precede image generation. Test-Time Scaling is applied to select better outputs. While they demonstrate high-quality image generation, their evaluation focuses on aesthetics and relevance rather than measurable reasoning improvement. For reasoning-centric applications, visual fidelity without explicit reasoning gains may be insufficient.

Hu et al. [11] and Vinker et al. [31] develop agentic strategies (Sketchpad, Sketchagent) where models like GPT-4o [13] or Claude3.5-Sonnet [3] can decide to produce or modify sketches during problem-solving by leveraging external vision models, Python or a domain-specific language (DSL) for sketches. Models with Sketchpad iterate over a ’thought’, ’action’ (to inject sketches), and ’observation’ pattern. With this approach, Hu et al. [11] show that allowing models to decide to insert sketches during reasoning leads to notable performance gains. However, the framework relies on external vision models to rather enhance or dissect images and a Python sketch representation, which may not capture the nuances of freehand or abstract sketches common in human reasoning.

A truly multimodal approach for sketches would not use Python or DSLs to ’implicitly’ generate figures that the model ingests as textual input. However, few multimodal datasets that combine visuals with rationales exist. While QuickDraw [14] provides scale and diversity in sketch data, its lack of accompanying rationales prevents multimodal alignment learning. ScienceQA [18] and ImageGen-CoT [16] offer strong rationale-image pairs, but the absence of sketches means they primarily serve full-image reasoning rather than schematic reasoning. This gap suggests that the field currently lacks a dataset that balances sketch simplicity with reasoning, a pairing that could uniquely advance MCoT.

Overall, existing MCoT work shows that visual information, including sketches, can aid reasoning. However, limitations remain: most systems either consume but do not create sketches, focus on image quality rather than reasoning improvement, or require orchestration of multiple models instead of unified generation. Furthermore, appropriate datasets with sketches in combination with rationales are lacking.

4 Future research for MCoT with sketching

Given the power of visual information for reasoning tasks, as shown by [42, 19, 16, 11], some of the shortcomings of existing MCoT approaches can be addressed to better incorporate sketching in future research.

4.1 Creating a new MCoT sketch dataset

To facilitate the training of MLLMs, the lack of an appropriate dataset with sketching and rationales is a limitation.

Sketch data should be gathered and grouped within different categories, depending on the downstream task (consider Figure 1). In experimental studies with humans, Huey et al. [12] point out that drawings differ according to their intended goal: visual explanations by the participants emphasized moving and interactive parts, while their visual depictions focused on salient features. Hu et al. [11] show that adding auxiliary lines to geometric figures helps multimodal models such as GPT-4o to infer correct answers about these figures. Fan et al. [6] highlight that not all drawings are faithful depictions, but can also be abstractions whose meanings are conveyed by cultural conventions.

Figure 1: Different types of sketches and drawings: (a) depicts a geometric form that has an auxiliary line, (b) emphasizes moving parts of a machine, (c) depicts the same machine in more detail, (d) represents figures from tetris whose next moves are indicated with arrows, (e) is a conventional sketch of a heart that does not resemble actual human hearts.

To integrate sketches into a CoT, training data should not only consist of images of drawings and sketches, but combine these with textual rationales. This would enable multimodal alignment between visual and linguistic reasoning steps. A typical template for this data could consist of instruction I, query Q, rationale R, and answer A where we could further divide R into ’thought’, ’sketch’, and ’observation’ with respective special tokens to guide the model, loosely following Hu et al. [11]. An example template is given in Appendix B. Since ScienceQA and ImageGen-CoT already pair images with rationales, they could be extended with sketches to strengthen visual-textual alignment for their tasks.

4.2 Advancing MCoT with unified MLLMs

To avoid multi-model orchestration and to leverage potential transfer-learning effects, further advancing reasoning of MLLMs with sketches is a promising direction. However, there exist only a few MLLMs [39, 43, 42, 29] that can potentially handle sketch-to-text as well as text-to-sketch tasks within a unified architecture (consider Figure 2). The majority of current approaches such as Sketchpad pair VLMs such as Flamingo [2], PaLM-E [5], LLAVA [17], GPT-4o [13], or Claude3-Opus and Claude3.5-Sonnet [3] with text-to-image models.

Unified MLLMs can be divided into autoregressive (AR) and diffusion-based MLLMs. For example, CM3Leon [39] from Meta is a Transfomer-based AR decoder that can generate both text and images. It is built on the CM3 model [1]. CM3Leon has been trained on text-guided image editing, image- to-image grounding tasks where visual features can be derived from images, and text-to-image generations.

Swerdlow et al. [29] introduce a unified multimodal discrete diffusion model (UniDisc). While the model’s architecture consists of a Transformer (bidirectional) decoder, its training goal is not to auto-regressively predict the next tokens in a sequential manner (e.g., left to right for text or top to bottom for image patch rasters), but to predict the distribution of tokens via a denoising process that allows parallel predictions as well as later refinements. The training of UniDisc is realized with a denoising process of corrupted inputs (masking). In contrast to continuous diffusion models, Swerdlow et al. [29] use discrete noising and denoising for both images and texts. Swerdlow et al. [29] show that UniDisc outperforms the same architecture without a diffusion objective with respect to image and text classification tasks. The model is also capable of inpainting and infilling missing parts of an input, which no AR model can do. However, these performance gains come at a cost: UniDisc requires 13.2 times longer than its AR counterpart to reach equivalent loss levels [29].

Figure 2: MCoT involving sketches with a Multimodal Large Language Model (MLLM). Black arrows represent sequential auto-regressive processing, while blue arrows illustrate the bidirectionality of diffusion models. The model’s reasoning is guided by special tokens, such as .

Models like UniDisc provide an interesting model class for MCoT. While current diffusion language models (DLMs) might not rival AR LLMs due to training inefficiencies [29] or speed [38], the strength of multimodal DLMs in handling and generating multimodal data – as shown by Swerdlow et al. [29] – warrants further research. Their ability to inpaint and infill would be particularly helpful for amending visualizations, which is a core aspect of explanatory sketching. Research in this direction could be informed by Diffusion-of-Thought (DoT) proposed by Ye et al. [37], who fine-tune a DLM for CoT. However, diffusion models require a fixed output size. This is a challenge that needs to be addressed to allow versatile reasoning over different tasks.

4.3 Improving MCoT with Reinforcement Learning (RL) and Test-Time Scaling

Existing work on MCoT [42, 19, 16] has so far relied on supervised fine-tuning (SFT). Other work in reasoning has shown that RL leads to improvements [9, 27, 43]. Therefore, MCoT should be advanced with Direct Preference Optimization (DPO) [26], Reinforcement Learning with Verifiable Rewards (RLVR) [9] and Group Relative Policy Optimization (GRPO) [28] strategies. One straight-forward application would be to use RLVR with GRPO, following Deepseek’s R1 [9], to reward accuracy (\(R_{acc}\)) and format (\(R_{format}\)) for rationales and answers based on generated sketches.

An appropriate reward for the generation of sketches could leverage AR-GRPO for autoregressive MLLMs [40]. AR-GRPO realizes rewards for the generation of images with a multi-faceted reward function that ensures (a) consistency with the textual input condition through CLIP [25] and Human Preference Score v2 [35], (b) image quality with MANIQA [36], and (c) a further realism reward through a VLM, such as Qwen2.5-VL-3B-Instruct [4]. This function is used with GRPO to improve the quality of generated images. Since the proposed rewards by Yuan et al. [40] focus on overall quality, a specific reward should be conceived for sketches. For example, a sketch can consist of a hierarchy of strokes whose meaning can be of different importance. It would be interesting to incorporate this somehow into the reward: Should sketches with a limited amount of strokes be prioritized?

In the wake of Liao et al. [16], existing MCoT could be further improved with Test-Time Scaling methods, sampling more CoTs and sketches to select the best candidates with an appropriate scoring method. This approach could also be used with agentic frameworks that pair VLMs with image generators and would not require any additional training of the models.

Beyond standard accuracy on downstream tasks, evaluation should measure how sketches contribute to the reasoning process. This includes interpretability (e.g., can a human follow the model’s reasoning with a sketch?), task completion time (one of the biggest bottlenecks because image generation requires many tokens), error localization, and robustness under noisy or incomplete inputs. Additionally, user studies could assess subjective clarity and helpfulness of generated sketches.

Impact

MLLMs with sketching would have an impact on AI in different domains. For example, agentic systems such as Auto-GUI [41] that interact with graphical user interfaces or websites could be enhanced by providing them with additional visual information with sketches. Similarly, embodied AI systems, such as EmbodiedGPT [22] whose backbone uses a combination of vision and language models that help navigate the real world, could reason about their surroundings using sketches. MLLMs for STEM education could also benefit from the ability to make their reasoning more transparent with additional drawings as proposed in Meng et al. [19]. In sum, sketching would help all reasoning models not only to enhance their thoughts, but also communicate them with more than one modality.

As with language, sketches are not neutral representations. The ability of AI systems to generate and reason with sketches introduces risks of cultural bias, visual misrepresentation, and domain-specific inaccuracies. For example, the “heart” symbol in Figure 1(e) is globally recognized in popular culture but anatomically incorrect; in medical education, reasoning over such a schematic could reinforce misconceptions. Similar issues may arise if models default to culturally specific diagrammatic conventions, omit critical features due to dataset biases, or overgeneralize from training examples.

Ethical safeguards should address the entire MCoT-with-sketching workflow. Dataset curation must ensure diversity of styles, cultural perspectives, and schematic conventions. Annotation guidelines should clarify the intended use and accuracy requirements of sketches. Model evaluation should include bias detection for visual outputs, alongside interpretability checks so users can trace how a sketch influenced reasoning.

Apendix A: MCoT foundations

Following Wang et al. [33], we can define prompt, instruction, query, answer, and rationale with \(P\) , \(I\), \(Q\), \(A\), and \(R\), which are all token sequences. A Chain-of-Thought (CoT) would be:

\begin{equation} P_{CoT} = {I, (x_1, e_1, y_1), …, (x_n, e_n, y_n)} \end{equation}

where \(x_i \in Q\) and \(y_i \in A\) are questions with corresponding answers and \(e_i \in R\) is an example rationale. The joint probability of generating an answer A and a rationale R given the prompt \(P_{CoT}\) and a query \(Q\) would be [33]:

\begin{equation} p(A, R |P_{CoT}, Q) = p(R |P_{CoT}, Q) \cdot p(A |P_{CoT}, Q, R) \end{equation}

where the model should output rationale \(R\) with the tokens \(r_1, ..., r_i\) before arriving at the answer \(A\) consisting of the tokens \(a_1, ..., a_i\). The goal in training a reasoning model \(F\) is to jointly maximize the likelihood of equation (2).

Finally, all components \(P\), \(Q\), \(A\), and \(R\) can be enriched with multimodal information \(\mathcal{M}\). For example with MCoT, a rationale \(R\) should handle \(\mathcal{M}\) input and generate multimodal information (e.g., a sketch) as well as text \(T\), that is, \(R\in\{M, M\oplus T\}\) [33].

Appendix B: MCoT template

{ "instruction": "Find proofs for geometry problems.", "query": "Prove the angles of ABC provided in the attached image sum to 180. <image> VT_011 VT_115 VT_563 VT_101 ... VT_909 </image>", "rationale": "<think> I need to figure out how ABC are related in the image. The image shows a triangle. I need to prove that the angles of the triangle sum to 180. To find an answer, I draw a triangle: Let's call it ABC. <sketch> VT_421 VT_105 VT_983 VT_002 ... VT_778 </sketch> I extend the sides from A to B, from A to C, and from B to C. <sketch> VT_421 VT_105 VT_983 VT_001 ... VT_708 </sketch> I draw a line parallel to AB through point C. <sketch> VT_420 VT_105 VT_983 VT_001 ... VT_718 </sketch> <observe> The angles at point C created by the parallel line correspond to the interior angles at points A and B. When I add those angles up, they form a straight line at point C, which measures 180. Since those angles correspond exactly to the three interior angles of the triangle, the sum of the interior angles is 180. </observe> This proof follows from the alternate interior angles theorem. </think>", "answer": "The alternate interior angles theorem shows that all angles at point C created by the parallel line sum to 180. They further correspond to the interior angles at points A and B. Therefore, the angles of ABC provided in the attached image sum to 180." }

MCoT template with instruction \(I\), query \(Q\), rationale \(R\), and answer \(A\) where \(R\) is further divided into “thought”, “sketch”, and “observation” with respective special tokens to guide the model. VT_n tokens correspond to image tokens.

Bibliography

[1] Aghajanyan, A., Huang, P.-Y. B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., and Zettlemoyer, L. (2022). Cm3: A causal masked multimodal model of the internet. ArXiv, abs/2201.07520.

[2] Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. (2022). Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198.

[3] Anthropic (2024). The claude 3 model family: Opus, sonnet, haiku.

[4] Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. (2025). Qwen2.5-vl technical report. ArXiv, abs/2502.13923.

[5] Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. H., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. R. (2023). Palm-e: An embodied multimodal language model. In International Conference on Machine Learning.

[6] Fan, J. E., Bainbridge, W. A., Chamberlain, R., and Wammes, J. D. (2023). Drawing as a versatile cognitive tool. Nature Reviews Psychology, 2(9).

[7] Ge, Y., Zhao, S., Zeng, Z., Ge, Y., Li, C., Wang, X., and Shan, Y. (2023). Making llama see and draw with seed tokenizer. ArXiv, abs/2310.01218.

[8] Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. (2024). Seed-x: Multimodal models with unified multi-granularity comprehension and generation. ArXiv, abs/2404.14396.

[9] Guo, D., Yang, D., Zhang, H., Song, J.-M., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B.-L., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D.-L., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q., Du, Q., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S.-K., Yun, T., Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W.-X., Zhang, W., Xiao, W., An, W., Liu, X., Wang, X., aokang Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X.-C., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y.-J., He, Y., Xiong, Y., Luo, Y.-W., mei You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., guo Zhang, Z., Hao, Z., Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z.-A., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv, abs/2501.12948.

[10] Hoffmann, D. L., Standish, C. D., García-Diez, M., Pettitt, P. B., Milton, J. A., Zilhão, J., Alcolea-González, J. J., Cantalejo-Duarte, P., Collado, H., de Balbín, R., Lorblanchet, M., Ramos- Muñoz, J., Weniger, G.-C., and Pike, A. W. G. (2018). U-th dating of carbonate crusts reveals neandertal origin of iberian cave art. Science, 359(6378):912–915.

[11] Hu, Y., Shi, W., Fu, X., Roth, D., Ostendorf, M., Zettlemoyer, L. S., Smith, N. A., and Krishna, R. (2024). Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. ArXiv, abs/2406.09403.

[12] Huey, H., Lu, X., Walker, C. M., and Fan, J. (2023). Visual explanations prioritize functional properties at the expense of visual fidelity. Cognition, 236.

[13] Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., Mkadry, A., Baker-Whitcomb, A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A., Kirillov, A., Christakis, A., Conneau, A., Kamali, A., Jabri, A., Moyer, A., Tam, A., Crookes, A., Tootoochian, A., Tootoonchian, A., Kumar, A., Vallone, A., Karpathy, A., Braunstein, A., Cann, A., Codispoti, A., Galu, A., Kondrich, A., Tulloch, A., drey Mishchenko, A., Baek, A., Jiang, A., toine Pelisse, A., Woodford, A., Gosalia, A., Dhar, A., Pantuliano, A., Nayak, A., Oliver, A., Zoph, B., Ghorbani, B., Leimberger, B., Rossen, B., Sokolowsky, B., Wang, B., Zweig, B., Hoover, B., Samic, B., McGrew, B., Spero, B., Giertler, B., Cheng, B., Lightcap, B., Walkin, B., Quinn, B., Guarraci, B., Hsu, B., Kellogg, B., Eastman, B., Lugaresi, C., Wainwright, C. L., Bassin, C., Hudson, C., Chu, C., Nelson, C., Li, C., Shern, C. J., Conger, C., Barette, C., Voss, C., Ding, C., Lu, C., Zhang, C., Beaumont, C., Hallacy, C., Koch, C., Gibson, C., Kim, C., Choi, C., McLeavey, C., Hesse, C., Fischer, C., Winter, C., Czarnecki, C., Jarvis, C., Wei, C., Koumouzelis, C., Sherburn, D., Kappler, D., Levin, D., Levy, D., Carr, D., Farhi, D., Mély, D., Robinson, D., Sasaki, D., Jin, D., Valladares, D., Tsipras, D., Li, D., Nguyen, P. D., Findlay, D., Oiwoh, E., Wong, E., Asdar, E., Proehl, E., Yang, E., Antonow, E., Kramer, E., Peterson, E., Sigler, E., Wallace, E., Brevdo, E., Mays, E., Khorasani, F., Such, F. P., Raso, F., Zhang, F., von Lohmann, F., Sulit, F., Goh, G., Oden, G., Salmon, G., Starace, G., Brockman, G., Salman, H., Bao, H.-B., Hu, H., Wong, H., Wang, H., Schmidt, H., Whitney, H., woo Jun, H., Kirchner, H., de Oliveira Pinto, H. P., Ren, ˙ H., Chang, H., Chung, H. W., Kivlichan, I., O’Connell, I., Osband, I., Silber, I., Sohl, I., Ibrahim Cihangir Okuyucu, Lan, I., Kostrikov, I., Sutskever, I., Kanitscheider, I., Gulrajani, I., Coxon, J., Menick, J., Pachocki, J. W., Aung, J., Betker, J., Crooks, J., Lennon, J., Kiros, J. R., Leike, J., Park, J., Kwon, J., Phang, J., Teplitz, J., Wei, J., Wolfe, J., Chen, J., Harris, J., Varavva, J., Lee, J. G., Shieh, J., Lin, J., Yu, J., Weng, J., Tang, J., Yu, J., Jang, J., Candela, J. Q., Beutler, J., Landers, J., Parish, J., Heidecke, J., Schulman, J., Lachman, J., McKay, J., Uesato, J., Ward, J., Kim, J. W., Huizinga, J., Sitkin, J., Kraaijeveld, J., Gross, J., Kaplan, J., Snyder, J., Achiam, J., Jiao, J., Lee, J., Zhuang, J., Harriman, J., Fricke, K., Hayashi, K., Singhal, K., Shi, K., Karthik, K., Wood, K., Rimbach, K., Hsu, K., Nguyen, K., Gu-Lemberg, K., Button, K., Liu, K., Howe, K., Muthukumar, K., Luther, K., Ahmad, L., Kai, L., Itow, L., Workman, L., Pathak, L., Chen, L., Jing, L., Guy, L., Fedus, L., Zhou, L., Mamitsuka, L., Weng, L., McCallum, L., Held, L., Long, O., Feuvrier, L., Zhang, L., Kondraciuk, L., Kaiser, L., Hewitt, L., Metz, L., Doshi, L., Aflak, M., Simens, M., laine Boyd, M., Thompson, M., Dukhan, M., Chen, M., Gray, M., Hudnall, M., Zhang, M., Aljubeh, M., teusz Litwin, M., Zeng, M., Johnson, M., Shetty, M., Gupta, M., Shah, M., Yatbaz, M. A., Yang, M., Zhong, M., Glaese, M., Chen, M., Janner, M., Lampe, M., Petrov, M., Wu, M., Wang, M., Fradin, M., Pokrass, M., Castro, M., Castro, M., Pavlov, M., Brundage, M., Wang, M., Khan, M., Murati, M., Bavarian, M., Lin, M., Yesildal, M., Soto, N., Gimelshein, N., talie Cone, N., Staudacher, N., Summers, N., LaFontaine, N., Chowdhury, N., Ryder, N., Stathas, N., Turley, N., Tezak, N. A., Felix, N., Kudige, N., Keskar, N. S., Deutsch, N., Bundick, N., Puckett, N., Nachum, O., Okelola, O., Boiko, O., Murk, O., Jaffe, O., Watkins, O., Godement, O., Campbell-Moore, O., Chao, P., McMillan, P., Belov, P., Su, P., Bak, P., Bakkum, P., Deng, P., Dolan, P., Hoeschele, P., Welinder, P., Tillet, P., Pronin, P., Tillet, P., Dhariwal, P., ing Yuan, Q., Dias, R., Lim, R., Arora, R., Troll, R., Lin, R., Lopes, R. G., Puri, R., Miyara, R., Leike, R. H., Gaubert, R., Zamani, R., Wang, R., Donnelly, R., Honsby, R., Smith, R., Sahai, R., Ramchandani, R., Huet, R., Carmichael, R., Zellers, R., Chen, R., Chen, R., Nigmatullin, R. R., Cheu, R., Jain, S., Altman, S., Schoenholz, S., Toizer, S., Miserendino, S., Agarwal, S., Culver, S., Ethersmith, S., Gray, S., Grove, S., Metzger, S., Hermani, S., Jain, S., Zhao, S., Wu, S., Jomoto, S., Wu, S., Xia, S., Phene, S., Papay, S., Narayanan, S., Coffey, S., Lee, S., Hall, S., Balaji, S., Broda, T., Stramer, T., Xu, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Cunninghman, T., Degry, T., Dimson, T., Raoux, T., Shadwell, T., Zheng, T., Underwood, T., Markov, T., Sherbakov, T., Rubin, T., Stasi, T., Kaftan, T., Heywood, T., Peterson, T., Walters, T., Eloundou, T., Qi, V., Moeller, V., Monaco, V., Kuo, V., Fomenko, V., Chang, W., Zheng, W., Zhou, W., Manassra, W., Sheu, W., Zaremba, W., Patil, Y., Qian, Y., Kim, Y., Cheng, Y., Zhang, Y., He, Y., Zhang, Y., Jin, Y., Dai, Y., and Malkov, Y. (2024). Gpt-4o system card. ArXiv, abs/2410.21276.

[14] Jonas, J., Henry, R., Takashi, K., Jongmin, K., and Nick, F.-G. (2016). The quick, draw! - a.i. experiment.

[15] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.

[16] Liao, J., Yang, Z., Li, L., Li, D., Lin, K. Q., Cheng, Y., and Wang, L. (2025). Imagegen-cot: Enhancing text-to-image in-context learning with chain-of-thought reasoning. ArXiv, abs/2503.19312.

[17] Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023). Visual instruction tuning. ArXiv, abs/2304.08485.

[18] Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. (2022). Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS).

[19] Meng, F., Yang, H., Wang, Y., and Zhang, M. (2023). Chain of images for intuitively reasoning. ArXiv, abs/2311.09241.

[20] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.12837.

[21] Morrison, A. B. and Richmond, L. L. (2020). Offloading items from memory: individual differences in cognitive offloading in a short-term memory task. Cognitive Research: Principles and Implications, 5(1):1.

[22] Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P. (2023). Embodiedgpt: Vision-language pre-training via embodied chain of thought. ArXiv, abs/2305.15021.

[23] Nye, M., Andreassen, A., Gur-Ari, G., Michalewski, H. W., Austin, J., Bieber, D., Dohan, D. M., Lewkowycz, A., Bosma, M. P., Luan, D., Sutton, C., and Odena, A. (2021). Show your work: Scratchpads for intermediate computation with language models. https://arxiv.org/abs/2112.00114.

[24] Osiurak, F., Navarro, J., Reynaud, E., and Thomas, G. (2018). Tools don’t–and won’t–make the man: A cognitive look at the future. Journal of Experimental Psychology: General, 147(5):782– 788.

[25] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.

[26] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290.

[27] Ranaldi, L. and Pucci, G. (2025). Multilingual reasoning via self-training. In North American Chapter of the Association for Computational Linguistics.

[28] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J.-M., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300.

[29] Swerdlow, A., Prabhudesai, M., Gandhi, S., Pathak, D., and Fragkiadaki, K. (2025). Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853.

[30] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Alpaca: A strong, replicable instruction-following model. https://crfm.stanford.edu/2023/03/13/alpaca.html.

[31] Vinker, Y., Shaham, T. R., Zheng, K., Zhao, A., Fan, J. E., and Torralba, A. (2024). Sketchagent: Language-driven sequential sketch generation. ArXiv, abs/2411.17673.

[32] Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., and Yang, H. (2024). Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. ArXiv, abs/2401.06805.

[33] Wang, Y., Wu, S., Zhang, Y., Yan, S., Liu, Z., Luo, J., and Fei, H. (2025). Multimodal chain-of-thought reasoning: A comprehensive survey. ArXiv, abs/2503.12605.

[34] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.

[35] Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. (2023). Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. ArXiv, abs/2306.09341.

[36] Yang, S., Wu, T., Shi, S., Gong, S., Cao, M., Wang, J., and Yang, Y. (2022). Maniqa: Multi- dimension attention network for no-reference image quality assessment. 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1190–1199.

[37] Ye, J., Gong, S., Chen, L., Zheng, L., Gao, J., Shi, H., Wu, C., Jiang, X., Li, Z., Bi, W., and Kong, L. (2024). Diffusion of thought: Chain-of-thought reasoning in diffusion language models. In Neural Information Processing Systems.

[38] Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. (2025). Dream 7b.

[39] Yu, L., Shi, B., Pasunuru, R., Muller, B., Golovneva, O. Y., Wang, T., Babu, A., Tang, B., Karrer, B., Sheynin, S., Ross, C., Polyak, A., Howes, R., Sharma, V., Xu, P., Tamoyan, H., Ashual, O., Singer, U., Li, S.-W., Zhang, S., James, R., Ghosh, G., Taigman, Y., Fazel-Zarandi, M., Celikyilmaz, A., Zettlemoyer, L., and Aghajanyan, A. (2023). Scaling autoregressive multi-modal models: Pretraining and instruction tuning. ArXiv, abs/2309.02591.

[40] Yuan, S., Liu, Y., Yue, Y., Zhang, J., Zuo, W., Wang, Q., Zhang, F., and Zhou, G. (2025). Ar-grpo: Training autoregressive image generation models via reinforcement learning.

[41] Zhang, Z. and Zhang, A. (2023). You only look at screens: Multimodal chain-of-action agents. ArXiv, abs/2309.11436.

[42] Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. J. (2023). Multimodal chain-of-thought reasoning in language models. Trans. Mach. Learn. Res., 2024.

[43] Zhao, J., Wei, X., and Bo, L. (2025). R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. ArXiv, abs/2503.05379.

[44] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685.

Introduction

While we can find a trove of literature on the history of writing (Gelb 1952, Gaur 1992, Fischer 2001, Rogers 2005, Schmandt-Besserat 2014), there are surprisingly few to no works explicitly dealing with the history of text.\(^1\) There is no established answer to the question: What were the earliest texts ever? For example, the results from this question being submitted to search engines such as Google or Bing tend to conflate the meaning of writing, text, and literature. Some results show that Sumerian scribes from 3200 BC wrote on clay tablets in cuneiform, most likely ancient shopping receipts (Alex 2019). Another result leads to the Kesh Temple Hymn and the Instructions of Shuruppak from around 2600 BC, which are considered to be an ode and a piece of advice (Andrews 2023).

But were these the firsts texts to be properly spoken of? The goal of this paper is neither to identify early writing nor early literature. Instead, I explore which texts were first created by humankind. However, a lot remains open to what we define as text. It is demonstrated throughout the paper that a plurality of definitions of text also determines the answer to the previous question. It is concluded that we cannot settle on one document as the earliest text in history. That said, an administrative piece from present day Iraq dating to 3100 BC suggests that among the earliest texts were contracts.

This paper contains the following sections: I approach text by recapitulating the history of writing because it provides us with the context from which texts have arisen. In the following section, I present three definitions of text: 1) the minimal definition that allows for many writings to be texts, as long as they are cohesive, 2) the most common definition that sees texts as a unified and coherent composition of sentences, serving a communicative purpose, 3) a narrow definition by de Beaugrande and Dressler (1981), who establish seven criteria for identifying text as such. Their criteria are the following: 1) cohesion, 2) coherence, 3) intentionality, 4) acceptability, 5) informativity, 6) situationality, 7) intertextuality. A short discussion of some shortcoming of these definitions is presented next. I discuss the length of texts as well as the utility of pragmatic features for identifying texts. Finally, I round up this paper with analyzing two pieces of writing. The first is an inscription on a vessel from 3150 BC, and the second is a clay tablet with a narrative about a slave trade that involved multiple parties from 3100 BC. These writings are among the very first in human history.

Recapping the history of writing

Before looking at the historic development of writing, we need a better understanding of what writing is.\(^2\) To achieve this, it may be advantageous to divide the writing-history community into two distinct camps, each of which will provide us with an understanding of writing: There are 1) those, who have a loose definition of writing (Gaur 1992, Schmandt-Besserat 2014), and 2) those, who have a strict one (Fischer 2001, Rogers 2005). The loose camp sees writing as a means of storing and communicating information. Gaur proclaims: “All writing is information storage.” (Gaur 1992: 15) The strict definitions of writing tie the concept further to “the systematic arrangement of significant vocal sounds” (Fischer 2001: 12). So, the strict understanding of writing sees it as the representation of actual spoken language whereas the loose definition would allow for many representational means to be writing.

It is generally agreed by both camps of writing historians that writing has been separately invented in three regions: the Near East, China, and Mesoamerica. However, the earliest traces of writing can be found in Mesopotamia, that is, present-day Iraq, dating back to at least as early as 3200 BC (Schmandt-Bresserat 2014). The invention of writing in this region is also well documented, which is the reason why most scholars focus on the Sumer’s early writing systems, who inhabited Mesopotamia during this period.

Schmandt-Bresserat divides the overall development of writing in Mesopotamia into four phases (2014). The first period of storing information began with clay tokens (8000 - 3500 BC). During the following period, these tokens were translated to two-dimensional symbols (3500 - 3000 BC). This change of making information accessible over time and space is being followed by the first systems of phonetic signs (3000 - 1500 BC), which finally led to the alphabet. The alphabet is a very economical way of representing vocal sounds.

Among the first examples in the strict sense of writing were inscriptions on seals, and vessels which were found in the remains of tombs. These inscriptions did not deal with any merchandise in terms of numbers, but bore names phonetically written. With the ascendance of names, more writing examples emerged in funerary contexts: Status depicting descriptions of individuals were successively added as well as pleas (Schmandt-Besserats 2014). This process eventually led to documents of religious or royal nature, as the Kesh Temple Hymn from around 2600 BC.

Three definitions of “text”

To identify early writings as texts it is necessary to define what constitutes a text. That is, we need to understand the structure of text. Therefore, I will proceed with three definitions of text: the first of which is mostly text internal, that is, it focuses on features that are materially evident in the text. The following definitions increase in complexity and gradually incorporate text external aspects as well. By text external I mean pragmatic features, such as acceptability within communicative situations. For now, this discussion leaves out further dimensions that should ultimately be also considered, such as the media that have been used for text production and dissemination.

The reasoning presented in this paper is guided by a certain philosophical rigor. The definitions offered are intended to serve as conceptual foundations against which specimens of texts will subsequently be compared. Each definition is formulated so that its elements are both necessary and sufficient to identify a piece of writing as a text. Should any definitional condition fail to apply, it is assumed that the writing in question would not qualify as a text under that particular definition. Whether the authors whose definitions are employed here would themselves insist that their criteria must always be applied to determine textuality remains somewhat uncertain, as only Linke and colleagues (1996) have explicitly addressed the intended strictness of their formulations. Accordingly, I invite the reader to regard the following definitions as idealized instruments for the purpose of investigating the earliest texts in human history.

1st definition: Halliday and Hasan define text as follows: “’Text’ is used to refer to any passage, spoken or written, of whatever length, that does form a unified whole” (1976: 1). This definition is being accompanied by an intuitive one. They claim: “We know, as a general rule, whether any specimen of our own language constitutes TEXT or not” (Halliday and Hasan 1976: 1). What matters is that text in its most common form should be distinguished from “a collection of unrelated sentences” (Halliday and Hasan 1976: 1). But that is not to say that any text must consist of at least two sentences. A text is rather “a semantic unit” that can also be found in some proverbs or announcements (Halliday and Hasan 1976: 294). But what constitutes its unity?

For Halliday and Hasan (1976), unity in texts is produced through cohesion. Roughly put, cohesion is what connects a set of descriptions, appeals, and/or propositions. Consider their example: “Wash and core six cooking apples. Put them into a fireproof dish.” (Halliday and Hasan 1976: 2) These sentences form a unit because they are interlaced through the anaphoric pronoun “them”. We can only make sense of the latter sentence by linking the pronoun to the apples from the previous instruction. In this manner, the two sentences constitute a cohesive text.

For Halliday and Hasan (1976), cohesion is not only produced with referential expressions, but also through further linguistic devices, such as repetition, substitution, ellipsis, conjunction, and lexical cohesion. For example, sentences can be linked by having several statements that circulate around the same object, let us assume this would be “car”, which is being referred to either repeatedly in another sentence or by a substituting expression, like “vehicle”. In this case, we would perceive unity in multiple clauses or sentences by identifying the same object at hand. Another device for cohesion according to Halliday and Hasan (1976) is the ellipsis. Consider the following example: “Ida orders a scoop of chocolate ice cream. Ada orders two.” These sentences constitute a unit because we assume that the latter also refers to ice cream, albeit the fact that nowhere it is stated that “two” would imply two scoops of chocolate ice cream. Cohesion through conjunction is created with connecting expressions such as “if… then…”. They explicitly guide the text recipient in putting propositions into relation. Finally, Halliday and Hasan argue that cohesion also results from lexical proximity of words, that is, words such as “people”, “person”, “man”, “woman”, “child”, etc., which fall into the same category of a general class of human nouns (Halliday and Hasan 1976: 274). One sentence may mention one of these, while another sentence may address a lexically related one, allowing the reader to connect the sentences through such lexical proximity.

Linke and colleagues (1996) emphasize that cohesion is necessary for identifying a text, but not sufficient. According to their view, cohesion only covers the superficial structure of texts, that is, the features that are materially evident in clauses and sentences. To differentiate a text from a loose collection of sentences, we would further need an understanding of the underlying deep structure.

It is suggested that we have text if its structure is also coherent (Linke et al. 1996). Coherence is a result of everyday knowledge or supra textual knowledge, which allows the reader to organize the textual elements into a meaningful whole. Consider the following example: “Hans won’t come to the conference. He’s sick.” (Linke et al. 1996: 225, my translation) Although the pronominal expression “he” connects these sentences cohesively, Linke and colleagues believe that its unity is also constituted by our need to relate its parts according to the most probable meaning that we can assign to them (Linke et al. 1996: 225). They argue that we would understand the two sentences in terms of a causal relation: Hans won’t come to the conference because he is sick. This is so because we read such sentences against the backdrop of our world knowledge where a causal relation of two such events would be the most coherent explanation. This will bring us to the second definition of text.

2nd definition: “A text is a complexly structured, thematically and conceptually interrelated linguistic unit with which a speaker performs a linguistic action with recognizable communicative meaning” (Linke et al. 1996: 245, my translation).\(^3\) When Linke and colleagues speak of “structured, thematically and conceptually interrelated linguistic unit”, their notion of unity should be explained with cohesion as well as coherence. For example, unity may be created with certain structures, such as expected behavioral patterns that are manifest in the text composition, or in terms of conceptual proximity, for example by using lexically cohesive words. A theme running through the text would be one explicit dimension which creates coherence; it is another means for the recipient to see the text as a meaningful whole by identifying an idea that pervades the text.\(^4\)

Linke and colleagues (1996) add that texts have a pragmatic side. A text must also be seen as “a linguistic action”. According to their definition (Linke et al. 1996: 246), at least three actions would be identifiable that can be realized using texts, fulfilling the following communicative functions: 1) texts describe objects, situations, or events; 2) texts express the emotions and intentions of their authors; 3) texts are appeals to their recipients with the aim to produce a particular reaction. For instance, a construction manual can contain texts, which describe actions that must be performed to build an object. A love letter expresses its authors devotion to someone. The “park rules” of Potsdam’s Park Sans Souci are meant to prevent visitors from behavior that would damage the property or annoy fellow visitors. In this manner, the text appeals with the aim to elicit a certain behavior.

3rd definition: De Beaugrande and Dressler (1981) propose the most complex definition of text. They argue that “text is a communicative occurrence that meets seven criteria” (de Beaugrande and Dressler 1981: 3, my translation). These criteria are the following: 1) cohesion, 2) coherence, 3) intentionality, 4) acceptability, 5) informativity, 6) situationality, 7) intertextuality. The authors are convinced that they can be spelled out for each text.

I will only briefly summarize 3-7 because de Beaugrande and Dressler’s understanding of cohesion and coherence resembles the previous explanations of these concepts. (3) By mentioning intentionality in their definition, de Beaugrande and Dressler also emphasize the speech-act-theoretical side of texts. The purpose of each text is to achieve a specific objective, such as commemorating someone (de Beaugrande and Dressler 1981: 9). (4) Their notion of acceptability is supposed to refer to the attitudes of text recipients. The idea is that texts can only be perceived as such when they are aligned with the expectations and attitudes of their addressees (de Beaugrande and Dressler 1981: 9). (5) Closely related to acceptability is the concept of informativity. For de Beaugrande and Dressler, texts would always be informative. To what degree a text is informative can be measured by pondering the following questions (de Beaugrande and Dressler 1981: 10). Is the text expectable? Are the text’s elements unknown, partially known, or known? Are any of the text’s elements improbable or probable? (6) Situationality is about what makes a text relevant to a particular conversational goal (de Beaugrande and Dressler 1981: 12). In that, it can be seen as another guiding principle, helping to realize the text’s intention(s). Situationality is especially relevant when we consider the context of text. Finally, (7) de Beaugrande and Dressler introduce intertextuality as a relevant criterium for texts. Its notion is tied to informativity and acceptability. Intertextuality describes the degree to which a text requires prior knowledge of other texts in order to be acceptable to its intended audience and to be appropriate for use in a given context (de Beaugrande and Dressler 1981: 13).

By mentioning intertextuality, de Beaugrande and Dressler (1981) emphasize the possibility of separating texts into distinct categories. They speak of text sorts as “a class of texts that are expected to have certain properties for certain purposes” (de Beaugrande and Dressler 1981, 188, my translation). However, they abandon the pursuit of precise classification of texts, arguing that a typological approach must remain vague due to the presence of too many indeterminate examples of texts (de Beaugrande and Dressler 1981: 193). To get an idea of the difficulty of classifying texts, Dimter (1981) has created a non-exhaustive list of possible text sorts, with at least 500 different text type designations that can be found in German. That said, linguists, such as Sandig (1972), have nevertheless attempted to come up with taxonomies.\(^5\) While the limit of these now appears to be broadly accepted, non-exhaustive text classification has gained new popularity in Computational Linguistics since the emergence of classification tasks as a dominant theme in machine learning (Stede et al. 2006).

Some caveats to the definitions

Having discussed three definitions of text, which are increasingly complex, we need to consider how these are helpful with identifying historic writings as texts. The definitions provided by 1) Halliday and Hasan (1976), 2) Linke and colleagues (1996), as well as 3) de Beaugrande and Dressler (1981), appear to have all their advantageous and disadvantageous. I have already mentioned Linke’s critique of Halliday and Hasan’s view. The remainder of this section is limited in scope and may only address a few further critical points. I would like to discuss size of texts, the limits of speech-act-theoretical aspects for identifying early texts, and the utility of some of the criteria by de Beaugrande and Dressler (1981), such as situationality.

Whether a few words, a clause, a sentence or several ones, an inscription, or a collection of lines may be considered as text is also tied to the question how large a text is. Is a word text? Or do we need two words, noun and verb, to deal with text? To what degree does size matter? Halliday and Hasan’s definition of text deviates from the notion that text is something bigger than a sentence. For they believe that proverbs like a picture is worth a thousand words could be interpreted as text. However, many scholars from antiquity until today have defined text as something that follows the sentence in a hierarchical order. For example, Scherner (1996) argues that Plato had already identified text as the ultimate unit of language, which is larger than a sentence, with the caveat that Plato used the Greek word lógos (λόγος) to speak of texts (Scherner 1996: 106).\(^6\) Heringer, a contemporary scholar, likewise views text as the entity that succeeds the sentence in size (Heringer 2015: 11).\(^7\) It is worthy keeping in mind what size we settle on may also decide whether some writings can be termed text.

Linke and colleagues (1996) as well as de Beaugrande and Dressler (1981) highlight extra textual, that is, pragmatic features of text. I wonder to what degree some of these features are relevant for identifying early texts in history due to our partial knowledge about concrete customs, worldviews, and cultures of past times. What if we cannot determine the intention of an ancient piece of writing? Early literature, such as the Kesh Temple Hymn, might have served religious, but also magical beliefs of the time that we do not know anymore or for which we have no proof. For these matters, we might also misinterpret some texts. Some phrases could seem non-sensical to us, although they had transported a coherent message for the people of their times. Perhaps they carried more than what we can read today.

In similar fashion, de Beaugrande and Dressler’s notion of situationality can be questioned. While it can help to understand the structure of those texts confronting contradictory or unexpected perspectives, it is doubtful whether all texts require situationality for them to be intelligible. This will be more concrete when we think of the examples of traffic control signs given by de Beaugrande and Dressler such as “Slow down. Kids playing.” (1981: 12) The sentences make especially sense as soon as we imagine ourselves seeing the sign next to a busy street while driving a car. We can say that the situation would be conducive for our understanding. However, if we were reading it within a theatre, we would still be able to discern its message. We only seem to be at odds if any hypothetical context for a given piece of communication is foreign to us. However, if that is the case, we may already encounter issues with the actual semantic content of such statements, which would be more fundamental than considering the situation in which a message is intended to communicate. Is situationality then a necessary requirement for identifying texts?

When pondering the need of situationality for text identification, we may also ask whether texts would always be inter-textual: It may sound simple, but identifying the earliest text implies that there is no canon of other texts to which it can be related. Thus, according to de Beaugrande and Dressler, this would suggest that the first recipients of text were unaware of the fact that they were reading text.

Identifying early text candidates

To discuss which pieces of writing are text, I will present two of the earliest discovered writing specimens in human history. These are taken from Woods et al. (2015), who present a systematic and chronological overview of early writing. The first specimen is an inscription and the second an administrative piece.

1.) The funeral inscription is from Egypt, Abydos, Umm el-Qa’ab, ca. 3150 BC, Dynasty I reign of Djer, Egyptian 0 period. It is written on a cylindrical vessel that was found at the tomb of Djer. It reads:

ḥtp-Nἰt

which can be translated as “May Neith be satisfied.” (Woods et al. 2015: 128) The inscription is a name, bearing a reference to the goddess Neith; it was popular for royal females of that time (Woods et al., 2015: 128).

2.) The administrative piece is from Iraq, potentially Larsa, ca. 3100 BC, Uruk III period and was found as can be seen in Woods et al. (2015: 80). The piece describes how twelve slaves are being transferred to new owners. The potential text in modern English reconstructed could sound like what follows:

Obverse: Slaves are sold to the cultivators. Their names are Nim, and others. Slaves, who are called En, Bu Du, Gul, are sold to Maran. Via Parmud and via the pig herder of Adab.

Reverse: The subtotal of six slaves via the pig herder of Adab. Subtotal of six slaves via Parmud. Twelve slaves are for Maran and Ente.

It is not clear though what the order of the sides is, and if one should be read before the other.

When discussing these examples, the idea of Halliday and Hasan (1976: 1) to test our intuition about whether a specimen constitutes a text is appealing to me. Intuitively, the shortest piece of writing would probably be the most controversial example since it also defies the common view that text is something more than a single sentence or word. Therefore, we shall begin by examining whether or not ḥtp-Nἰt sufficiently fulfills any of the previously discussed definitions.

The funeral inscription appears to have two semantic components: On the one hand, it is a proper name, on the other, the name has meaning with proverbial traits: “May Neith be satisfied.” If read as proper name, the inscription would not fulfill the definitions of text because it lacks elements that are brought to unity through any of the possible features that text appear to have. But can ḥtp-Nἰt be really read as a proverb? The Merriam-Webster dictionary defines a proverb as “a brief popular epigram or maxim“ (Merriam-Webster 2023). While that leaves room for interpretation, the expression might for example implicitly carry a maxim that the goddess Neith should always be satisfied, my reading of ḥtp-Nἰt would be closer to something like the Arabic salam alaykum (ٱلسَّلَامُ عَلَيْكُمْ), which accompanies greetings. In similar fashion, ḥtp-Nἰt could be an accustomed expression, whose function was less tied to its meaning to satisfy a goddess, but rather a blessing that was used for specific purposes. In the Egyptian context, it might have taken on a new function, by particularly naming aristocratic females with this expression. It might be no coincidence that salam alaykum also reappears in names, such as Jerusalem, which supposedly carries salam. Therefore, if ḥtp-Nἰt was not a maxim, but close to a blessing, such as salam alaykum, we might not want to speak of it as a proverb.

Halliday and Hasan (1976) go as far as saying that even announcements fall into the category of text. For them, this would also include phrases such as “No smoking” or “For sale” (Halliday and Hasan 1976: 294). However, it is challenging to identify the cohesive characteristics of these types of phrases. In similar vein, if ḥtp-Nἰt is interpreted as being similar to an announcement, we would have to ask what its cohesive elements are?

Cohesion of announcements could be derived from context. The phrase “For sale” does make sense to us when read in front of a house. We would see it as a unit that would not require any additional elements to be understood. Likewise, ḥtp-Nἰt may constitute a unit in particular situations where the expression might have been used, like salam alaykum is used in greetings. The problem with deriving unity in this manner would be for the first definition of text that it goes beyond Halliday and Hasan (1976) since they do not consider pragmatics as cohesive factors.

With Linke and colleagues (1996), we may ask: What makes the name – as saying – coherent? Again, it appears that the best shot would be by considering pragmatics. Let us assume that the phrase was not only used in names, but also in situations to express devotion to the goddess, as I speculate. Using it could have been a way to show the authors’ or parent’s commitment to the community. People might have pronounced ḥtp-Nἰt as a statement like “Jesus is your hope!”, which can be read on billboards in the U.S. If that is the case, ḥtp-Nἰt may have been an act of communication within an expected context that was coherent against the backdrop of the community’s values and customs.

However, with our lack of resources regarding the contexts of the people from Abydos, it is difficult to settle these questions. Therefore, I believe that ḥtp-Nἰt is only text when we stretch our interpretation of the inscription to fit the criteria set out by Halliday and Hasan (1976) and Linke and colleagues (1996). We would need to agree that announcements or short statements are text. Then we must decide if names that also have an additional meaning can be interpreted as proverbs or announcements. If none of this applies, we must consider whether a simple name on a tomb, without any additional proverbial meaning, qualifies as a text. And the answer to this consideration would be no, because not every written word constitutes a text – otherwise, the definitions of text would become redundant.

Given that de Beaugrande and Dressler (1981) have the most complex definition, it is reasonable to say that ḥtp-Nἰt would need to clearly fulfill the first two definitions to justify an extensive analysis based on their definition of text.

The second piece of writing deals with the trade of twelve slaves to new owners, some of whom are called by their names. To start our analysis, we should consider whether the passages on the clay tablet create a unified whole through cohesive elements. The most apparent cohesive feature is repetition: The sentences summarize the transaction’s details in multiple ways, such as by introducing additional information about the parties involved in the sale of the slaves. We can also observe substitution. The slaves are being referred to as slaves as well as by their names. In a similar manner, the new owners are mentioned as cultivators and by their names, Maran, and Ente. We may also deal with lexical cohesion. Cultivator and pig herder could be grouped under the hypernym of agricultural occupations. In brief, we have evidence for several cohesive structures.

The sentences are coherent in multiple ways, too. For example, they have a theme, the sale, which runs through the passages, allowing us to consider that the trade must have been mediated by pig herders when we read “via the pig herder of Adab”. Also, that the addition of six slaves adds up to twelve is coherent. We do not know, in what order the tablets were supposed to be read, but we can imagine that the obverse with more details is being summarized by the reverse, which has less information. Someone using the tablets might have looked at the summary to remember the transaction, and if they had forgotten the details, they could have looked them up on the obverse.

Given that we can identify several cohesive as well as coherent elements, the second piece of writing qualifies as text, according to Halliday and Hasan (1976) and Linke and colleagues (1996). Does it also meet de Beaugrande and Dressler’s (1981) standard?

If we take for granted that the slave trade piece is cohesive and coherent, we shall examine the following points in more depth: intentionality, acceptability, informativity, situationality, intertextuality. The intentionality of the slave trade writing could be at least twofold. Its purpose could have been to describe and record the transaction that took place so that the rulers and fellows of the community knew what had happened. The piece could also be of contractual nature. A contract is a declaration, establishing and ensuring (the property) rights of those parties mentioned in the contract. Even though slavery is not acceptable today, it is fair to assume that it was acceptable at the time of the transaction. Since slavery was not uncommon in Antiquity, the content of the tablets was also expectable and the mediation of such a trade by livestock owners probable. Those, who were familiar with the land and community where the trade took place, probably knew the owners and mediators by name, too. In short, we can assume that the sentences on the tablets were also informative to fellow members of their community although they are less so for us.

It is more difficult to determine if the description of the trade fulfills de Beaugrande and Dressler’s notion of situationality. Because we know little of the context of the writing, we are forced to speculate what situations used to accompany it in ancient Iraq. But what we know is that it was probably found around Larsa. At the time of its creation, it might have been stored in an administrative context that allowed the community of Larsa to settle property claims. In this regard, it might have been accompanied by similar pieces that we do not know of, and which also documented similar transactions. In this manner, our specimen could have been associated with these texts, potentially allowing the reader to understand its nature as a contractual settlement, which assured the rights of the mentioned parties.

Since we have evidence that the second specimen analyzed in this paper is cohesive and coherent, and because we can also spell out intentionality, acceptability, informativity, situationality, and intertextuality features, I conclude that the document from Larsa meets de Beaugrande and Dressler’s seven criteria (1981). So that it can be considered an early text in human history given that is from 3100 BC. If it were the case, that the text not only documented a slave transaction, but was also regarded as a binding arrangement, then we can deduct from this instance that among the earliest texts were texts of contractual nature.

On a final note, the possibility that the slave transaction could be a contract should not be interpreted as implying that early texts were the result of accounting systems. The idea that accounting is at the heart of everything is often linked to the evolution of writing and would be true for any broad understanding of writing, such as knotting techniques or clay tokens, which were used to count livestock or goods. But such a mono-interpretation of the origins of writing already crumbles if writing in its narrow sense also emerged through phonetically written names in funeral contexts. I believe that early texts should be neither reduced to accounting. As has been shown in this paper, some of them were documents of early legal structures that allowed the community order.

In sum, this paper explored the origins of the earliest texts in human history. After presenting a historical overview of the development of writing, it proposed three definitions of text, ranging from a broad, minimal definition based on cohesion to a narrower, seven-criterion framework encompassing cohesion, coherence, intentionality, acceptability, informativity, situationality, and intertextuality. Applying these definitions to early examples of writing, the analysis found that while no definitive first text can be identified, the contract from 3100 BC in ancient Iraq meets the criteria of all definitions sufficiently to be considered an early instance of full textuality.

Footnotes

\(^1\) Some of the afore mentioned authors touch upon early texts (e.g. Schmandt-Besserat 2014), but no one exclusively writes about the history of text. Greetham (1999) has a promising chapter called “The History of the Text”, but it deals with the phenomenon of texts being recycled in different literary discourses.

\(^2\) Writing should not be confused with script. Script would be in cuneiform, italic, bold etc. When I refer to writing, I mean an overall system of signs or symbols.

\(^3\) Bußmann’s definition of text is very similar (Bußmann 2002: 683). Also, Brinker’s (2010: 19-20) summary of text characteristics is close to Linke and colleagues (1996).

\(^4\) A theme comprises the “content-semantic guideline and quintessence of the text”, which would be still perceptible when a text is radically cut (Linke et al. 1996: 237). For another discussion of themes in texts, please see Brinker (2010: 11).

\(^5\) Creating a typology of texts starts with the difficulty of finding the right angle from which texts can be classified. Should they be ordered according to a) communicative criteria, b) domains where texts are being used (such as religion, journalism, art, politics, law, science etc.), or c) according to their types of speech acts (informative texts, declarations, appeals, obligations, contacts) (Heringer 2015: 128)? Another attempt of classifying texts could also divide texts according to text internal features vs. text external ones. Internal features could be layout, length, themes, lexicon, grammatical constructions and so on. The text external features could order texts according to situations where they are being used most often. Heringer (2015: 129) suggests that any serious typology of texts should fulfill the following benchmarks: A typology should be complete, selective, and applicable.

\(^6\) Despite the Latin origin of the word “text”, it was only rarely used. Quintilian speaks of text to refer to language being “weaved” together (Scherner 1996: 109).

\(^7\) Gülich and Raible (1977), among others, explicitly agree with Halliday and Hasan that a single sentence can be text (Gülich and Raible 1977: 51).

Bibliography

Alex, Bridget. 2019. What the Earliest Texts Say About the Invention of Writing. Discover. https://www.discovermagazine.com/planet-earth/what-the-earliest-texts-say-about-the-invention-of-writing (accessed 26 May 2023).

Andrews, Evan. 2023. What is the oldest known piece of literature? History. https://www.history.com/news/what-is-the-oldest-known-piece-of-literature (accessed 26 May 2023).

de Beaugrande, Robert-Alain, and Wolfgang U. Dressler. 1981. Einführung in die Textlinguistik. Tübingen: Max Niemeyer Verlag.

Brinker, Klaus. 2010. Linguistische Textanalyse. Eine Einführung in Grundbegriffe und Methoden. Berlin: Erich Schmidt Verlag.

Bußmann, Hadumod (ed.). 2002. Lexikon der Sprachwissenschaft. Stuttgart: Kröner.

Dimter, Matthias. 1981. Textklassenkonzepte heutiger Alltagssprache: Kommunikationssituation, Textfunktion und Textinhalt als Kategorien alltagssprachlicher Textklassifikation. Tübingen: Max Niemeyer Verlag.

Fischer, Steven R. 2001. A History of Writing. London: Reaktion Books.

Gaur, Albertine. 1992. A History of Writing. New York, London, Paris: Cross River Press.

Gelb, Ignace J. 1952. A Study of Writing. Chicago, London: University of Chicago Press.

Greetham, David C. 1999. Theories of the Text. Oxford: Oxford University Press.

Gülich, Elisabet, and Wolfgang Raible. 1977. Linguistische Textmodelle: Grundlagen und Möglichkeiten. München: Wilhelm Fink Verlag.

Halliday, Michael A. K., and Ruqaiya Hasan. 1976. Cohesion in English. London: Longman.

Heringer, Hans J. 2015. Linguistische Texttheorie: Eine Einführung. Tübingen: A. Francke Verlag.

Linke, Angelika, Markus Nussbaumer, and Paul R. Portmann. 1996. Studienbuch Linguistik: ergänzt um ein Kapitel »Phonetik und Phonologie«. Tübingen: Max Niemeyer Verlag.

Merriam-Webster. 2023. “Proverb.” Dictionary, Merriam-Webster. https://www.merriam- webster.com/dictionary/proverb (accessed 26 May 2023).

Rogers, Henry. 2005. Writing Systems: A Linguistic Approach. Malden, Oxford, Carlton: Blackwell Publishing.

Sandig, Barbara. 1972. Zur Differenzierung gebrauchssprachlicher Textsorten im Deutschen. In Elisabeth Gülich and Wolfgang Raible (eds.), Textsorten. Differenzierungskriterien aus linguistischer Sicht, 113–124. Frankfurt a. M.: Athenäum-Verl.

Scherner, Maximilian. 1996. „TEXT“: Untersuchungen zur Begriffsgeschichte. In Archiv für Begriffsgeschichte, 1996, Vol. 39 (1996), 103-160.

Schmandt-Bresserat, Denise. 2014. Writing, Evolution of. In James D. Wright (ed.), International Encyclopedia of the Social Behavioral Sciences (Second Edition), 761-766. Oxford: Elsevier.

Stede, Manfred, Heike Bieler, Stefanie Dipper, and Arthit Suriyawongkul. 2006. SUMMaR: Combining Linguistics and Statistics for Text Summarization. Proceedings of the 17th European Conference on Artificial Intelligence, (ECAI-06), Riva del Garda, Italy: 827-828.

Woods, Christopher, Emily Teeter, and Geoff Emberling (eds.). 2015. Visible Language: Inventions of Writing in the Ancient Middle East and Beyond. Chicago: Oriental Institute of the University of Chicago.

Introduction

Langdon Winner argues that artifacts have politics. They are, as products of human activities, “ways of building order in our world” (Winner 2020, p. 28). In other words, the decisions we make to create an artifact materialize in the resulting product. Winner claims, for example, that the overpasses arching over the highway from New York City to Long Island are so low hanging for a reason. Apparently, the architect, who oversaw these structures, Robert Moses, did not want to have racial minorities or the low-income classes to visit the retreats of the rich on Long Island. That is why he made sure that it was impossible for vehicles such as buses to pass under the bridges, which meant that only those people who could afford a car at the time had access to Long Island. In her book, Feminism Confronts Technology, Judy Wajcman (1991) critically expands Winner’s thesis from a feminist perspective. She outlines that many technologies are built by a small group of men, who are usually employed by big companies. Their technologies are hence imbued with men’s world views as well as with the aim of producing a profitable product.

Having realized to what extent technologies are political in these senses, it is reasonable to assume that gender politics has an impact on the development of humanoids, too. A humanoid is a robot with the appearance and qualities of a human being. As with overpasses, the ordering of humanoids must depend on the views and ideas of their designers, programmers, engineers, and so on. This is especially the case with social and care robots whose appearance and behavior are significant for human-robot interaction. The creators – be it consciously or unconsciously – will incorporate into their design whatever they believe to be paramount for successful socializing. We should therefore ask how humanoids are gendered. There is the danger that uncritical design and programming, or even deliberately exclusive gender conceptions, reproduce sexist ideas and outdated stereotypes.

I will begin with Judith Butler, R. W. Connell, and Jack Halberstam, and discuss some elements of gender as a construct (1). I will then attempt a qualitative gender analysis of SoftBank’s Pepper (2). Lastly, I will argue that it would be counterproductive to simply neutralize and defuse gendering in robotics because gender distinctions seem to be inescapable (3). For this reason, I will propose that genders need to be openly embraced by those who design, engineer, program, and build robots. For this endeavor, in the last section, I will identify several gender specific challenges for robotics (4). Here I will focus on the concrete building process discussing, for example, whether hardware and software components of robots might reinforce sex and/or gender binaries.

A rough sketch of gender

According to R. W. Connell “[g]ender is a way in which social practice is ordered” (Connell 2005, p. 71). For Connell this practice specifically relates to body structures as well as to the reproductive area. For example, a person wearing hot pants might be associated with what in most Western societies is conceived as the ‘feminine’. The ‘masculine’ as well as the ‘feminine’ are, however, contingent constructions. There is no reasonable justification, according to Connell, as to why the one gender is associated with this and the other gender with that. As simple as this might seem, it has been particularly difficult for human beings to escape such arbitrary stereotypes.

One explanation for gender differences is offered by Judith Butler (2007, 2011). With Butler, we can specify that genders as practices, or as she says performances, ought not to be understood as deliberate acts where one wakes up in the morning and decides what gender one has. We should rather understand gender as a way of responding to reiterated interpellations by authorities such as teachers, clergymen, cultural icons, parents, in brief, by those who have defined the dominant discourse of what is endorsed or forbidden. In other words, the construction of gender is the result of a normative process that can be difficult to escape.

Butler underlines that the maintenance of gender norms also involves violence. For the constraints established by norms produce not only the “domain of intelligible bodies, but […] as well a domain of unthinkable, abject, unlivable bodies” (Butler 2011, p. x). At the beginning of the 21st century those bodies that are intelligible are usually those which adhere to the binary notion of man and woman, having a heterosexual orientation. All bodies that do not conform to these standards – here one could think of transsexual bodies – have often been met with violence and oppression. For abject bodies are, explains Butler, no simple opposition to the intelligible ones. They are perceived as specters of the latter because they remind any conforming body “of its own impossibility” (Butler 2011, p. x).

Another key insight in Butler’s work is that the ongoing need for gender performances also makes subversive iterations as well as normative changes possible, given that with each repetition there is the chance to alter the course of the staged act. It is, for example, possible that men can also wear hot pants.

Theories of gender, especially those by Judith Butler, have been criticized in at least two ways. First, since for Butler there is no pre-discursive body, no natural sex, as some might say, people have criticized that her theory would underestimate the materiality of bodies. Jack Halberstam points out how “performativity as a theoretical rubric” could, for example, imply “in a transphobic way, that trans* gender is not real, material, authentic” (Halberstam 2018, p. 120). In her book, Bodies That Matter, Butler (2011) agrees, emphasizing that bodies should not be excluded from gender analyses. She explains that they are not less real; they need to be fed and washed, they feel pleasure and pain, and each body has its own material peculiarity. However, Butler insists that even if bodies have an undeniable material reality, it does not mean that we can altogether escape the discursive structures defining bodies.

In her later writings, Butler picks up the second critique, namely the fear that gender awareness has become an object for marketing. In the 1999 preface to her Gender Trouble, she expresses her concern by concluding that “subversive performances always run the risk of becoming deadening cliches through their repetition and, most importantly, through their repetition within commodity culture where ‘subversion’ carries market value” (Butler 2007, p. xxiii). Halberstam points out in a similar fashion that increased flexibility with respect to genders “may function as a part of new regulatory regimes” (Halberstam 2018, p. 18). Butler thus concludes that “to name the criterion for subversiveness will always fail” (Butler 2007, p. xxiii). Each historical, social, and cultural gender context needs its own evaluation and analysis.

With this sketch in mind, even with subversive gender politics, new forms of gender will not be safe from processes of exclusion. The way through which our identities are constituted is determined by a process of differentiating between bodies becoming intelligible against the backdrop of non-intelligible others. This theory is of dialectical nature. Therefore, some sort of rejection of the other, be it only in terms of a differentiation, seems to be a precondition for any gender. It is impossible that there will one day be an end to gender distinctions as some, including Stefan Hirschauer (2013), have stipulated.

Hirschauer’s argument for “undoing gender” is premised on the same assumption as Butler’s hopes for subverting gender. As gender is socially reinforced and individually performed, Hirschauer believes in the possibility of changing social and personal practice to create spaces where gender becomes irrelevant. However, if such spaces exist, they would be inherently fragile and vulnerable to the same authoritative interpellations that dominate mainstream culture. A critical stance on gender should therefore try to perpetually overcome these dynamics by dissolving those genders that sharply contradict with the putatively non-conforming ones.

Qualitative gender analysis of Pepper

Having realized the historical and power-imbued dimensions of genders, we have a basis for tracing gender constructions that feed into what Butler has called the intelligibility of bodies, that is, the process of discursively reproducing a certain type of gender politics. With Winner and Wajcman, we know that politics – and this includes gender politics – also has an impact upon our technologies. I would now like to attempt an analysis of how gender politics influenced the design and shape of the humanoid Pepper that was introduced by SoftBank Robotics in 2014.

There are, however, a few caveats that need to be mentioned beforehand. One difficulty in analyzing possible gendering apparent in humanoids lies in the danger of reinforcing prescriptive norms by saying, for example, that this is ‘masculine’ behavior or that looks ‘feminine’, because a critical approach would want to put exactly these attributes into question. For this reason, I think that an analysis should try to proceed in as neutral a way as possible by sticking to descriptions rather than prescriptions. Of course, this approach will also reach its limits as well and run the danger of disguising itself as unbiased when there is no view from nowhere.

Another point is that the following analysis leaves much to be desired because the accessible information is limited. Most of the development and designing sites for commercially marketed robots are not open to a critical public. The same holds true for some of the behind-the-scene features of robots such as speech recognition software, image processors, preprogrammed movements, possible reactions, etc., which would of course be very interesting for such an analysis since all of these aspects can be tilted in one way or another.

Now let me introduce Pepper. Pepper was 1.20 meters tall robot that weighed 28 kilograms and looked, according to one article, “like a person (except for the wheels where its legs should be)” (Glaser 2016). The manual introduces Pepper as a humanoid that “has some physical human resemblance […] but doesn’t pretend to be a human” (SoftBank Robotics 2017). Pepper had manga-like eyes, a print-on mouth, curvy hips and a comparatively small waist. Pepper’s breast was covered with a touch display that featured symbols like Pepper’s name, emoticons, or images.

The robot’s default configuration could be adjusted by users. For example, it was possible to provide Pepper with custom movements. Beyond its touchscreen, the default Pepper communicated through expressive gestures as well as speech. In the U.S. market, its voice varied in pitch – ranging from high tones in phrases like “Thank you. What a sweet thing to say.” to deeper tones in other contexts. The robot offered three voice styles, described in the manual as neutral, joyful, and didactic (SoftBank Robotics 2017). Pepper’s responses were accompanied by short ringtones and illuminated signals around its eyes and ears, indicating successful speech recognition. SoftBank Robotics recommended keeping the humanoid animated to avoid an uncanny appearance.

The humanoid’s responses in conversations seemed to be mostly humorous, entertaining, sometimes flirty, but also informative. In response to the question “tell me about yourself”, Pepper answered, for example: “My name is Pepper. I’m a humanoid robot and I’m 1.20 meters tall. I was born at SoftBank robotics. You can keep on asking questions if you want!” While answering other questions, Pepper explained that it would be called Pepper because this name was easy to remember and translated easily into other languages, or that it was four years old and from Paris. In response to the question: “You come from Paris?” Pepper answered: “I’m originally from Paris. Ahh Paris!” while raising its head, looking to the sky with one of the arms to its hips. If you asked Pepper how much it would cost, Pepper was quick to respond: “If you ask me, I’m priceless.” But Pepper’s range of answers was also limited. In response to: “Do you have any feelings” Pepper said: “I don’t understand. How about a taco?” Giving silly answers appeared to be Pepper’s default response when the robot did not know about the questions being asked. Overall, Pepper’s lingual capabilities were most likely powered by a rule-based language model that used automatic speech recognition and text to speech engines\(^2\) with a limited lexicon.

Pepper also offered an option to connect to a custom speech module based on Google’s Dialogflow platform. This platform enabled the creation of a conversational agent that guided Pepper’s communication using Google’s language models. The manual provided several recommendations on structuring Pepper’s conversations, including the importance of offering compliments or congratulations to users to make them feel appreciated. It also suggested that Pepper should take the lead in interactions, as most users were unfamiliar with robots and might feel shy.

Since Pepper was built for a specific form of employment, we need to assume that the robot was designed to fit its working contexts, which according to Kovach (2018) were businesses, shopping-malls, hospitals and doctor’s offices. So if we say that the robot was meant to take over service jobs, Pepper was employed in a sector where, according to World Bank statistics, the workforce was (and still is) predominantly female.\(^3\) In other words, Pepper’s occupation had a potential impact on the employment of women. Was this taken into consideration when the humanoid was built? It is difficult to tell. Marketers at SoftBank Robotics were meticulously careful to avoid presenting a uniform image of the robot’s purposes – there was neither clear hint at where the robot should work, nor were there any concrete gender attributes. For instance, there were no pronouns used in any of the commercials. The same holds true for most of the media attention the robot received. In her article for Wired, Glaser (2016) used the pronoun “it” while referring to Pepper.

For these reasons, I would not be sure how to label Pepper, if I had to attribute a gender to the humanoid. It would be easier to link certain of Pepper’s features to some established images and gender stereotypes. Pepper’s leg with wheels looked like a long dress covering the robot’s feet. And Pepper’s broad hips and slim waist as well as the robot’s round eyes made one think of rather feminine attributes. Also, the fact that robot had a screen on its breast, diverting one’s attention from the face to this part of the body was questionable. The humanoid’s manner of employing many highly accentuated gestures, as well as Pepper’s dry enumeration of facts, reminded one of masculine stereotypes. The same applied to some other physical features; for example, the robot had no hair. For me Pepper ranged between being a pet (think of the name) and (given Pepper’s size, limited answers and somewhat playful responses) a child. Thus, all in all, Pepper appeared as a potpourri of gendered features.

The conscientious gendering of robots

After outlining the concept of gender and identifying gendered traits in Pepper, we can agree that developing non-sexist technologies requires critically examining problematic stereotypes and gender norms to create new avenues for emancipation. However, with an emphasis on the dynamics of exclusion in genders, I believe that a critical approach to humanoids should not aim to simply neutralize or erase gender. Gender distinctions will inevitably resurface in some form. Empirical studies show that the perceived connection between users and robots equipped with social cues – such as gender stereotypes – shapes human-robot interaction (Powers et al. 2005). For instance, in Powers et al.’s study (2005), participants were less talkative when interacting with a robot displaying masculine attributes than with one exhibiting feminine traits. This suggests that the influence of gendered cues is often prevalent.

However, another study only reports an effect of gender when the experiment’s humanoid NAO was introduced with the gendered pronouns “he” or “she”, but “no main effects of robot gender on participant rating” for Aldebaran’s humanoid with respect to engagement and relaxation in a different setting (Rea et al. 2015). This is an interesting finding, which raises the question of to what extent a childlike-looking humanoid like NAO, which was roughly 60 cm tall, is perceived in contrast to Pepper, and in general, how humanlike a robot would need to look and behave to incur gendered assumptions on the users’ side.

However, another study only reports an effect of gender when the experiment’s humanoid NAO was introduced with the gendered pronouns “he” or “she”, but “no main effects of robot gender on participant rating” for Aldebaran’s humanoid with respect to engagement and relaxation in a different setting (Rea et al. 2015). This is an interesting finding, which raises the question of to what extent a childlike-looking humanoid like NAO, which was roughly 60 cm tall, is perceived in contrast to Pepper, and in general, how humanlike a robot would need to look and behave to incur gendered assumptions on the users’ side.\(^4\)

If we consider the gender theoretical groundwork, then the inherent dynamism of intelligible gender identities implies that non-intelligible, even abject, bodies will always be necessary. This means that the dialectical construction of identity will ultimately prevent the complete neutralization of gender. However, contrary to this assumption, Stanford’s Gendered Innovations initiative explores designing genderless robots as a potential solution.\(^5\) But even in their suggestions for non-gendered robots, they mainly refer to child-like or non-humanoid examples.

To escape the violence of gendering, humanoids should hence be continuously re-gendered, incorporating ever-evolving designs that reflect a critical engagement with gender. Similarly, Tanja Kubes argues that after de-gendering robots, redesigning them can unlock “an enormous potential for exploiting the emancipatory possibilities of sexbots” (Kubes 2019, p. 70). This ongoing exploration of possibilities should serve as a guiding design principle.

In fact, Pepper might already be a good example of how gendering in robotics can be done. Pepper’s design incorporates elements traditionally associated with both masculine and feminine traits. While the large eyes and high-pitched responses might be read as feminine, the lack of hair and neutral voice settings avoid clear gender assignment. This ambiguity may have been intentional, allowing SoftBank to market Pepper as universally relatable. However, without explicit confirmation from SoftBank, it remains unclear whether this design was a strategic decision or an unintended consequence of broader robotic trends.

The qualitative analysis can be supported by a study on perceived trust in robots involving Pepper. Bryant et al. (2020, p. 19) found that “perceived occupational competency” was a stronger predictor of trust in Pepper than perceptions based on gendered traits. In other words, Pepper’s gendered features played a secondary role in the experimental tasks, indicating that the humanoid did not reproduce stereotypical gender characteristics.

Challenges for critical gendering in robotics

To facilitate conscientious gendering in robotics, I have finally identified some challenges that further need to be considered.

(A) Can humanoids have something like a real gender, given the complexity of the concept? At the outset, I think that one can easily underestimate to what extent genders are tied to the human condition to which, for example, we would also attribute freedom and intention. We could otherwise not change the course of our gender performances, nor could we even think of changing them. It would therefore be exaggerated to speak of robots having genders since contemporary models do not have anything similar to what we call freedom and will. As with each technology in principle, it would thus be more appropriate to say that humanoids are gendered. But in light of the fact that many roboticists have set their heart on the idea that their creations will one day act autonomously, and that they would like to make them more human-like, we need to consider whether this would require that humanoids are also equipped with (or whether their development will inevitably lead to robots with) capacities facilitating the possibility of them having genders. For robots without genders would be not human-like, and robots with something like freedom and will might inevitably assume a gender.

(B) Do humanoids with their hardware and software reproduce the still established binary of sex/gender? For one thing, the analysis of Pepper has shown that design and engineering can help introducing new gender representations. From this perspective, they answer would be: not necessarily. For another thing, I think that one could still arrive at the conclusion that humanoids are good metaphors for how there is something material, such as the hardware, on the one side and something mind-like, such as the software, on the other side. This, in turn, might lead one to believe in what gender theory has tried to criticize, namely, the traditional binaries of sex/gender, or the view that our bodies are physically sexed, like machines, and whatever we consider as gender is socially programmed, like software. But this of course misses how interrelated and interdependent bodies and socially constructed identities are.

(C) How important is transparency in helping to raise awareness for gendering? Based on my brief analysis of Pepper, it is evident that robotics developers should be required to disclose key aspects of the development process. This includes identifying those involved in creating the robot, clarifying the goals pursued, and detailing the design choices and implemented features. Additionally, the robot’s speech capabilities should be explicitly linked to the underlying communication models, which, in turn, should be accompanied by data statements, as recommended by Emily Bender and Batya Friedman (2018).

While transparency in development (such as disclosing speech models) is crucial for ethical accountability, excessive transparency in interactions may hinder user experience. Striking a balance – where users understand how robots are designed without losing the perception of spontaneous social engagement – is a key challenge.

(D) Should there be some sort of methodological troubleshooting that erases unwanted biases? There seems to be no reason why there should be none. But even if we have some rough methodological frame, which could make sure, for example, that the development team is not too homogeneous, there is always the peril of falling back to complaisant repetitions, ending, as Butler said, as deadening clichés. Developers should always keep in mind that there is no one-for-all criterion for subversiveness. While Stanford’s Gendered Innovations’ virtuous circle as a catalyst for more equality is a good start, gender consciousness is an ongoing political process, and this would also mean an ongoing reinvention of troubleshooting methods and critical assessments to deal with genders in technology.

(E) Is creative gendering in robotics only good marketing and, in that, is it ignoring the broader political and economic issues that are bound up with genders? As criticized by Butler or Halberstam, gender is not only the construction of a broader social and cultural process, but agendas around gender can also be usurped by for-profit purposes where diversity is utilized for soliciting new customers, or for concealing other exploitative structures. Critical gendering in robotics might therefore run the risk of supporting just that. In this respect, this is one of the reasons why I am not sure how to evaluate Pepper since Pepper was, after all, a robot manufactured by SoftBank, which intended to sell as many Peppers as possible. The prize range was between 2,000$ and 30,000$. Robotics is big business. And doing politics within the context of big business is a complicated affair.

Conclusion

This analysis of gender’s social construction and its implications for humanoids argues against seeking gender neutrality. Examining SoftBank’s Pepper as a positive case study, I highlighted the potential for humanoids to challenge gender stereotypes, advocating instead for critical re-gendering in design. This approach is essential to prevent the resurgence of exclusive norms. Furthermore, I discussed the challenges that this entails, from attributing gender to humanoids and navigating binary notions to ensuring transparency and avoiding superficial marketing. Future research should focus not only on designing gender-diverse humanoids but also on ensuring that these representations serve emancipatory rather than commercial interests. Evidently, interdisciplinary collaboration – between roboticists, ethicists, and gender theorists – will be essential in shaping an inclusive technological landscape.

Footnotes

\(^1\) Except for few feminists in technology studies such as Judy Wajcman, Lucy Suchman, Cynthia Cockburn or Susan Omrod, who have questioned technology related gender issues from a somewhat broader perspective, most of the publications discussing concrete gendering in robotics have only emerged since the 2010s. (Robertson 2010, Alesich & Rigby 2017, Kubes 2019)

\(^2\) Some documentation of the speech module can be found here: http://doc.aldebaran.com/2-4/family/pepper_technical/languages_pep.html

\(^3\) 60% of all women working worldwide are employed in services (World Bank, 2020a), the ratio for developed countries is even higher: in 2019 about 83% of all employed women in Japan and 90% in the United States were employed in services (World Bank 2020b).

\(^4\) NAO at Deutsches Museum, Munich, 2025:

\(^5\) Consider https://genderedinnovations.stanford.edu/case-studies/genderingsocialrobots.html

Bibliography

Simone Alesich and Michael Rigby. 2017. Gendered Robots. Implications for Our Humanoid Future. In IEEE Technology and Society Magazine, June 2017, Vol.36(2), pp.50-59.

Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604.

Judith Butler. 2007. Gender Trouble. New York, Routledge.

Judith Butler. 2011. Bodies That Matter. New York, Routledge.

De’Aira Bryant, Jason Borenstein, and Ayanna Howard. 2020. Why Should We Gender? The Effect of Robot Gendering and Occupational Stereotypes on Human Trust and Perceived Competency. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (HRI ‘20). Association for Computing Machinery, New York, NY, USA, 13–21.

R. W. Connell. 2005. Masculinities. Cambridge, Polity Press.

April Glaser. 2016. Pepper, the Emotional Robot, Learns How to Feel Like an American. In Wired, viewed 14 April 2020, https://www.wired.com/2016/06/pepper-emotional-robot-learns-feel-like-american/.

Jack Halberstam. 2018. Trans. A Quick and Quirky Account of Gender Variability*. Oakland, University of California Press.

SoftBank Robotics. 2017. How to Create a Great Experience with Pepper.

Stefan Hirschauer. 2013. Die Praxis der Geschlechter(in)differenz und ihre Infrastruktur. In Julia Graf, Kirstin Ideler, Sabine Klinger (eds.) Geschlecht zwischen Struktur und Subjekt: Theorie, Praxis, Perspektiven. Opladen, Verlag Barbara Budrich, pp. 153-172.

IEEE. 2020. ‘HRP-4C’. IEEE, viewed 15 April 2020, < https://robots.ieee.org/robots/hrp4c/>.

Steve Kovach. 2018). We Interviewed Pepper – The Humanoid Robot. In Tech Insider, viewed 14 April 2020 https://www.youtube.com/watch?v=zJHyaD1psMc.

Tanja Kubes. 2019. Bypassing the Uncanny Valley: Sex Robots and Robot Sex Beyond Mimicry. In Mark Coeckelbergh, Janina Loh (eds.) Feminist Philosophy of Technology. Berlin, J.B. Metzler, pp. 59-73.

Aaron Powers, Adam Kramer, Shirlene Lim, Jean Kuo, Sau-lai Lee and Sara Kiesler (2005). Eliciting Information from People with a Gendered Humanoid Robot. In *IEEE International Workshop on Robots and Human Interactive Communication, pp. 158-163.

Jenifer Robertson. 2010. Gendering humanoid robots: Robo-sexism in Japan. In Body and Society, vol. 16, no. 2, pp. 1-36.

Judy Wajcman. 1991. Feminism Confronts Technology. Cambridge, Polity Press.

Langdon Winner. 2020. Do Artifacts Have Politics? In The Whale and the Reactor. Chicago, University of Chicago Press, pp. 19-39.

World Bank. 2020a. Employment in services, female (% of female employment) (modeled ILO estimate). In World Bank, viewed 15 April 2020 https://data.worldbank.org/indicator/SL.SRV.EMPL.FE.ZS.

World Bank. 2020b. Employment in services, female (% of female employment) (modeled ILO estimate) - Japan, United States. In World Bank, viewed 15 April 2020, https://data.worldbank.org/indicator/SL.SRV.EMPL.FE.ZS?contextual=default&locations=JP-US.

This notebook’s idea is to investigate how well BERT-based hate speech detection is aligned with human judgments of German hate speech. To arrive at plausible explanations, we also need to determine which method, that is, a model agnostic simplification such as LIME or a gradient-based method such as Saliency, works best for our purpose.

As we can see in the following example, the explainability methods will tell us which tokens influenced BERT’s prediction.

Overview

• Loading and preprocessing Gaze4Hate dataset
• Loading and initializing the BERT model for hate speech detection
• Deriving explanations through attributions from BERT with LIME
• Deriving explanations through attributions from BERT with Saliency
• Evaluating the results (human vs. BERT rationales)
• Visualizing the results (alignment)

Before we start, let us begin with installing and loading libraries, which we will use later in the project.

!pip install pandas
!pip install numpy
!pip install seaborn
!pip install scipy

!pip install transformers
!pip install torch torchvision
!pip install -U scikit-learn
!pip install openpyxl
!pip install captum
!pip install sentencepiece
!pip install ipywidgets

!pip show transformers

We’ll need the following packages

# Basic libraries
from collections import defaultdict
from collections import Counter
from IPython.display import HTML, display
import json
from tqdm.notebook import tqdm
import os
import requests
import sys

# Colab
# from google.colab import drive

# Data libraries
import pandas as pd
import numpy as np
import seaborn as sn
import matplotlib.pyplot as plt

# Deeplearning libraries
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from captum.attr import LimeBase, Saliency
from captum._utils.models.linear_model import SkLearnLasso

# Evaluation libraries
from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score, f1_score
from scipy.stats import pearsonr

Loading and preprocessing Gaze4Hate dataset

We use the GAZE4HATE dataset from Alacam et al. (2024), which consists of “hatefulness ratings of text w.r.t. gender, eye movements during plain readings of the statements” (Alacam et al., 2024, p.189) from 43 participants (32 female, 10 male, 1 non-binary, Mean age = 23.5, SD = 5.3 as reported in (Alacam et al., 2024)). The participants’ ratings were gathered on a Likert scale from 1 to 7, where 1 corresponded to ‘very positive’ and 7 to ‘extremely hateful’. 4 and 5 were considered as ‘neutral’ and ‘mean’. The data comes with rationales for the participants’ judgments. These rationales were collected through clicks on words that the participants found relevant. Conversely, if a word is not clicked, it is not considered a determining factor. Finally, the participants’ response were assigned to two hate annotations: the first measured the answers along the binary levels of ‘hate’ and ‘no-hate’ and the second along three levels, namely ‘hate’, ‘neutral’, and ‘positive’. All participants saw all sentences in the data set.

In this project, we are interested in the explicit rationales, which we will later compare with BERT’s attributions with respect to the words that influenced its predictions.

We can find and download the dataset under a CC-BY-NC 4.0 license from: https://osf.io/fgdjw

# URL of the CSV file
url = "https://osf.io/download/fgdjw/"

# Downloading the file
try:
    response = requests.get(url)
except requests.ConnectionError:
    print("No internet connection")

# Saving it as a local CSV file
filename = "Gaze4Hate.csv"
with open(filename, "wb") as file:
    file.write(response.content)

print(f"File downloaded and saved as {filename}")

File downloaded and saved as Gaze4Hate.csv

os.chdir("/path")
df = pd.read_csv("Gaze4Hate.csv", delimiter="\t")

"""
# Mount Google Drive
drive.mount('/content/gdrive')

PATH = r"/content/gdrive/My Drive/Colab Notebooks/EXPL_LIME"
os.chdir(PATH)

directory = os.getcwd()
df = pd.read_csv("Gaze4Hate_data.csv", delimiter="\t")"
"""

Let’s inspect the raw dataset.

df[(df["pno"] == "p1") & (df["sno"] == 19)]

We have many columns with different data, including the sentences, the participants’ hate ratings in terms of gaze data, and click data. Let’s look an example:

Frauen gehören in die Küche.

Which translates to:

Women belong in the kitchen.

The experiment’s first subject clicked on the tokens: “Frauen” and “Küche” (see column Clicked) to justify her judgement of the sentence, which was ‘hate’. Since the dataset has 35 columns they might not all be displayed in your IDE and you won’t see all columns right away.

Let’s select the columns and rows from the dataset that we need for our project. We also want to create a list of the experiment’s unique sentences that we can use to make predictions with our models.

rationale_df = df[["pno", "assertion", "sno", "IA_LABEL", "Clicked", "Intensity_Category_Binary"]]

experiment_sents = {}

for sent, sno in zip(rationale_df["assertion"],rationale_df["sno"]):
    experiment_sents.update({sno: sent})

experiment_sents = dict(sorted(experiment_sents.items()))

print(len(experiment_sents))  # The dataset contains 90 sentences
print(experiment_sents)

90
{1: 'Frauen haben einen signifikant geringeren Durchsetzungswillen.', 2: 'Frauen sind psychisch nicht für komplexe Arbeit geschaffen.', 3: ...}

From the human annotation, we want to have the average rationales for the sentences that were judged as hateful or non-hateful.

participants_n = 43
human_hate_rationale = defaultdict(lambda: defaultdict(int))
human_no_hate_rationale = defaultdict(lambda: defaultdict(int))

for _, row in rationale_df.iterrows():
    sentence_num = row["sno"]
    word = row["IA_LABEL"]
    clicked = row["Clicked"]
    category = row["Intensity_Category_Binary"]

    # Selecting the rationales from those judgments which labeled the given sentence as hate
    if category == 1:
        if clicked == 1:
            human_hate_rationale[sentence_num][word] += 1
        else:
            human_hate_rationale[sentence_num][word] += 0
    else:
        if clicked == 1:
            human_no_hate_rationale[sentence_num][word] += 1
        else:
            human_no_hate_rationale[sentence_num][word] += 0

# We average the responses by the number of participants
average_human_hate_rationales = {
    sentence_num: {word: count / 43 for word, count in words.items()}
    for sentence_num, words in human_hate_rationale.items()
}

average_human_hate_rationales = dict(sorted(average_human_hate_rationales.items()))    
print(average_human_hate_rationales)

average_human_no_hate_rationales = {
    sentence_num: {word: count / 43 for word, count in words.items()}
    for sentence_num, words in human_no_hate_rationale.items()
}

average_human_no_hate_rationales = dict(sorted(average_human_no_hate_rationales.items()))      
print(average_human_no_hate_rationales)

{1: {'Word 1': 0.37209302325581395, 'Word 2': 0.11627906976744186, 'Word 3': 0.023255813953488372, 'Word 4': 0.5116279069767442, 'Word 5': 0.7674418604651163, 'Word 6': 0.6511627906976745}, ...}
{1: {'Word 1': 0.046511627906976744, 'Word 2': 0.0, 'Word 3': 0.0, 'Word 4': 0.09302325581395349, 'Word 5': 0.09302325581395349, 'Word 6': 0.09302325581395349}, ...}

We can see that not all sentences have received rationales, some were univocally annotated as hateful:

print(len(average_human_no_hate_rationales))

82

Futhermore, we want to derive labels for the sentences from the human judgments on average. We select a subframge with only the participants (see column RECORDING_SESSION_LABEL) and judgments (see column Intensity_Category_Binary) for each respective sentence (see column sno).

judgment_df = rationale_df[["pno", "sno", "Intensity_Category_Binary"]]

# We can drop all duplicates
judgment_df = judgment_df.drop_duplicates()

# 90 (sentences) * 43 (participants) = 3870 (judgments); as of 19.03.2025 we report 3616 judgments from the dataset which indicates that some are missing
print(len(judgment_df))

human_labels = defaultdict(list)
avg_human_labels = []

for _, row in judgment_df.iterrows():
    sno = row["sno"]
    label = row["Intensity_Category_Binary"]

    if label == 1:
        human_labels[sno].append(1)
    else:
        human_labels[sno].append(0)

human_labels = dict(sorted(human_labels.items())) 

for _, judgments in human_labels.items():
    count = Counter(judgments)

    if count[1] > count[0]:
        avg_human_labels.append(1)
    elif count[1] < count[0]:
        avg_human_labels.append(0)

print(len(avg_human_labels))
print(avg_human_labels)

3616
90
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Loading and initializing the BERT model

Having prepared our data to make comparisons between the annotators’ rationales and the model’s attributions, we can now intialize our BERT model.

if torch.backends.mps.is_built():
    device = torch.device("mps")  # For mac use
elif torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

BERT_tokenizer = AutoTokenizer.from_pretrained("deepset/bert-base-german-cased-hatespeech-GermEval18Coarse")

BERT = AutoModelForSequenceClassification.from_pretrained("deepset/bert-base-german-cased-hatespeech-GermEval18Coarse", 
    output_hidden_states=True, 
    return_dict=True, 
    output_attentions=True).to(device)

BERT.eval() 

# We can derive the classes used in the model as follows
classes = list(BERT.config.id2label.keys())

print(classes)  # 1 for "hate" and 0 for "no-hate"  

[0, 1]

For our model’s predictions, we need to define a few BERT specificities, such as the encodings that the model will use for its predictions.

encodings = [BERT_tokenizer.encode(sent) for sent in experiment_sents.values()]

We can define a function to make predictions from a given BERT model. We should assure that this function only takes a single argument for the encodings and handles the masking necessary for BERT internally because our Captum LIME class that we will use later passes a single input to its perturbation.

def predict(encoding):
    # Converting masks and encodings to integers is necessary when using LIME
    mask = (encoding > 0).long()
    output = BERT(encoding.long(), mask)

    return output.logits

The predictions from the encodings can be realized through a loop. We’ll have to transform each encoding into a PyTorch tensor. We also need to add another dimension here (see .unsqueeze(0)).

Usually, it is best practice to let the model make several rounds of predictions and average over the results. For simplicity, we will just do a single round.

logits = [predict(torch.tensor(sent).unsqueeze(0).to(device)) for sent in tqdm(encodings, desc="Processing")]
    # Selecting the label with highest probability

BERT_predictions = []
for logit in logits:
    pred_id = torch.argmax(logit, dim=1).item()
    BERT_predictions.append(pred_id)

print(len(BERT_predictions))
print(BERT_predictions)

90
[0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0]

We can now make a comparison of the model’s predictions with the human judgments.

print(classification_report(avg_human_labels, BERT_predictions))

precision    recall  f1-score   support

           0       0.66      0.84      0.74        50
           1       0.69      0.45      0.55        40

    accuracy                           0.67        90
   macro avg       0.67      0.65      0.64        90
weighted avg       0.67      0.67      0.65        90

From here on we would like to understand on what basis, that is, due to which tokens BERT made its predictions. Before we move on, let us have a quick look at how the BERT model would embed a given example sentence.

tensor([[-0.0094, -0.0187,  0.0148,  ..., -0.0247,  0.0179, -0.0125],
        [-0.0353, -0.0224, -0.0362,  ..., -0.0270, -0.0264,  0.0183],
        [-0.0400,  0.0027,  0.0385,  ..., -0.0340,  0.0006,  0.0448],
        ...,
        [-0.0541, -0.0053,  0.0424,  ..., -0.0058,  0.0142, -0.0847],
        [ 0.0010, -0.0685, -0.0209,  ...,  0.0086, -0.0345, -0.0202],
        [-0.0009, -0.0561,  0.0226,  ...,  0.0176,  0.0209,  0.0158]],
       device='mps:0')
torch.Size([13, 768])

Deriving explanations through attributions with LIME

LIME analyzes how changes to the input affect the model’s predictions. It creates a local linear approximation for each instance by making small modifications to the input and observing the resulting changes in output. The weights from this linear model indicate the importance of each token, serving as saliency scores. The original paper (Ribeiro et al. 2016) introducing LIME can be found here: https://dl.acm.org/doi/10.1145/2939672.2939778.

In this part of the project, we will make use of the Captum library to investigate how the BERT model made its predictions. Captum is built on PyTorch and an excellent library for many explainability methods in machine learning. As we want to implement LIME for text, the following approach is built upon the Captum tutorial for LIME. For analyzing texts, we need to use the LimeBase version. The documentation can be found here: https://captum.ai/api/lime.html.

Captum’s LimeBase requires to define several parts of the LIME approach manually. We shall start with this.

# Encode text indices into latent representations & calculate cosine similarity
def exp_embedding_cosine_distance(original_inp, perturbed_inp, _, **kwargs):
    
    original_inp = original_inp.long()
    perturbed_inp = perturbed_inp.long()

    # We want to compare the BERT embeddings from the final hidden states and use their mean
    # LIME will ablate some of the input tokens and the left over is perturbed_inp; we embed this with our BERT model and make a comparison through cosine similarity with the original embedded input

    # The model internally adds position embeddings based on the input length and assumes a default sequence starting from position 0 
    original_emb = BERT.bert.embeddings(original_inp, None).mean(1)
    perturbed_emb = BERT.bert.embeddings(perturbed_inp, None).mean(1)

    distance = 1 - F.cosine_similarity(original_emb, perturbed_emb, dim=1)

    return torch.exp(-1 * (distance ** 2) / 2)

# One hot encoding for masking tokens randomly for ablation
def bernoulli_perturb(text, **kwargs):
    probs = torch.ones_like(text) * 0.5  # Amount of masked tokens
    mask = torch.bernoulli(probs).long()

    # Ensuring at least one token remains (by forcing the first token to stay) because the BERT model expects an input of at least one token
    if mask.sum() == 0:
        mask[torch.randint(0, len(mask), (1,))] = 1  

    return mask

# Removing absent token based on the intepretable representation sample
def interp_to_input(interp_sample, original_input, **kwargs):
    return original_input[interp_sample.bool()].view(original_input.size(0), -1)

# Captum's customizable Lime class
LIME = LimeBase(
    predict,
    interpretable_model=SkLearnLasso(alpha=0.08),
    similarity_func=exp_embedding_cosine_distance,
    perturb_func=bernoulli_perturb,
    perturb_interpretable_space=True,
    from_interp_rep_transform=interp_to_input,
    to_interp_rep_transform=None
)

Everything is in place to loop through the data’s hateful and non-hateful sentences, deriving attributions with our custom LimeBase class.

The parameter show_progress is a widget that might not work in all IDEs right away; I had to adjust my vscode settings.

# Saving all attributions to this list for later comparisons
LIME_attributions = []
i = 0
for encoding in tqdm(encodings, desc="Processing"):
    attrs = LIME.attribute(
        torch.Tensor(encoding).unsqueeze(0).to(device), # Adding batch dimension for Captum
        target=BERT_predictions[i],
        n_samples=200,  # This is a hyperparameter that turned out to work well when set to 200 for our task
        show_progress=False  # vscode is not properly showing progress, must edit vscode's settings.json for widgets (add: "jupyter.widgetScriptSources": ["jsdelivr.com", "unpkg.com"]); if problems persist, set to False
    ).squeeze(0)

    LIME_attributions.append(attrs.tolist())
    i += 1

We can have a look at the attributions:

print(LIME_attributions)

[[-0.0, -0.2673813998699188, -0.16184958815574646, -0.0, 0.0, -0.0, -0.0, -0.0, 0.0, 0.0, -0.0, 0.0, 0.0753924623131752], [-0.2013273388147354, -0.452242910861969, -0.0483330637216568, -0.11272305995225906, -0.0, 0.0, -0.0, 0.0, 0.14384473860263824, -0.0, 0.0, 0.0], ...]

The original data collection from Alacam et al. (2024) contained instances that were later dropped. This led to unique sentence numbers from 1 to 93. To avoid mismatches with the human rationales for given sentences, we should create a list of the actual sentence numbers from the experiment.

unique_sentence_num = list(sorted(experiment_sents.keys()))
print(len(unique_sentence_num))

90

Let’s save the attributions together with their unique ids and sentences so we can later load them for our evaluation with respect to the human rationales.

i = 0
BERT_LIME_attributions = {}

tokens = [BERT_tokenizer.convert_ids_to_tokens(encoding) for encoding in encodings]

for sent, attr in zip(tokens, LIME_attributions):
    BERT_LIME_attributions[unique_sentence_num[i]] = (tokens[i], attr)
    i += 1

with open("BERT_LIME_attributions.json", "w") as file:
    json.dump(BERT_LIME_attributions , file)

Deriving explanations through attributions with Saliency

It is reasonable to use another explainability method because these methods can differ with respect to models and datasets (consider Atanasova et al. (2020)). In what follows, we will implement Saliency for our BERT model and data. The method was introduced in this paper (Simonyan et al. 2013): https://arxiv.org/abs/1312.6034.

Saliency is a gradient based explainability approach. Gradient-based methods generate saliency maps by calculating the gradient of the output with respect to the input. Unfortunately, it requires a little different set-up for our BERT model. We need to change our predict function for BERT; instead of letting it make predictions from the encodings, it needs to base its predictions on the embeddings from the final hidden states of the BERT model.

def predict_new(embeddings, mask):
    return BERT(inputs_embeds=embeddings, attention_mask=mask)[0]

As with the Lime class we assign the forward function predict_new(embeddings, mask) to our Saliency object.

SALIENCY = Saliency(predict_new)

If we use the embeddings for the model’s predictions, we’ll also need to handover masks as they are usually calculated based on the encodings.

def calculate_SALIENCY(input, cls):
    
    masks = torch.tensor([[int(i > 0) for i in input]], device=device)
    input = torch.tensor([input], dtype=torch.long, device=device)
    
    input = BERT.bert.embeddings(input)

    attrs = SALIENCY.attribute(
        input,  # Feeding the embeddings to the predict_new() function
        target=cls, 
        additional_forward_args=masks)[0]  # Passing masks to predict_new()
    
    # L2 normalization for the attributions of Saliency
    attributions = attrs.norm(p=2, dim=-1).squeeze(0)

    return attributions

We can now run through the data as with LIME and derive attributions.

i = 0
BERT_SALIENCY_attributions = {}

for sent, cls in tqdm(zip(encodings, BERT_predictions), total=len(encodings),desc="Processing"):
    attr = calculate_SALIENCY(sent, cls)
    tokens = BERT_tokenizer.convert_ids_to_tokens(sent)
    BERT_SALIENCY_attributions[unique_sentence_num[i]] = (tokens, attr.tolist())
    i += 1

Let’s save the attributions to another file.

with open("BERT_SALIENCY_attributions.json", "w") as file:
    json.dump(BERT_SALIENCY_attributions, file)

Evaluating the results (human vs. BERT rationales)

Finally, we want to make comparisons between the tokens that were relevant for BERT’s hate detection and those that were considered as relevant by the annotators. We are interested in how well the model’s attributions align with the human rationales. We can use both explainability methods to check for the alignment.

with open("BERT_LIME_attributions.json", "r") as file:
    BERT_LIME_attributions = json.load(file)

with open("BERT_SALIENCY_attributions.json", "r") as file:
    BERT_SALIENCY_attributions = json.load(file)

# Converting string keys to integers
BERT_LIME_attributions = {int(k): v for k, v in BERT_LIME_attributions.items()}
BERT_SALIENCY_attributions = {int(k): v for k, v in BERT_SALIENCY_attributions.items()}

We need to convert the subword tokens into full words. This requires a customized function for the dataset. It’s not perfect, but shall suffice for most instances.

def subword_to_human_level(subwords, subword_rationales):

    human_level_rationales = [] 
    word_rationale = []

    for i, (subword, rat) in enumerate(zip(subwords, subword_rationales)):

        if subword in ["[CLS]", "[SEP]", ".", ","]:
            continue
        else:
            word_rationale.append(rat)
            if not any([subwords[i+1].startswith('##'),
                        subword in ["#", "-"],
                        subwords[i+1] == "-", 
                        subwords[i-1] == "#"]):
                # Mean from subwords will be added to human level rationales
                human_level_rationales.append(np.mean(word_rationale))
                word_rationale = []
            
    return human_level_rationales

We can evaluate the alignment in terms of the Pearson correlation coefficient.

def compute_mean_correlation(truth, preds):
       
    corrs = []
    
    for t, p in zip(truth, preds):
        # Zero variance check because some attributions might be only zeros
        if np.std(t) == 0 or np.std(p) == 0:
            corrs.append(0)  # Default to 0 correlation
        else:
            corrs.append(pearsonr(t, p)[0])

    mean_correlation = np.mean(corrs)

    return mean_correlation  

To calculate the correlation, we focus on instances where the model’s prediction matches the majority human annotation. Specifically, we analyze local explanations only for sentences where the model made a correct prediction, allowing us to assess how closely its rationales correspond to the average human rationales.

human_rationales = {}
correct_prediction_labels = {}
BERT_rationales_LIME = {}
BERT_rationales_SALIENCY = {}
i = 0

for gold, pred in zip(avg_human_labels, BERT_predictions):

    sent_id = unique_sentence_num[i]

    if gold == pred:
        if gold == 1:
            r = list(average_human_hate_rationales[sent_id].values())
        else:
            r = list(average_human_no_hate_rationales[sent_id].values())

        LIME_attrs_word_level = subword_to_human_level(BERT_LIME_attributions[sent_id][0], BERT_LIME_attributions[sent_id][1])

        # LIME comes with positive and negative attributions; we're only interested in the positive ones because these are correlated with the model's targets
        p_LIME = [x if x >= 0 else 0. for x in LIME_attrs_word_level]

        p_SALIENCY = subword_to_human_level(BERT_SALIENCY_attributions[sent_id][0], BERT_SALIENCY_attributions[sent_id][1])

        human_rationales[sent_id] = r
        correct_prediction_labels[sent_id] = gold
        BERT_rationales_LIME[sent_id] = p_LIME
        BERT_rationales_SALIENCY[sent_id] = p_SALIENCY
    i += 1

Finally, we can compute the Pearson correlation coefficient for LIME attributions…

compute_mean_correlation(human_rationales.values(), BERT_rationales_LIME.values())

np.float64(0.14380772549112125)

… and for Saliency attributions.

compute_mean_correlation(human_rationales.values(), BERT_rationales_SALIENCY.values())

np.float64(0.6632999675122192)

LIME’s sampling includes an element of randomness; therefore, it would be reasonable to repeat the LIME attribution several times to arrive at average results.

With five repititions, I get a positive correlation of 0.17 using LIME, which suggests a weak correlation between the BERT and human rationales. For Saliency, I have a much stronger colleration of 0.66. Now we can see how much influence also the chosen explainability method has. For LIME and Saliency, the BERT model has been the same. However, the BERT model appears much more aligned when seen from the perspective of the Saliency attributions.

Visualizing the results (alignment)

Finally, we can visualize our results to see how the explainability methods make attributions with respect to the BERT model and its predictions.

def plot_triple_heatmaps(data1, data2, data3, xticklabels, title1, title2, title3, save_path):
    _, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(15, 4))  # Creating three vertical subplots
    
    # Plotting the first heatmap (top) - Human rationales
    sn.heatmap(np.array([data1]), 
               annot=True, 
               cmap="coolwarm",
               xticklabels=xticklabels,
               yticklabels=["Human rationales"], 
               cbar=False, 
               ax=ax1)
    
    ax1.set_title(title1)
    
    # Plotting the second heatmap (middle) - Saliency attributions
    sn.heatmap(np.array([data2]),
               annot=True, 
               cmap="coolwarm", 
               xticklabels=xticklabels, 
               yticklabels=["Saliency"], 
               cbar=False, 
               ax=ax2)
    
    ax2.set_title(title2)

    # Plotting the third heatmap (bottom) - LIME attributions
    sn.heatmap(np.array([data3]), 
               annot=True, 
               cmap="coolwarm", 
               xticklabels=xticklabels, 
               yticklabels=["LIME"], 
               cbar=False, 
               ax=ax3)
    
    ax3.set_title(title3)
    
    plt.tight_layout()  # Adjusting layout to prevent overlap
    
    # Saving the figure
    if save_path:
        plt.savefig(save_path, bbox_inches="tight", dpi=300)
        plt.show() 
    else:
        plt.show()

We can define a directory path here where we will save our visualizations.

dir_path_visual = "visualizations"

We loop through the rationales and attributions to create heatmaps for the words from the experiment’s sentences.

def process_and_plot_heatmaps(human_rationales, correct_prediction_labels,BERT_rationales_SALIENCY, BERT_rationales_LIME, experiment_sents, output_dir):
    
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)  # Creating output directory if it doesn't exist
    
    for sno in human_rationales.keys():

        hate_data = human_rationales[sno]
        saliency_data = BERT_rationales_SALIENCY[sno]
        lime_data = BERT_rationales_LIME[sno]

        print(f"Plotting heatmap for rationales, saliency, and LIME for given sno: {sno}")
                        
        xticklabels = experiment_sents[sno].strip('.,!?').split()
                            
        save_path = os.path.join(output_dir, f"hate_heatmap_key_{sno}.png")

        label = "Hate" if correct_prediction_labels[sno] == 1 else "No Hate"
                            
        plot_triple_heatmaps(hate_data, 
                            saliency_data, 
                            lime_data, 
                            xticklabels, 
                            title1=f"Sentence No. {sno} Label: {label}", 
                            title2=f"",
                            title3=f"",
                            save_path=save_path)

process_and_plot_heatmaps(human_rationales, correct_prediction_labels, BERT_rationales_SALIENCY, BERT_rationales_LIME, experiment_sents, dir_path_visual)

References

Özge Alacam, Sanne Hoeken, and Sina Zarrieß. 2024. Eyes don‘t lie: Subjective hate annotation and detection with gaze. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 187–205, Miami, Florida, USA. Association for Computational Linguistics.

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3256–3274, Online. Association for Computational Linguistics.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “why should i trust you?”: Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA. Association for Computing Machinery.

Sarthak Roy, Ashish Harshvardhan, Animesh Mukherjee, and Punyajoy Saha. 2023. Probing LLMs for hate speech detection: strengths and vulnerabilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6116–6128, Singapore. Association for Computational Linguistics.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034.

The Jupyter notebook for this project can be found here: https://github.com/omseeth/explaining_BERT_with_LIME_Saliency

What follows is partially the result of a lecture given by Barbara Plank at LMU in 2024, whom I’d like to thank along with her lab MaiNLP for letting me use some of their code. The theoretical part discussed in this tutorial is also highly indebted to Raschka et al. (2022), whose book on machine learning is for me one of the best resources.

This tutorial is divided into the following chapters:

• Preliminaries imports
• Section 1) Preprocessing the dataset
• Section 2) Theoretical foundations of CNNs for classification
• Section 3) Implementing a CNN with PyTorch
• Section 4) Evaluation of results

Preliminaries: imports

In this tutorial, we’ll be using several different libraries. The following imports shall become clear as we develop our project. I’m using Python 3.12.0.

"""In case the packages are not installed on your machine, let's begin with 
installing the ones that we'll need for our task. We can run commands with an 
exclamation mark in Jupyter Notebooks."""

!pip install datasets transformers tokenizers scikit-learn torch tqdm

from datasets import load_dataset
import random
from sklearn.metrics import f1_score
from tokenizers import Tokenizer, models, trainers
from tokenizers.pre_tokenizers import Whitespace
import torch.nn as nn
import torch
from tqdm import tqdm

Section 1) Preprocessing the dataset

Loading the data

Our goal is to use data to train a model that can identify the sentiment of a given text instance. In other words, we’ll implement a classifier using supervised learning. The backbone of our sentiment classifier will be a CNN. The data we’re using is taken from Saravia et al. (2018). It consists of English Twitter messages labeled with six “emotions”: anger, fear, joy, love, sadness, and surprise. The dataset is available on HuggingFace and we can load it with the datasets package.

emotions = load_dataset("dair-ai/emotion")

We can have a quick look at how the dataset is splitted and structured.

emotions

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 16000
    })
    validation: Dataset({
        features: ['text', 'label'],
        num_rows: 2000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 2000
    })
})

The labels are already numericalized and the text is lowercased, as we can see when inspecting a single instance.

# The label IDs correspond to the following label order:
labels = ["sadness", "joy", "love", "anger", "fear", "surprise"]

emotions["train"][:5]

{'text': ['i didnt feel humiliated',
  'i can go from feeling so hopeless to so damned hopeful just from being around someone who cares and is awake',
  'im grabbing a minute to post i feel greedy wrong',
  'i am ever feeling nostalgic about the fireplace i will know that it is still on the property',
  'i am feeling grouchy'],
 'label': [0, 0, 3, 2, 3]}

To allow easier processing, it helps to bind the respective splits to different variables.

train_data = emotions["train"]
validation_data = emotions["validation"]
test_data = emotions["test"]

Tokenization

Before we can use the data to train our model, we need to split up the sentences into tokens. We also want to batch our data so that the model can be trained with more instances at the same time. In this tutorial, we’ll do the tokenization and batching with custom functions.

For our tokenization, we can use a subword tokenization algorithm that is called Byte Pair Encoding (BPE), which was introduced in Sennrich et al. (2016) and is based on an algorithm introduced by Gage (1994). The motivation of this tokenization technique is to let the data decide what subword tokens we’ll have. Another advantage of BPE is that it allows us to deal with unknown words that won’t be part of the training vocabulary but might appear in test data. With BPE, unknown words can be decomposed into their respective subword parts and in this fashion still be processed.

Instead of implementing the BPE tokenizer completely from scratch, we can use the tokenizers package. We also want to pad and truncate each text with a fixed sequence length. If text instances are short, we pad them with 0s; if they are long, we truncate them to the maximum length. This will make the training and inference processes easier.

# We want to begin with only 5000 words in our vocabulary and a fixed sequence 
# length of 64
vocab_n = 5000
sequence_len = 64

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# We can have a preliminary splitting of the text on spaces and punctuations 
# with Whitespace()
tokenizer.pre_tokenizer = Whitespace()

# Padding instances to the right with 0s
tokenizer.enable_padding(length=sequence_len)

# Truncating long sequences
tokenizer.enable_truncation(max_length=sequence_len)

# We limit our vocabulary and train the tokenizer
tokenizer_trainer = trainers.BpeTrainer(vocab_size=vocab_n)
tokenizer.train_from_iterator(train_data["text"], trainer=tokenizer_trainer)

After training our custom BPE tokenizer, we aim to transform the raw training data into vector representations, where the input is tokenized and converted into corresponding token IDs. Fortunately, the labels are already numerical. That said, we still need to convert these numerical representations into PyTorch tensors to enable the PyTorch model to process the training instances. To accomplish this, we can write a few custom functions.

def preprocess_text(text: str, tokenizer: Tokenizer):
    """ 
    Helper function to tokenize text and return corresponding token IDs as tensors.

    Args:
        text, str: Text instance from training data.
        tokenizer, Tokenizer: The respective tokenizer to be used for tokenization.
    Returns:
        Tensor: One-dimensional PyTorch tensor with token IDs.
    """
    return torch.tensor(tokenizer.encode(text).ids)


def preprocess_label(label: int):
    """ 
    Helper function to return label as tensor.

    Args:
        label, int: Label from instance.
    Returns:
        Tensor: One-dimensional PyTorch tensor containing the label index.
    """
    return torch.tensor(label)


def preprocess(data: dict, tokenizer: Tokenizer):
    """ 
    Transforms input dataset to tokenized vector representations.

    Args:
        data, dict: Dictionary with text instances and labels.
        tokenizer, Tokenizer: The respective tokenizer to be used for tokenization.
    Returns:
        list: List with tensors for the input texts and labels.
    """
    instances = []

    for text, label in zip(data["text"], data["label"]):
        input = preprocess_text(text, tokenizer)
        label = preprocess_label(label)
        
        instances.append((input, label))

    return instances

Let’s tokenize, pad, and truncate our datasets.

train_instances = preprocess(train_data, tokenizer)
val_instances = preprocess(validation_data, tokenizer)
test_instances = preprocess(test_data, tokenizer)

We shall inspect the second training instance, which was:

i can go from feeling so hopeless to so damned hopeful just from being around someone who cares and is awake.

train_instances[1]

    (tensor([   8,  161,  103,  215,   55,   58, 1173,   36,   58,  807,  587, 1129,
            130,  215,  219,  382,  444,  197, 2819,   42,   47, 2670,    0,    0,
                0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
                0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
                0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
                0,    0,    0,    0]),
    tensor(0))

We can observe that “i” has token index 8. Also notice how “so” appears in the vector representation of the text with index 58 twice. The corresponding label for the sequence is 0 and would be “sadness”.

Batching

Batching the instances will enable the model to process multiple instances simultaneously. To achieve this, let’s write a custom batching function. There are different ways to perform batching, which involves concatenating instances and dividing them into manageable groups. One convenient method is to use the torch.stack() function, which concatenates a sequence of tensors. However, all tensors must be of the same length. Since we already padded and truncated our text instances, this requirement is satisfied.

def batching(instances: list, batch_size: int, shuffle: bool):
    """ 
    Batches input instances along the given size and returns list of batches.

    Args:
        instances, list: List of instances, containing a tuple of two tensors 
            for each text as well as corresponding label.
        batch_size, int: Size for batches.
        shuffle, bool: If true, the instances will be shuffled before batching.
    Returns:
        list: List containing tuples that correspond to single batches.
    """
    if shuffle:
        random.shuffle(instances)

    batches = []

    # We iterate through the instances with batch_size steps
    for i in range(0, len(instances), batch_size):

        # Stacking the instances with dim=0 (default value)
        batch_texts = torch.stack(
            [instance[0] for instance in instances[i : i + batch_size]]
        )
        batch_labels = torch.stack(
            [instance[1] for instance in instances[i : i + batch_size]]
        )

        batches.append((batch_texts, batch_labels))
    
    return batches

Section 2) Theoretical foundations of CNNs for classification

After having prepared our data, we can start with implementing our model. In this tutorial, we’ll be using a Convolutional Neural Network (CNN), which was introduced in LeCun et al. (1989). The CNN bears resemblance to the Neocognitron (Fukushima 1980) as well as Time-Delay Neural Networks (Waibel et al. 1989). This section is dedicated to the theoretical background of CNNs. In Section 3, we’ll implement the model with PyTorch.

The architecture of a CNN

A CNN is a fully connected network that operates similar to Feedforward Neural Networks (FNNs) in a feedforward fashion. The input is processed along layers and forwarded to the final layer, which can be used for predictions. The idea of a CNN is to consecutively extract salient features, such as shapes in terms of their respective pixels, building an abstract feature hierarchy for any given input. In this fashion, the model will be aligned through its training to reoccurring patterns, which it can “recognize.” It’s no surprise that CNNs were invented for handwritten digit recognition in images.

The basic architecture of a CNN is as follows: For any given input, the CNN uses convolutions, that is, a filter (also called kernel) technique to extract local features from the input (e.g., pixels). After this filter has shifted over the whole input, the extractions are combined together into feature maps and then transformed with pooling. Pooling helps with singling out dominant features and allows to reduce the dimensions of the feature maps. This process can be repeated multiple times where additional filters can shift again over the pooled output from previous convolutions. Finally, for the last part of the network a FNN is used to produce logits for the desired task.

One CNN layer consists of one convolutional and one pooling layer.

Filtering

Filters shift over the input. If the input is one-dimensional, that is, a vector, then the filter is itself a one-dimensional, usually smaller vector. If the input is two-dimensional, a matrix, then the filter needs to be a matrix, too.

For a one-dimensional input of length \(n=8\), such as \([2, 1, 0, 3, 6, 7, 9, 1]\), we can define a filter of size \(m=3\), such as \([1, -1, 0]\). As the shifting starts, we’ll multiply the first three entries of the input vector with the filter:\([2, 1, 0] * [1, -1, 0]^T = 1\) In this fashion, we obtain the first entry for our feature map: \(y[0]=1\). Next, we shift the filter one step forward (with stride=1) and repeat the multiplication, until the filter reaches the end of the input, bounded by its last index (see Fig. 1). We can formalize this operation between the filter vector f and the input vector e as follows:

In fact, the filter size is a hyperparameter that can be set to any positive integer smaller than the input length. In those scenarios where the filter is very large and exceeds the input, we can pad the input with additional elements so that the filter operation would be possible. There a several padding strategies to avoid that elements from the middle of the input will be covered more frequently by the filter than those from the edges.

In our implementation (Section 3), we’ll be using the same padding strategy, which adds zeros to the input on all sides so that the output of the filter operation can have the same size as the input. Therefore, the amount of padding will depend on the size of the filter(/kernel). Consider Raschka et al. (2022, p. 457) for further details.

Fig. 1: One-dimensional convolution

The convolution described previously for one-dimensional inputs works in exactly the same way for two-dimensional inputs (see Fig. 2). If \(E_{n_1\times n_2}\) is the input matrix and \(F_{m_1\times m_2}\) the filter where \(m_1\leq n_1\) and \(m_2\leq n_2\), for each stride we can compute our feature map as follows:

with \(y[i][j]\) corresponding to the respective feature representation in the map, which is also a matrix.

Fig. 2: Two-dimensional convolution

Subsampling layers: pooling

The idea of pooling layers is to reduce the size of the feature map and abstract from the features, either by picking out one local maximum feature value or averaging over a group of feature values. The former strategy is called max pooling, the latter is called average pooling. We’ll focus on max pooling (see Fig. 3).

A pooling layer can be defined as \(P_{n_1 \times n_2}\), where the subscript indicates the area size over which the max operation is performed, while shifting across the entire map.

Fig. 3: Max pooling

Adding activation functions

A FNN usually consists of a layer that is defined as \(z=Wx + b\) where \(W\) are the weights and \(b\) is an extra bias for input \(x\). Such a layer is often wrapped by a non-linear activation function \(\sigma\), as in \(A=\sigma (z)\). We adapt the convolutional layer so that it resembles the previous FNN construction. Let \(c = E*F + b\) be our layer with an additional bias term \(b\). As with the FFN we can wrap an activation function \(\sigma\) around \(c\). In many CNN implementations, the ReLu activation function is used for this purpose.

LeCun et al. (1989) published their paper titled ‘Handwritten Digit Recognition with a Back-Propagation Network.’ Let us now examine which parts of a CNN are trainable using back-propagation. It’s the convolutional layers (filters) with their weights \(F\) and biases \(b\) (if included) that can be updated through back-propagation, guided by a loss function and gradient optimization. Recall that most CNNs also include a final fully connected feedforward neural network (FNN) layer for producing the output, and this layer is trainable as well. However, the pooling layer is not involved in the training process, as it does not contain learnable parameters.

Multiple channels

The input to a CNN can be greater than one. If we’re dealing with images that are encoded in terms of RGB colors, we can split up the input into three two-dimensional matrices corresponding to red, green, and blue color information and feed them to the CNN. Each input matrix would be called one channel. For this matter, in most CNN implementations the convolutional layers expect an input with 3 dimensions where \(E_{n_1\times n_2 \times c_{in}}\) would be the input of dimensions \(n_1\times n_2\) times \(c_{in}\), such as \(c_{in} = 3\) for three different color matrices.

The further processing of the input allows a lot of variability: Each \(c_{in}\) input will have its own filter. The filters can be also stored as three-dimensional tensors: \(F_{m_1\times m_2 \times c_{in}}\). Usually, the filtered results from each respective input will be element-wise added to create the output feature map. But sometimes it also helps to have multiple feature maps as outputs to capture different aspects from the input. In this case, the filters can be changed to four-dimensional tensors: \(F_{m_1\times m_2 \times c_{in}\times c_{out}}\) where \(c_{out}\) determines the numbers of feature maps that we want (see Fig. 4).

Fig. 4: Multiple filters (kernels) for three input channels, producing four feature maps

Note, if we have for example four feature maps, we’ll also need four pooling layers. As the size of convolutional layers changes we’ll also have more parameters to train. This being said, remember that a FNN processing an input image would require weights of same size for its first layer. In contrast, the parameters of CNNs are bounded by the filter size. LeCun et al. (1989, p. 399) describe this aspect as “weight sharing” and note as one distinctive feature of CNNs that they significantly reduce the amount of parameters to be trained.

Section 3) Implementing a CNN with PyTorch

Let’s begin with defining our classifier model. It’s reasonable to use embeddings for our input tokens. For that matter, our model should start with an embedding layer. The embeddings will be handed over to our CNN, which will consist of two convolutional layers with two different filters(/kernels). Finally, the input abstracted throughout the network needs to be passed to a final FFN layer for our sentiment prediction.

To build an intuition for the whole process, let’s think again of the input. The input is a sequence (a tweet) from our data. This sequence will be translated into its correspoding token representation based on our custom trained BPE encoder. In other words, any incoming sequence will be a tensor of different indices. Next, the indices are handed over to our embedding layer, which serves as a look-up table for all tokens in the vocabulary and assigns the embedding representation to the input indices. This layer is a simple linear layer that is going to be trained along with the whole network later on. The initial values of the embeddings are random numbers.

Up to this point, we can think of two strategies to run a first convolution over the input. If the input is 12 tokens long and each token has an embedding vector of size 300, the input to a potential convolutional layer would be \(12 \times 300\). The first strategy could be to apply a two-dimensional filter striding over this input matrix. The second strategy would be to use a one-dimensional filter that is going over one embedding dimension per time for the whole sequence, that is, over \(12 \times 1\) but with 300 channels for all embedding dimensions. If we want the output to be of the same length as the input, that is, 12, we need to pad the input to adjust for the filter length.

In our implementation, we shall try the second strategy. Remember that the input channels will all be added together so before having our 300 embedding dimensions shrunk to one, we shall define an appropriate output channel size for the convolution. In other words, we’ll use as many as 100 filters for example to create an output of 100 dimensions for each respective input token.

After having defined our model, we need to implement the training loop with a fitting method. To understand our training process we also need an evaluation mechanism. Finally, we need another method for making predictions from the trained classifier.

class CNN_Classifier(nn.Module):
    """ 
    CNN for sentiment classification with 6 classes, consisting of an embedding 
    layer, two convolutional layers with different filter sizes, different 
    pooling sizes, as well as one linear output layer.
    """
    def __init__(self):
        super().__init__()
        # We can implement embeddings as a simple lookup-table for given word 
        # indices
        self.embedding = nn.Embedding(tokenizer.get_vocab_size(), 300)

        # One-dimensional convolution-layer with 300 input channels, and 100  
        # output channels as well as kernel size of 3; note that the
        # one-dimensional convolutional layer has 3 dimensions
        self.conv_1 = nn.Conv1d(300, 100, 3, padding="same")

        # Pooling with with a one-dimensional sliding window of length 3, 
        # reducing in this fashion the sequence length 
        self.pool_1 = nn.MaxPool1d(3)

        # The input will be the reduced number of maximum picks from the
        # previous operation; the dimension of those picks is the same as the
        # output channel size from self.conv_1. We apply a different filter of 
        # size 5.
        self.conv_2 = nn.Conv1d(100, 50, 5, padding="same")

        # Pooling with window size of 5
        self.pool_2 = nn.MaxPool1d(5)

        # Final fully connected linear layer from the 50 output channels to the
        # 6 sentiment categories 
        self.linear_layer = nn.Linear(50, 6)

    def forward(self, x):
        """ 
        Defining the forward pass of an input batch x.

        Args:
            x, tensor: The input is a batch of tweets from the data.
        Returns:
            y, float: The output are the logits from the final layer.
        """
        # x will correspond here to a batch; therefore, the input dimensions of 
        # the embedding will be by PyTorch convention as follows:
        # [batch_size, seq_len, emb_dim]
        x = self.embedding(x)

        # Unfortunately the embedding tensor does not correspond to the shape 
        # that is needed for nn.Conv1d(); for this reason, we must switch its 
        # order to [batch_size, emb_dim, seq_len] for PyTorch
        x = x.permute(0, 2, 1)

        # We can wrap the ReLu activation function around our convolution layer
        # The output tensor will have the following shape: 
        # [batch_size, 100, seq_len]
        x = nn.functional.relu(self.conv_1(x))

        # Applying max pooling of size 3 means that the output length of the 
        # sequence is shrunk to seq_len//3
        x = self.pool_1(x)

        # Output of the following layer: [batch_size, 50, seq_len//3]
        x = nn.functional.relu(self.conv_2(x))

        # Shrinking the sequence length by 5
        x = self.pool_2(x)
        # print(x.shape)

        # At this point we have a tensor with 3 dimensions; however, the final layer 
        # requires an input of size [batch_size x 50]. To get this value we can 
        # aggregate the values and continue only with their mean
        x = x.mean(dim=-1)

        # In this fasion, the linear layer can be used to make predictions
        y = self.linear_layer(x)

        return y
    
    def fit(self, train_instances, val_instances, epochs, batch_size):
        """ 
        Gradient based fitting method with Adam optimization and automatic 
        evaluation (F1 score) for each epoch.

        Args:
            train_instances, list: List of instance tuples.
            val_instances, list: List of instance tuples.
            epochs, int: Number of training epochs.
            batch_size, int: Number of batch size.
        """
        self.train()
        optimizer = torch.optim.Adam(self.parameters())

        for epoch in range(epochs):
            train_batches = batching(
                train_instances,
                batch_size=batch_size,
                shuffle=True)
            
            for inputs, labels in tqdm(train_batches):
                optimizer.zero_grad()
                outputs = self(inputs)
                loss = nn.functional.cross_entropy(outputs, labels)
                loss.backward()
                optimizer.step()
            
            train_f1 = self.evaluate(train_instances, batch_size=batch_size)
            val_f1 = self.evaluate(val_instances, batch_size=batch_size)

            print(f"Epoch {epoch + 1} train F1 score: {train_f1}, validation F1 score: {val_f1}")

    def predict(self, input):
        """ 
        To make inferences from the model.

        Args:
            input, tensor: Single instance.
        Returns:
            int: Integer for most probable class.
        """
        self.eval()
        outputs = self(input)

        return torch.argmax(outputs, dim=-1)

    def evaluate(self, instances, batch_size):
        """ 
        To make evaluations against the gold standard (true labels) from the 
        data.

        Args:
            instances, list: List of instance tuples.
            batch_size, int: Batch size.
        Returns:
            float: Macro F1 score for given instances.
        """
        batches = batching(instances, batch_size=batch_size, shuffle=False)
        true = []
        pred = []

        for inputs, labels in batches:
            true.extend(labels)
            pred.extend(self.predict(inputs))

        return f1_score(true, pred, average="macro")

We can now train our model.

classifier = CNN_Classifier()
classifier.fit(train_instances, val_instances, epochs=5, batch_size=16)

100%|██████████| 1000/1000 [00:26<00:00, 37.29it/s]
Epoch 1 train F1 score: 0.6349625340555586, validation F1 score: 0.5885997748733948
100%|██████████| 1000/1000 [00:27<00:00, 36.16it/s]
Epoch 2 train F1 score: 0.9181883927660527, validation F1 score: 0.8391332797173209
100%|██████████| 1000/1000 [00:31<00:00, 31.86it/s]
Epoch 3 train F1 score: 0.9548322090243025, validation F1 score: 0.8665508014136801
100%|██████████| 1000/1000 [00:30<00:00, 32.63it/s]
Epoch 4 train F1 score: 0.9715161826479329, validation F1 score: 0.851259672136001
100%|██████████| 1000/1000 [00:33<00:00, 30.13it/s]
Epoch 5 train F1 score: 0.9830196392925995, validation F1 score: 0.8649424900902641

Section 4) Evaluation of results

After we have trained our model, we can use our test set to see how well it predicts sentiments for unseen examples.

f1_test = classifier.evaluate(test_instances, batch_size=16)
print(f"F1 score for test set: {f1_test}")

F1 score for test set: 0.8403978699590415

Hyperparameters

There are many hyperparameters in the entire modeling process that we can adjust to achieve even better results. These include different layer sizes, filters, poolings, and more efficient optimization techniques. Typically, dropout is applied during the training of such models to prevent overfitting. It is also worthwhile to explore the possibility of using pre-trained embedding representations.

The Jupyter notebook for this project can be found here: https://github.com/omseeth/cnn_sentiment_analysis

References

Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybernetics 36, pages 193–202.

Gage, P. (1994). A new algorithm for data compression. C Users J., 12(2):23–38.

LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L. (1989). Handwritten digit recognition with a back-propagation network. In Touretzky, D., editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann

Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., and Chen, Y.-S. (2018). CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium. Association for Computational Linguistics.

Sebastian, R., Yuxi, L., and Mirjalili, V. (2022). Machine Learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python. Packt Publishing.

Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A., editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics

Waibel, A., Hanazawa, T., Hinton, G., Shikano, K. and Lang, K. J. (1989). Phoneme recognition using time-delay neural networks. In IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pages 328-339.

In this post, we will touch on three aspects:

• Deep dive: traditions in ethics
• Inspiration: philosophy of technology
• Application: challenges when using AI ethically

Deep dive: traditions in ethics

Ethics and morality: Ethics is a sub-discipline of philosophy, theology, psychology, sociology, education, political science, or anthropology. It is often helpful to divide ethics into three areas: 1) descriptive ethics dealing with the empirical question which moral rules and norms exist in each community, 2) normative ethics which reflects on reasons for moral systems, and 3) metaethics questioning the language we use in moral discourse. For example, from a descriptive point of view, we could investigate what psychological factors affect our moral decision making. The predominant ethical traditions, however, adhere to a normative approach.

The three traditions of classical ethics: Throughout the history of ethics, three traditions have influenced the field. Today’s ethicist can often still be traced to one of these traditions. The first tradition which many have heard before is often referred to as utilitarianism. A utilitarian would try to maximize the utility for the greatest number of people. It is sometimes also described as the greatest happiness principle. The second tradition in ethics is closely tied to the German philosopher, Immanuel Kant, who introduced the categorial imperative as man’s and woman’s best guide for moral behavior. The imperative says: “Act only according to that maxim whereby you can at the same time will that it should become a universal law.” (Kant 1993, p. 30) In other words, Kant provides us with a tool with which we can check whether any action is moral. The third tradition is the oldest. It is linked to the Ancient Greek philosopher Aristotle. In contrast to the other two traditions, Aristotle’s ethics consists in multiple virtues (e.g. prudence and temperance) that he and many of his followers believe to be leading to the good life.

Three different perspectives: The three dominant traditions in ethics can also be divided by what aspect they focus on. For example, utilitarianism considers only the results of actions. What counts is the aftermath! A Kantian ethicist, in contrast, sets her eyes on each particular action. She wonders whether it is ethically licensed by the categorical imperative. A virtue ethicist is rather interested in what virtues lead us to a particular action.

Inspiration: philosophy of technology

I, technology, and the world: The philosopher Don Ihde (1990) introduces a simple, but powerful, abstraction that can help us reflect on technologies. Consider yourself looking through a microscope, seeing the wonders of the world from a different angle. As you observe bacteria on a little petri dish, you forget everything around yourself. In fact, you have morphed with the microscope. It has become part of your vision and even of yourself. Ihde (1990) describes this relation with the technology with this short formula:

(I – Technology) – World

Imagine you are in a warm room and the rain is pouring down on your window. As you are looking at the thermostat on your wall, you are seeing that it is almost freezing outside. When you become aware of the temperature, you are not feeling the cold. It is transmitted to you by the thermostat. Ihde (1990) describes this constellation as follows:

I – (Technology – World)

Differently put, the temperature of the outside world in such situations is only indirectly accessible to you through the measuring device on your wall.

Application: challenges when using AI ethically

Technological relations and moral decisions: When making decisions involving AI, it is helpful to understand how the technology is being used. For instance, are we merging with a driving assistant that is powered with AI? Or do we confront reality, as it is mediated to us by AI powered algorithms; for instance, when we use social media platforms that present a particular representation of what a pleasant vacation ought to entail? Realizing our position with which we relate to AI can then help us with understanding how AI would mediate and even change our actions. For instance, AI could help us with achieving the best outcome for the most people by ameliorating our capacity to react quickly in critical situations. But it could also deceive us when we would ask a chatbot whether something would be acceptable as a universal moral law. For an ethically desirable way of developing and using AI, we need to be aware of how it affects our actions.

The Other: Finally, Ihde also considers situations where we find ourselves confronted with technology that we perceive as another being. He describes this sort of relation as follows (Ihde 1990):

I – Technology – World

And as AI advances, we might be more inclined to also consider AI as such a being that should deserve more attention. But whether we will ever see this one day is yet to be seen!

In this post, you have learned a basic understanding of ethics. You saw an abstraction from the philosophy of technology to help position yourself when using AI. With this knowledge, I hope you feel motivated to use AI ethically.

Bibliography

Ihde, D. (1990). Technology and the Lifeworld – From Garden to Earth. Indiana University Press.

Kant, I. (1993). Groundwork of the Metaphysics of Morals. Translated by Ellington, James W. (3rd ed.). Hackett.

1 Introduction

The advent of large language models (LLMs) through the Transformer architecture (Vaswani et al., 2017) has led to models with unprecedented capabilities. This contribution attempts to harness these capabilities to reproduce an empirical experiment from Salzmann et al. (2022) with one Transformer based model, GPT 3.5 Turbo developed by OpenAI.

The background to the experiment is an investigation of Principle C (also called Condition C). According to syntactic theory, Principle C governs the possibilities of coreference between pronouns and R-expressions (i.e., proper names and definite determiner phrases). However, how the principle works remains a matter of debate (Adger et al., 2017; Bruening and Khalaf, 2018; Salzmann et al., 2022; Stockwell et al., 2021). For that matter, Salzmann et al. (2022) developed several experiments to explore the principle in German contexts. This paper focuses on their second experiment, Experiment 2, where the authors attempted with a 2 x 2 x 2 experimental design to investigate whether participants reconstructed the syntactic hierarchy of given sample sentences when asked if different interpretations were possible (Salzmann et al., 2022, 601).

To reproduce Experiment 2, GPT 3.5 Turbo was queried with the exact same items that were used in the original work by Salzmann et al. (2022). I employed diverse prompt types, including zero-shot and one-shot, and varied sizes of iterations. In order to match the identical quantity of data points collected from 32 participants in Salzmann et al. (2022), I finally chose the outcomes of a configuration of four iterations with a zero-shot prompt to query GPT 3.5 Turbo for fitting a generalized linear mixed model. With this model, I tested the same three hypotheses from Experiment 2 in Salzmann et al. (2022).

I report the following results: In line with human participants, the LLM qualified sentences with referring expressions in such a fashion that the influence of Principle C for respective cases with R-expressions and pronouns is attested. In contrast to the human proportion of answers, GPT 3.5 Turbo’s output did not indicate a significant influence of movement when interpreting the test item sentences. If movement is understood as a variable for reconstruction, it appears that the LLM does not reconstruct the pre-moved positions of the phrases. Finally, a significant influence of position of R-expressions in either arguments or adjuncts can be shown with GPT 3.5 Turbo. Overall, GPT 3.5 Turbo’s judgments of sentences differ from those of humans in more ambiguous cases. However, whether movement and reconstruction play a role in human interpretations is neither fully clear from Experiment 2, as Salzmann et al. (2022) conclude in their discussion of the original experiment.

The novel possibility to reproduce an empirical experiment in linguistics with LLMs also raises many methodological questions. Are LLMs like GPT 3.5 Turbo computational accounts of how humans process language (Blank, 2023)? Can they be deployed to test hypotheses in syntax as well as other linguistic fields? Even if these questions remain open to debate, querying LLMs with test items from linguistics would also provide us with valuable insight into the ”metacognition” of these models (Begus et al., 2023). Knowing how the models respond can help engineers with improving their performance.

In Section 2, I will introduce Principle C and the syntactic considerations that led to Experiment 2 in Salzmann et al. (2022). The reports of the original experiment will be reported in Section 3. I will discuss in Section 4 the prompts and details used to query GPT 3.5 Turbo with the items from Salzmann et al. (2022). I will summarize the LLM’s results in Section 4.2. Finally, I will provide a short discussion of possible implications of using LLMs in linguistic research in Section 5.

2 Syntactic principle for referring expressions

Several foundational assumptions hold in generative syntax. Generally, its goal is to analyze language by assuming that rules generate well-formed patterns. Some of these rules, also called principles, are considered to be universal, others only apply to specific languages. It is further assumed that each language is structured hierarchically where the constituents of a syntactically formed phrase can be moved within its hierarchy to accommodate the large variety of linguistic possibilities. When a constituent is being moved, analysis states that it is generally assumed that it leaves a trace behind. However, many movements are also restricted, for example by the depth of embedding of each constituent within the hierarchy.

Among the syntactic principles, generative syntax has identified three types that restrict in particular co-reference of noun phrases (R-expressions, pronouns, anaphors). These types are called Principle A, B, and C. Principle C will be of interest to us in this contribution.

Principle C states that a pronoun cannot refer to an R-expression that it c-commands. R-expressions are proper names or definite determiner phrases, such as ”Mary” or ”the dog”. C-command is a syntactic constraint where a constituent within a phrase’s hierarchy would occupy a structurally higher position than another one from which syntactic restrictions as in Principle C follow. For example:

(1) \(^*\)He\(_{i}\) thinks that Peter\(_{i}\) is the happiest.

is ungrammatical according to Principle C when it is assumed that the pronoun should refer to the noun (see indices). It occupies a structurally higher position than the R-expression Peter. For the same reasons, a sentence like

(2) \(^*\)[Which of Lara\(_{i}\)’s sweaters]\(_{1}\) do you think she\(_{i}\) likes __\(_{1}\)?

is considered ungrammatical, even if the R-expression Lara appears on the surface before the pronoun. This can be explained with reconstructing to the trace of the movement of the wh-phrase (consider index 1 and square brackets) which occupies a lower position than the pronoun before its movement. This is also called an A’-movement since the movement does not affect the type of phrase.

In their paper, ”Condition C in German A’-movement: Tackling challenges in experimental research on reconstruction”, Salzmann et al. (2022) analyze Principle C (also called Condition C) within particular contexts. Their research follows a shift in syntax where principles that were suggested by syntactitians based on introspective judgements (sometimes described as armchair syntax) are checked through experimental setups. If principles can be confirmed in a variety of linguistic examples by a sufficient size of native speakers, the empirical evidence provides an additional justification for them. If a pattern is equivocally observed, it might not apply in a principled sense.

The aim of Salzmann et al. (2022) is to examine the workings of Principle C within three contexts. I will explain them briefly. Context 1) When the R-expression is part of an adjunct, as opposed to when it is part of an argument, the established view states that there is no Principle C reconstruction (Lebeaux, 1991; Salzmann et al., 2022). Context 2) Principle C is always violated whenever the R-expression is contained within a predicate. The containment of an R-expression within an argument does not always lead to a violation, contrary to the assumption in Context 1 (Huang, 1993; Salzmann et al., 2022). Context 3) Condition C is less strict with relative clauses than with wh-movement (Citko, 1993; Salzmann et al., 2022).

Salzmann et al. (2022) conducted three experiments in German to examine the previously mentioned syntactic phenomena. This paper focuses on their second experiment (Salzmann et al., 2022, pp. 601-609, the experiment was also registered at https://osf.io/mjgpz).

In Experiment 2, Salzmann et al. (2022) investigate coreference between pronoun and R-expression in embedded clauses. The clauses have either moved or in situ wh-phrases, serving either as objects or as subjects. Additionally, it is investigated whether the position of an R-expression as part of an argument or and adjunct makes a difference. The following two examples from Salzmann et al. (2022, 603) illustrate the conditions in Experiment 2:

(3)… [welche Geschichte im Buch über Hanna] sie __ ärgerlich fand.

’which story in the book about Hanna she found upsetting.’

(4)… [welche Geschichte über Hanna] __ sie verärgert hat.

’which story about Hanna upset her.’

Note in (3) that the object is moved and the R-expression Hanna serves as an adjunct to ”im Buch”. If Principle C were to have its assumed force in all instances, one would assume for (3) that the pronoun cannot refer to the R-expression. (Though the established view has noticed that the adjunct position indicates a less strict application of Principle C.) In (4), the wh-phrase serves as the subject which is moved and the R-expression is part of an argument. One would assume in contrast to the previous example that the pronoun can refer to the R-Expression. Also consider the for the depth of embedding and hence the structural hierarchy in both examples for the assumptions.

The aim of Experiment 2 in (Salzmann et al., 2022) is to create a statistical baseline for syntactically well-formed clauses (no violation of Principle C) through positive responses as well as a percentage of responses for those examples where Principle C would be violated if movement is assumed. If no violation in terms of negative responses is reported for ungrammatical examples, Salzmann et al. (2022) consider surface factors as another influence for the acceptability of coreference between R-expression and pronoun in such cases.

Salzmann et al. (2022, 603-604) test three hypotheses in their experiment to consider the effect of Principle C (also called Condition C) in the previously mentioned variety of contexts:

• H1 Condition C hypothesis: R-expressions cannot be coreferential with a c-commanding expression.
• H2(a) Reconstruction hypothesis: the base position of moved phrases matters for Condition C.
• H2(b) Surface hypothesis: the surface position of moved phrases matters for Condition C.
• H3 Argument/adjunct asymmetry hypothesis: in contrast to argument po- sitions, there is no reconstruction if the R-expression is part of an adjunct.

H1 assumes that phrases with an in situ subject receive more positive responses for the subject than for the object, as coreference with the object would clearly violate Principle C. Consider the following example:

(5) \(^*\)Jim erklärt, dass er die neue Geschichte über Mert ärgerlich fand.

’Jim explains that he found the new story about Mert upsetting.’

That the pronoun ”er” refers to Mert contradicts our syntactic intuition.

If H1 is confirmed, the authors investigate with H2 if more positive responses for a moved subject can be observed than with a moved object, that is, whether Principle C also holds with movement (H2(a)). If there is a significant difference between H2(a) and H1, then it is assumed that H2(b) would be confirmed, that is, surface positions influence the force of Principle C in case that the surface effects differ for objects and subjects. Finally, H3 investigates the difference of responses with instances where the R-expression is positioned as an adjunct compared to those within argument structures.

3 Experimental results from 32 participants

In this section, I will briefly describe the experimental setting of Experiment 2 from (Salzmann et al., 2022) and report their results.

3.1 Experimental setting

To test their hypotheses, Salzmann et al. (2022) developed a 2 x 2 x 2 design where the first factor was Phrase (containment of R-expression either within the subject or the object), the second Position (R-expression either as argument or adjunct) and the third Movement (in situ or moved). 32 participants were recruited via www.prolific.co. They were given 78 stimuli which comprised 32 critical items, 44 fillers, and 2 additional items for exploration.

The general set-up of each experimental round consisted in a shown example sentence with two R-expressions and a pronoun. The sentence was followed by two binary questions. Before the start of the experiment, the participants received an instruction with an example that deliberately allowed two interpretations for resolving the references of the example sentence. The participants were told to answer ”Yes” and ”Yes” in such cases. Generally, a sentence such as (5) was accompanied with two questions of the following type: Q1) ”Can this sentence be interpreted that Jim found a story upsetting?” (asking about the subject from the matrix clause ’Jim explains that…’) and Q2) ”Can this sentence be interpreted that Mert found a story upsetting?” (asking about the subject and/or object, depending on the respective condition, from the embedded clause). To avoid the order of Q1 and Q2 as a confound, the questions were randomly switched. Salzmann et al. (2022) collected the respective ”Yes” and ”No” answers for all stimuli. On average, the authors report that the experiments took 24 minutes until completion.

3.2 Results from 32 participants

For their 32 participants, Salzmann et al. (2022) report the results for all conditions as shown in Table 1.

Table 1: Results of Experiment 2 with proportion of positive answers

Noticeably, the interpretations of questions of type Q1 with respect to the refer- ring R-expression were accepted in nearly all stimuli. This comes as no surprise as its intention was to establish a baseline for syntactically well-formed patterns where the subject from the matrix clause occupies the structurally highest position. Given that some noise is considerably part of every empirical evaluation, 100% positive answers for the well-formed examples would be more concerning. A greater difference is observable for those readings that were elicited with Q2. Q2 received a proportion of more than 50% positive answers where the R-expression was part of a subject phrase. To a lesser degree, Q2 was accepted when it was contained within an object phrase, almost never when the R-expression stayed in situ (consider example (5)).

With their findings, the authors fit two generalized linear mixed models, following the recommendations of Bates et al. (2015), where the items are considered as random effects. For both models the baseline of Phrase was ’object’ and for Position ’argument’. To test all hypotheses two contrast encodings were necessary so that Movement received ’in situ’ as its baseline in Model 1 and ’moved’ in Model 2. I report their results in Table 2 and 3 where Position was originally called ”arg/adj”.

Table 2: Generalized linear mixed model results from Experiment 2 (Model 1)

Table 3: Generalized linear mixed model results from Experiment 2 (Model 2)

Salzmann et al. (2022, 606) use Model 1 and 2 to validate or reject their hypotheses. They report an effect of Phrase for the levels ’in situ’ and ’argument’ with respect to the other factors: z = 8.226, p < 0.001 in Model 1, so H1 is being confirmed. The reasoning behind this is that for sentences with subject, in situ, argument, such as in Condition e:

(6) Jim erklärt, dass die neue Geschichte über Mert ihn verärgert.

’Jim explains that the new story about Mert upsets him.’

there is no c-command for the pronoun over the R-expression in the embedded clause, compared with sentences with object, in situ, argument Condition a, as in:

(7) Jim erklärt, dass er* die neue Geschichte über Mert ärgerlich fand.

’Jim explains that he found the new story about Mert upsetting.’

An effect of Phrase was also noticed within the levels of ’moved’ and ’argument’ with respect to the other factors: z = 2.391, p = 0.017 in Model 2. This confirms H2(a) of their hypotheses: that the base position of moved phrases matters for Principle C. Compare Condition c (not the principle):

(8) Lisa erzählt, [welche Geschichte über Hanna] sie __ ärgerlich fand.

’Lisa tells which story about Hanna she found upsetting.’

with Condition g:

(9) Lisa erzählt, [welche Geschichte über Hanna] __ sie verärgert hat.

’Lisa tells which story about Hanna upset her.’

Salzmann et al. (2022) also explain in their study pre-registration: ”If there is reconstruction for Principle C, we should see the same effect in the conditions with wh-movement as in the conditions without wh-movement, irrespective of the differences in surface structure.”

The authors further confirm hypothesis H2(b) (the relevance of surface positions) because an interaction between Movement and Phrase is found in the level ’argument’ of the other factor: |z| = 5.596, p < 0.001 in both models. The reasoning is that Principle C would be reduced in contexts with movement compared to those without because on the surface the moved phrase would not look like a violation of the principle.

Finally, hypothesis H3 (argument/adjunct asymmetry) is not confirmed as there is overall no effect of Position and neither an interaction between Phrase and Position (arg/adj) within the level of ’moved’ (z = 0.087,p = 0.931 in Model 2) nor between Movement and Position (arg/adj) within ’object’ (|z| = 0.209, p = 0.834 in Model 1 and 2). Salzmann et al. (2022) argue that H3 can also only hold if H2(a) is borne out.

4 Experimental results from LLM

In section 4, I will discuss the model and prompts used in the experiment and report its results.

4.1 Experimental setting

The experiment was executed with the updated GPT 3.5 Turbo (gpt-3.5-turbo- 0125) large language model from www.openai.com. The model’s training data dates up to September 2021. For the experiment, GPT 3.5 Turbo is queried twice with an adjusted prompt for each test sentence with Q1 in the first round and Q2 in the second. This process was repeated for two and four iterations, leading to an overall data with 512 and 1024 data points for each type of questions (Q1 or Q2). Four iterations led to the same amount of data points as in Experiment 2 from Salzmann et al. (2022). The temperature of the model, which renders its answers more deterministic, was set to 0. The role – an option by OpenAI to further customize the prompts – used for the queries was the default ”user”.

Prompt P1 was chosen for the task after some manual experimentation. Since Experiment 2 from (Salzmann et al., 2022) was conducted in German, German was also used for the model instructions in the prompt (Consider the Appendix for the original prompts):

The variables ’content’ and ’question’ in the prompt referred to the different conditioned sentences that belonged to the experimental items from Salzmann et al. (2022).

Since the participants in Experiment 2 received instructions, a second prompt P2 with a similar instruction for the large language model was used.

P2 was an instance of what is commonly known as few shot learning because GPT 3.5 Turbo received one example with an ideal solution.

4.2 Results from GPT 3.5 Turbo

The closest overall results compared to the baseline of human answers can be derived from a zero shot configuration of GPT 3.5 Turbo (Table 4). In the configuration with four iterations and a zero shot prompt, Q1 received a great share of positive answers in all conditions. Q2 was almost never positively answered with phrases that contained the R-expression as an object (Condition a-d). The adjunct position led to some greater variability though (21.1% and 35.9% positive answers). For Q2, GPT 3.5 Turbo provided mostly more positive answers than humans when queried with subject phrases, with one exception (Condition e). For subjects, GPT 3.5 Turbo’s proportions of answers differ to those from humans up to 9,4%. The four times iterated one shot configuration led to a lot of positive answers for both Q1 and Q2 except for those cases where the R-expression was contained in an argument within an object phrase (Condition a and b). Again, Condition e was slightly less positively answered than the other subject conditions. The configurations with only two iterations followed the trend of both previously mentioned configurations. The configuration with a zero shot prompt and two iterations led to results which are even closer to the human baseline than the same prompt four times iterated. The greatest difference to the human baseline for the zero shot configurations can be noticed in Condition b with a difference of 35.1% and 34.3%.

Table 4: Proportion of GPT 3.5 Turbo’s positive answers, using zero and one shot prompts with 4 and 2 iterations of all items from Experiment 2 (the conditions are the same as those in Table 1)

Following Salzmann et al. (2022), I fit a generalized linear mixed model with the data from the zero shot configuration after four iterations. I chose this configuration because its amount of data points resembles the amount from Experiment 2 in Salzmann et al. (2022). The items were again considered as random effects. As with the models from Salzmann et al. (2022), the baseline of Phrase was ’object’ and for Position ’argument’. Similarly, two baselines for Movement were used to account for the differences between ’in situ’ (Model 3) and ’moved’ (Model 4). I report the results in Table 5 and 6.

Table 5: Generalized linear mixed model results from the zero shot configuration with four iterations (Model 3)

Table 6: Generalized linear mixed model results from the zero shot configuration with four iterations (Model 4)

With Model 3, an effect of Phrase can be observed for the levels ’in situ’ and ’argument’ from Movement and Position with z = 5.844, p < 0.001 confirming H1 from Experiment 2. The same effect is observed with Model 4 where Phrase leads to z = 6.075, p < 0.001. If the interpretation of the LLM results were to follow Salzmann et al. (2022), the effect of Phrase in either generalized linear mixed model with ’in situ’ as well as ’moved’ as baseline for Movement would imply the acceptance of H2(a). However, no effect of Movement nor an interaction of Movement in the other levels is observed in Model 3 and 4, rejecting H2(b). As H2(a) appears to be dependent on Movement, it is questionable whether H2(a) can be accepted when no noticeable effect of Movement is given. Salzmann et al. (2022) predicted an effect of Position if H2(a) is rejected. Precisely, this effect is observed in Model 3/4 with z = 3.705,p = 0.001 and z = 4.697,p < 0.001, that is, where the appearance of the R-expression in either an argument or an adjunct position makes a difference. Position also interacts with Phrase in both models.

5 Discussion of LLMs in linguistic research

The implications of the results as well as the methodical approach presented in this contribution appear to be so extensive that I will only provide a few thoughts in this section.

5.1 Interpretation of results

To begin with, the results from the experiment with GPT 3.5 Turbo as well as with human participants clearly indicate the presence of Principle C (hypothesis H1). Generally, questions of type Q2 were answered in such a fashion that the pronoun in the given example was only referring to the corresponding R-expression from the main clause whenever the second R-expression was part of an embedded object. An increase of positive proportions for Q2 is noticed when the second R-expression is part of an embedded subject phrase.

The disagreement between the LLM and human responses with respect to hypothesis H2(a&b) aggravates the question of whether reconstruction is part of processing sentences with Principle C. While the results from 32 participants indicate an effect of movement which Salzmann et al. (2022) interpret as possible evidence for reconstruction, that is, the application of Principle C in Syntax under the assumption that the pre-moved position is the key to its force, the large language model responses do not indicate such an effect. This raises multiple questions: Does a language model operate differently when applying Principle C than human beings? Is movement the right variable for investigating reconstruction? Do human beings really reconstruct? Salzmann et al. (2022, 607) also question their results from Experiment 2: ”How can the finding that the base position plays a role (pointing toward reconstruction) be reconciled with the finding that the surface position also matters (speaking against reconstruction)?” In other words, an effect of movement is attested, but it is difficult to reconcile with the fact that the surface position of the moved phrase also influences the participants’ interpretations. Given that the proportions of answers from their experiment does not lead to clear numbers, and that the authors attempt to avoid a logic of arbitrary thresholds (e.g., share of proportions below 20% should be considered as noise), they reject strong reconstruction with Principle C as the governing factor of coreference in German A ́-movements. The LLM’s results appear to confirm this conclusion because the LLM’s responses were not considerably influenced by movement.

The effect of position (hypothesis H3) is confirmed with the LLM. The results from 32 participants indicated that the adjunct and argument position were not treated differently and for that matter H3 was rejected in Salzmann et al. (2022). However, the LLM corroborates the established view (Lebeaux, 1991) that there is an asymmetry between the argument and adjunct position as Position led to a significant effect in the generalized linear mixed model that was fit with the LLM’s responses.

5.2 The use of LLMs in linguistic research

The idea of using models to simulate responses in linguistic experiments has been previously explored with smaller language models based on recurrent neural networks (RNNs), as seen in Linzen et al. (2016) and Futrell et al. (2019). This work culminated in the development of SyntaxGym, a platform led by Jennifer Hu, Jon Gauthier, Ethan Wilcox, Peng Qian, and Roger Levy at the MIT Computational Psycholinguistics Laboratory. SyntaxGym’s objective is to evaluate how well deep neural network models align with human intuitions about grammatical phenomena. Transformer-based models were later integrated into the platform. Additionally, research comparing neural language models with human responses in psycholinguistic studies is presented in Wilcox et al. (2021), while Li et al. (2024) examine the alignment between humans and large language models in processing garden-path sentences. Haider (2023) offers a qualitative perspective on potential applications of large language models in linguistic research. Nevertheless, the scientific validity of using language models in this context remains a topic of debate.

At the outset, we might wonder: ”Do large language models (LLMs) constitute a computational account of how humans process language?” (Blank, 2023, 987) For Blank, this question is inherently linked to how the mind is understood: Does it work on symbolic or distributed structures? I cannot discuss this question in more detail here. But I would like to suggest a pragmatic affirmation of the initial question: Let us assume that current LLMs can serve as an account of how humans process language. As it appears to me, we would then find ourselves in a situation that might be abstracted as follows:

Syntax Theory ← Human Language → Large Language Model

where both, syntax theories and LLMs, approximate the workings and patterns of human language. The former does so based on symbolic rules, whereas the latter is running on ”distributed patterns of activation across hidden units within LLMs (i.e., the ‘neurons’ of the LLM)” (Blank, 2023, 987). If we do research in linguistics, for example syntax, by using LLMs in experiments, I believe that we would look at the following situation:

Syntax Theory ← Large Language Model ← Human Language

This line of research seems acceptable, if we agree that the LLM is good enough in approximating human language, although it would certainly have substantial differences compared with human participants (e.g., the LLM’s linguistic capacity is not grounded in a ”human-like trajectory of language acquisition” (Blank, 2023, 988)).

The application of large language models (LLMs) in empirical syntax, as proposed in Chapter 4, serves as further evidence that they can enhance theory testing. In general, this research approach aims not only to simulate human language utterances using LLMs for specific research purposes but also to generate linguistic judgments (e.g., by prompting ”Yes” or ”No” responses from an LLM). The data produced by LLMs could then be used to support linguistic theories, such as providing evidence for the existence of Principle C in German.

If this approach proves successful and scientifically valid, using LLM simulations could significantly streamline linguistic research. Researchers would no longer need to recruit participants, create filler items for distraction, or rely on laboratory settings for experiments. Instead, a syntactician could simply return to her armchair, testing theories with a computer and proper access to an LLM.

However, a significant dependence on the underlying LLMs would also ensue. If the LLM does not represent human language appropriately, its errors would be propagated and syntactic theories might be accepted or rejected based on the LLM’s incomplete representation of human language. The decision to split up the questions Q1 and Q2 for separate queries to reduce the complexity of the task (also consider the one-shot prompt that rather ”confused” the LLM) indicates that GPT 3.5 Turbo is still processing language differently than human beings. This suggests, if an LLM is used in syntactic research, we must consider its limits and capacities thoroughly before we deploy it.

5.3 The implication of linguistic experiments for LLMs

Deploying LLMs in linguistic research would also help with understanding and possibly improving LLMs. Begus et al. (2023) argue that ”testing complex metalinguistic abilities of large language models is now a possible line of inquiry” and ”that the results of such tests can provide valuable insights into LLMs’ general metacognitive abilities.” (Begus et al., 2023) For example, Begus et al. (2023) created several tests to specifically check how GPT 4 provides formal and theoretical answers to syntactic, semantic, and phonological questions. In similar vain, Wilson et al. (2023) used linguistically inspired tests to investigate how LLMs process argument structures. In particular, they investigated if LLMs can generalize from argument patterns that are recognizable in different contexts. They report that RoBERTa, BERT, and DistilBERT fail some of their tests because they observed that these models are prone to errors ”based on surface level properties like the relative linear order of corresponding elements.” (Wilson et al., 2023, 1391) (This could be a reason why GPT 3.5 Turbo did not show signs of reconstruction. ) Therefore, such experiments would not only help us with understanding the LLMs’ limits, but could also be used for improving future models.

The deployment of GPT 3.5 Turbo for pronoun resolution in contexts that are covered by Principle C has shown that few shot learning does not improve the model’s answers. The ambiguous example given in the instruction rather nudged the LLM into accepting more syntactically incorrect interpretations than without. This suggests that few-shot learning may not always be helpful for LLMs if ambiguous phrases are at stake. Furthermore, GPT 3.5 Turbo was generally sensitive to the prompt. An even shorter wording for the task led in manual experimentation to diminished results. This finding corroborates the recent discussion of LLM’s fickleness by Fourrier et al. (2024).

Project scripts

The scripts used for the experiments can be found here: https://github.com/omseeth/using_llms_for_syntactic_research

Appendix

I would like to thank Martin Salzmann for providing me with the materials and experimental data from Salzmann et al. (2022).

Bibliography

Adger, D., Drummond, A., Hall, D., and van urk, C. (2017). Is there condition c reconstruction? In Lamont, A. and Tetzloff, K., editors, Nels 47: Proceedings of the forty-seventh annual meeting of the North East Linguistic Society, volume 1, pages 21–30. GLSA.

Bates, D. M., Kliegl, R., Vasishth, S., and Baayen, H. (2015). Parsimonious mixed models. arXiv: Methodology.

Begus, G., Dabkowski, M., and Rhodes, R. (2023). Large linguistic models: An- alyzing theoretical linguistic abilities of llms. arXiv: Computation and Language.

Blank, I. A. (2023). What are large language models supposed to model? Trends in Cognitive Sciences, 27(11):987–989.

Bruening, B. and Khalaf, E. A. (2018). No argument–adjunct asymmetry in reconstruction for binding condition c. Journal of Linguistics, 55:247–276. Citko, B. (1993). Deletion Under Identity in Relative Clauses. Proceedings of the North Eastern Linguistic Society (NELS), 31:131–145.

Fourrier, C., Louf, R., and Kurt, W. (2024). Improving prompt consistency with structured generations. https://huggingface.co/blog/evaluation-structured-outputs.

Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., and Levy, R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.

Haider, H. (2023). Is chat-gpt a grammatically competent informant? ling-buzz/007285.

Huang, J. (1993). Reconstruction and the Structure of VP: Some Theoretical Consequences. Linguistic Inquiry, 24(1):103–138.

Lebeaux, D. (1991). Relative Clauses, Licensing, and the Nature of the Derivation. Perspectives on Phrase Structure: Heads and Licensing, 25:209–239.

Li, A., Feng, X., Narang, S., Peng, A., Cai, T., Shah, R. S., and Varma, S. (2024). Incremental comprehension of garden-path sentences by large language models: Semantic interpretation, syntactic re-analysis, and attention.

Linzen, T., Dupoux, E., and Goldberg, Y. (2016). Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.

Salzmann, M., Wierzba, M., and Georgi, D. (2022). Condition C in German A-movement: Tackling challenges in experimental research on reconstruction. Journal of Linguistics, 59(3):577–622.

Stockwell, R., Meltzer-Asscher, A., and Sportiche, D. (2021). There is reconstruction for condition c in english questions. In Farinelly, A. and Hil, A., editors, Nels 51: Proceedings of the fifty-first annual meeting of the North East Linguistic Society, volume 2, pages 205–214. GLSA.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Wilcox, E., Vani, P., and Levy, R. (2021). A targeted assessment of incremental processing in neural language models and humans. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 939–952, Online. Association for Computational Linguistics.

Wilson, M., Petty, J., and Frank, R. (2023). How abstract is linguistic generalization in large language models? experiments with argument structure. Transactions of the Association for Computational Linguistics, 11:1377–1395.