This project was funded by the German Research Foundation. The first funding period established the Associative Read-Out Model (AROM) as the first interactive activation model including semantic representations. To define associative relations between symbolic units , we used the log likelihood that two words co-occur more often together in the sentences of a large text corpus than predictable by their single word frequencies. In this first period we focused primarily on implicit word recognition, while we focused on explicit memory in the second funding phase.

Before this project started, reading research has already converged on the opinion that single-word features affect visual word recognition. If there was a consensus on how to define associative-semantic relations, human performance in one task was taken to predict human performance in another task, which we saw as circular reasoning. Luckily, now more researcher seem to agree that a language-model based, computational definition of associative-semantic relations does not only save time, but some also agree that deriving such measures directly from a sample of human experience, i.e. text corpora, is a deeper explanation than circular reasoning. The project used information about word pairs likely co-occurring in sentences to define associations between words presented in sequence. The AROM – a neurocognitive computer simulation model – used this information to predict which word is recognized with which probability from the associative-semantic language context. The model was tested in a series of experiments at which participants judge the association strength of word pairs, do lexical decisions, or while they read or complete sentences. Moreover, we generalized the model to account for brain data. Thus we successfully tested the hypotheses that the visual feature units of the model can predict early occipital activation, that associative activations predict N400 amplitudes in brain-electric measures, or that associative competition affects left inferior frontal gyrus activation. Finally, our general definition of associative-semantic relations opened many theoretical and applied perspectives for the research fields of reading or psychology.

While the initial funding period focused on word recognition in the cortical long-term memory store, the second phase intended to add a hippocampal layer, including conjunction units that store a pattern separated representation of episodic memory traces. As the resulting complementary learning systems model provided fully symbolic representations, we see this as a way more explainable AI than state-of-the-art transformer models. In recognition memory tasks, we used the semantic feature overlap with studied and prime words to  predict recognition probabilities of words. We also examined the combination of semantic long-term and purely episodic associations within sentences for recognition memory. In eye tracking and brain-electric studies, we addressed the dynamics of implicit and explicit encoding and retrieval processes, while the neuroimaging studies aimed at providing a better understanding of the interplay between activated semantic long-term and episodic information in the human hippocampus.

At present, we are continuing with this line of research in the dissertation of Leo Sokolovič, where we use these models for the applied question of how executive symptoms in Alzheimer disease can be theoretically separated from impairments in episodic vs. semantic memory. Our recent Bayesian meta-analysis targeted the question which diagnostic markers can differentiate between Alzheimer’s disease, (subcortical) vascular vs. multi-infarct dementia, as well as (mild) cognitive impairment with a vascular etiology. While patients with vascular dementia performed worse in working memory and cognitive control tasks, Alzheimer disease was characterized by poor recall and recognition memory performance. Our localist connectionist simulations of recognition memory performance already allowed for a theoretical separation episodic and semantic processes. As Alzheimer also can come with executive symptoms, we recently extended this model by an explicit decision layer using leaky-noisy accumulators, to generate a solid theoretical foundation for the development of future diagnostic instruments.

Funding periods: 2014-2017 (HO 5139/4-1), 2017-2022 (HO 5139/4-2 and RA 1603/4-2) for Andre Rölke-Wellmann, Carsten Klein and Mareike Kleemann

Author: Markus J. Hofmann and Leo Sokolovic

(Selected) References

Peer-reviewed journals

• Sokolovič, L., & Hofmann, M. J. (2024). How to say ‘no’to a false memory: Leaky and noisy evidence accumulation during associative read-out.  Computational Brain & Behavior, 7, 357–377.
• Hofmann, M. J., Kleemann, M. A., Roelke-Wellmann, A., Vorstius, C., & Radach, R. (2022). Semantic feature activation takes time: longer SOA elicits earlier priming effects during reading. Cognitive Processing, 23, 309-318.
• Hofmann, M. J., Remus, S., Biemann, C., Radach, R., & Kuchinke, L. (2022). Language Models Explain Word Reading Times Better Than Empirical Predictability. Frontiers in Artificial Intelligence, 4, 1-20.
• Roelke, A., & Hofmann, M. J. (2020). Functional connectivity of the left inferior frontal gyrus during semantic priming. Neuroscience Letters, 735 (135236), 1-7.
• Roelke, A., Vorstius, C., Radach, R., & Hofmann, M. J. (2020). Fixation-related NIRS indexes retinotopic occipital processing of parafoveal preview during natural reading. NeuroImage, 215(116823), 1-11.
• Hofmann, M. J., Biemann, C., Westbury, C., Murusidze, M., Conrad, M., & Jacobs, A. M. (2018). Simple co‐occurrence statistics reproducibly predict association ratings. Cognitive Science, 42(7), 2287-2312.
• Roelke, A., Franke, N., Biemann, C., Radach, R., Jacobs, A. M., & Hofmann, M. J. (2018). A novel co-occurrence-based approach to predict pure associative and semantic priming. Psychonomic Bulletin & Review, 25, 1488–1493.
• Stüllein, N., Radach, R., Jacobs, A. M., & Hofmann, M. J. (2016). No one way ticket from orthography to semantics in recognition memory: N400 AND P200 effects of associations. Brain Research, 1639, 88-98.
• Hofmann, M. J., & Kuchinke, L. (2015). “Anything is good that stimulates thought” in the hippocampus Comment on “The quartet theory of human emotions: An integrative and neurofunctional model” by S. Koelsch et al. Physics of Life Reviews, 13, 58-60.
• Hofmann, M. J., Dambacher, M., Jacobs, A. M., Kliegl, R., Radach, R., Kuchinke, L., Plichta, M.M., Fallgatter, A. M., & Herrmann, M. J. (2014). Occipital and orbitofrontal hemodynamics during naturally paced reading: An fNIRS study. Neuroimage, 94, 193-202.
• Hofmann, M. J., & Jacobs, A. M. (2014). Interactive activation and competition models and semantic context: from behavioral to brain data. Neuroscience & Biobehavioral Reviews, 46, 85-104.
• Hofmann, M. J., Kuchinke, L., Biemann, C., Tamm, S., & Jacobs, A. M. (2011). Remembering words in context as predicted by an Associative Read-Out Model. Frontiers in Psychology, 2, 252, 1-11.

Conference proceedings

• Franke, N., Roelke, A., Radach, R., & Hofmann, M. J. (2017). After braking comes hasting: reversed effects of indirect associations in 2nd and 4th graders. Proceedings of the Cognitive Science Society (pp. 2025-2030), London, UK.
• Biemann, C., Remus, S., & Hofmann, M. J. (2015). Predicting word ’predictability’ in cloze completion, electroencephalographic and eye movement data. In Proceedings of Natural Language Proceesing and Cognitive Science (pp. 83-93). Krakow, Poland, 22.-24.9.2015.

Book chapters

• Hofmann, M. J., Biemann, C., & Remus, S. (2017). Benchmarking n-grams, topic models and recurrent neural networks by cloze completions, EEGs and eye movements. In: Sharp, B., Sedes, F., Lubaszewski, W. (Eds.). Cognitive Approach to Natural Language Processing (pp. 197-215), ISTE press, Elsevier.
• Radach, R. & Hofmann, M. J. (2016). Graphematische Verarbeitung beim Lesen von Wörtern. In: Domahs, U. & Primus, B. Laut, Gebärde, Buchstabe (Handbuch Sprachwissen, Band 2) (pp. 455-474), De Gruyter.