Knowledge-free and knowledge-based Theory of Mind reasoning in Large Language Models

Breaking the illusion: Revisiting LLM anthropomorphism

HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation

As LLMs have demonstrated remarkable performance across diverse domains, researchers have often utilized human categories to describe and evaluate their behavior. Such anthropomorphism results in the application of expectations, benchmarks, and interpretations typically reserved for humans to LLMs. LLM anthropomorphism has a number of benefits, such as facilitating understanding of LLMs, but risks misrepresenting fundamental differences between humans and LLMs and the reality of the progress being made. With that dichotomy in mind, we explore practical taxonomies for the application of anthropomorphic terms and human benchmarks to LLMs that mitigate the risks of LLM anthropomorphism.

Large Language Models (LLMs) have recently shown promise and emergence of Theory of Mind (ToM) ability and even outperform humans in certain ToM tasks. To evaluate and extend the boundaries of the ToM reasoning ability of LLMs, we proposed a novel concept, taxonomy, and framework, that Knowledge-Free and Knowledge-Based ToM reasoning, and developed a multi-round text-based game, called Pick the Right Stuff, as a benchmark. We have evaluated seven LLMs with this game and found their performance on Knowledge-Free tasks is consistently better than on Knowledge-Based tasks. In addition, we found that one of the models with the small parameter size, mistral: 7b-instruct, shows similar performance to other evaluated models with large parameter sizes and even outperforms several of them. Furthermore, it even achieved a performance almost similar to gpt-4o on Knowledge-Based tasks. These results raise a thought-provoking question about whether increasing model parameter size may effectively enhance LLMs capabilities, at least discussed in the context of ToM reasoning ability. We expect this work to offer insights into the ToM reasoning ability of LLMs and to pave the way for the future development of ToM benchmarks and also for the promotion and development of more complex AI agents or systems that are required to be equipped with more complex ToM reasoning ability.

While Large Language Models (LLMs) have demonstrated impressive abilities across various domains, they still struggle with complex problems characterized by multi-objective optimization, precise constraint satisfaction, immense solution spaces, etc. To address the limitation, drawing on the superior semantic understanding ability of LLMs and also the outstanding global search and optimization capability of genetic algorithms, we propose to capitalize on their respective strengths and introduce Lyria, a general LLM-driven genetic algorithm framework, comprising 7 essential components. Through conducting extensive experiments with 4 LLMs across 3 types of problems, we demonstrated the efficacy of Lyria. Additionally, with 7 additional ablation experiments, we further systematically analyzed and elucidated the factors that affect its performance.

quicknote.nvim

Quickly take notes, in-place

May, 2022

Quickly take notes, in-place

May, 2022