special_topics_unconventional_AI

Introduction

A reading list for a special topics class (R255) at the Computer Science and Technology Department at the University of Cambridge. This is part of Advanced Topics in Machine Learning.

The title of the course is:

Unconventional approaches in AI: complex systems perspectives, cognitive psychology, social sciences, computational models of creativity, explainable AI inspired by other disciplines and other unconventional models

This is AI or classical AI before big data. The time is now ripe to revisit these wonderful ideas and think about how to incorporate them in modern AI/deep learning. Insights from the past can inform future approaches to AI, especially in the age of big data.

Looking at the heritage of computing and its interdisciplinary past can inspire new approaches for the future. We need to learn lessons from the history of AI, what approaches worked and did not work in the past and how AI went through multiple winters.

These approaches can be used to develop techniques that can inspire explainable AI.

We will also read science fiction stories to understand the philosophy and ethics of AI!

Talk describing this topic

• Short introductory talk

https://www.youtube.com/watch?v=o7EXf265sTU

• Full talk

https://youtu.be/8s4vVPTGVfw

• Slides

https://github.com/neelsoumya/special_topics_unconventional_AI/blob/main/intro.pdf

https://github.com/neelsoumya/special_topics_unconventional_AI/blob/main/wrapup.pdf

Cognitive psychology, computational models of creativity and other unconventional models in AI

• Computational models of creativity and scientific insight (Pat Langley)

https://escholarship.org/content/qt54x8v354/qt54x8v354.pdf

• BACON.5: the discovery of conservation laws (Pat Langley)

https://dl.acm.org/doi/abs/10.5555/1623156.1623181

• A Computational Inflection for Scientific Discovery

https://cacm.acm.org/magazines/2023/8/274938-a-computational-inflection-for-scientific-discovery/fulltext

• Copycat (a computational model of analogy making)

https://dspace.mit.edu/handle/1721.1/5648

• The emergence of understanding in a computer model of concepts and analogy-making

https://www.sciencedirect.com/science/article/abs/pii/0167278990900865?via%3Dihub

• Structure mapping engine: Algorithms and examples

https://doi.org/10.1016/0004-3702(89)90077-5

• Learning new principles from precedents and exercises (Winston)

https://doi.org/10.1016/0004-3702(82)90004-2

• The overfitted brain: Dreams evolved to assist generalization

https://doi.org/10.1016/j.patter.2021.100244

• DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning

https://arxiv.org/abs/2006.08381

• On the measure of intelligence (Abstraction and Reasoning Corpus)

https://arxiv.org/abs/1911.01547

• Commonsense reasoning

https://dl.acm.org/doi/10.1145/2701413

Commonsense reasoning, Cyc and large language models

https://arxiv.org/pdf/2308.04445.pdf

Cyc database of commonsense reasoning (Doug Lenat and Gary Marcus)

http://web.archive.org/web/20230902080842/https://garymarcus.substack.com/p/doug-lenat-1950-2023

• Stories and narratives (Patrick Winston)

https://nautil.us/the-storytelling-computer-8380/

https://dspace.mit.edu/handle/1721.1/67693

• Argument technology for debating with humans

IBM Project Debater

https://www.nature.com/articles/d41586-021-00539-5

• Case based reasoning (Janet Kolodner)

http://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf

Papers from philosophy and consciousness studies

Can we (and should we) have consciousness in machines?

• Consciousness in Artificial Intelligence: Insights from the Science of Consciousness

https://arxiv.org/pdf/2308.08708.pdf

• Levels of AGI: Operationalizing Progress on the Path to AGI

https://arxiv.org/pdf/2311.02462.pdf

Papers related to cognitive architectures

• 40 years of cognitive architectures: core cognitive abilities and practical applications

https://link.springer.com/article/10.1007/s10462-018-9646-y

Papers related to understanding and reasoning in large language models (LLMs): The philosophy and psychology of large language models

• Can large language models reason or understand?

https://arxiv.org/pdf/2210.13966.pdf

• Are they capable of consciousness?

https://www.economist.com/by-invitation/2022/09/02/artificial-neural-networks-are-making-strides-towards-consciousness-according-to-blaise-aguera-y-arcas

• World models in large language models

https://arxiv.org/pdf/2310.02207.pdf

• Nature review

https://www.nature.com/articles/d41586-023-02361-7

• Sparks of Artificial General Intelligence: Early experiments with GPT-4

https://arxiv.org/abs/2303.12712

• A Theory for Emergence of Complex Skills in Language Models

https://arxiv.org/pdf/2307.15936.pdf

• LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

https://arxiv.org/pdf/2305.18354.pdf

• Large Language Models as General Pattern Machines

https://arxiv.org/abs/2307.04721

• Hubert Dreyfus’s crtique of Winograd’s block worlds program

https://archive.org/details/whatcomputerscan017504mbp/page/n39/mode/2up

• Can LLMs understand? (Far from being “stochastic parrots,” the biggest large language models seem to learn enough skills to understand the words they’re processing.)

https://www.quantamagazine.org/new-theory-suggests-chatbots-can-understand-text-20240122/

• Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

https://arxiv.org/abs/2305.00948

• Theory of Mind benchmark for large language models

  https://arxiv.org/abs/2402.06044

  https://github.com/seacowx/OpenToM

• Are Emergent Abilities of Large Language Models a Mirage?

  https://arxiv.org/abs/2304.15004

• A Philosophical Introduction to Language Models

  https://arxiv.org/abs/2401.03910

• Machine Psychology

  https://arxiv.org/abs/2303.13988

Artificial Stupidity

• Lessons for artificial intelligence from the study of natural stupidity

https://www.nature.com/articles/s42256-019-0038-z

Papers related to explainable AI (xAI) inspired by other disciplines

• Explanation in artificial intelligence: Insights from the social sciences (Miller)

https://arxiv.org/abs/1706.07269

Collective intelligence

• Collective intelligence for deep learning: A survey of recent developments

https://journals.sagepub.com/doi/full/10.1177/26339137221114874

• Eric W. Bonabeau. Control mechanisms for distributed autonomous systems: insights from social insects. 2001

In Design Principles for the Immune System and Other Distributed Autonomous Systems.

https://www-users.cs.york.ac.uk/susan/books/pages/s/LeeASegel.htm#9582

(login with your RAVEN ID and search the university library webpage)

https://idiscover.lib.cam.ac.uk/

• Neural cellular automata

https://distill.pub/2020/growing-ca/

Reading science fiction and understanding the philosophy and ethics of AI

Some papers and readings on using science fiction to understand the philosophy and ethics of AI.

1. The Mind’s I: Fantasies and reflections on self and soul. By Douglas Hoffstadter and Daniel Dennett

2. https://cacm.acm.org/research/how-to-teach-computer-ethics-through-science-fiction/

Miscellaneous Resources

• Analogy in AI

https://melaniemitchell.me/PostdocProjectDescription.pdf

• Abstraction and Reasoning Corpus

https://github.com/fchollet/ARC

• Abstraction and Reasoning Corpus Challenge

https://blog.jovian.ai/finishing-2nd-in-kaggles-abstraction-and-reasoning-challenge-24e59c07b50a

https://github.com/alejandrodemiquel/ARC_Kaggle

Domain specific languages may be required (as suggested by Chollet) like genetic algorithms and cellular automata

• Other paths to intelligence elsewhere in the animal kingdom

https://nautil.us/another-path-to-intelligence-23113/

• The storytelling computer

https://nautil.us/the-storytelling-computer-237502/

http://web.archive.org/web/20221102094120/https://nautil.us/the-storytelling-computer-237502/

• The Psychology of Invention in the Mathematical Field (Jacques Hadamard)

https://archive.org/details/eassayonthepsych006281mbp/page/n35/mode/2up

• The Theory of Mind

https://en.wikipedia.org/wiki/Theory_of_mind

• The structure of scientific revolutions, Kuhn

Code repositories and implementations

https://github.com/Tijl/ANASIME

https://github.com/crazydonkey200/SMEPy

https://github.com/fargonauts/copycat

Presentations on selected papers

Present and lead a discussion on one of these papers (or any other related paper: come speak with me). The idea is that you raise some interesting questions. This course is meant to teach you research skills (like thinking critically about a paper and literature review skills).

Writeup

In this course, each student would chose one paper. They would then do a presentation on it.

Towards the end of the term they would do a writeup/short report:

• on this paper, and

• the topic in general (unconventional AI). They would do a literature review of other papers in the field.

• They will then reflect/write on how these techniques can be incorporated in modern AI/deep learning.

The intention is for students to learn how to read papers, and compare and contrast them to other papers and then evaluate what this means for AI/deep learning.

Some writing prompts for the writeup are here:

• Short report (less than 4000 words). The idea is write a coherent narrative.

• Suggest how these ideas can be incorporated in modern AI/deep learning systems

• Why do you think these ideas were not successful in the 1950s/1960s?

• What kind of data would we need to ensure these techniques would work today?

• What lessons can we learn from the history of AI, what approaches worked and did not work in the past?

• What could be the disadvantages of these approaches?

• Rational reconstruction (analytical literature review/survey) of a research area

Other thoughts on the writeup:

• A detailed research proposal with some ground work already accomplished

• A hybrid of all of the above

Thoughts on a project:

• If you feel ambitious, you can do a coding project based on open source data (talk to me if you would like to do this).

Administrivia

https://www.cl.cam.ac.uk/teaching/2122/R255/

https://github.com/neelsoumya/special_topics_unconventional_AI/blob/main/admin_notes.md

Miscellaneous

• How to read papers

https://www.cs197.seas.harvard.edu/

https://docs.google.com/document/d/1bPhwNdCCKkm1_adD0rx1YV6r2JG98qYmTxutT5gdAdQ/edit#heading=h.yxlvj6bo3y2

• How to write

Write regularly

Keep a schedule

https://sites.google.com/site/neelsoumya/research-resources/scientific-writing

Video on writing

https://www.youtube.com/watch?v=DeVjXINr5Wk

Book on writing (please contact me to borrow a copy; also available from the library digitally)

How to Write a Lot: A Practical Guide to Productive Academic Writing by Paul J Silvia

Contact

You can also pick other papers that are broadly in this area/topic and that excite you. Please contact me to discuss further.

Soumya Banerjee

sb2333@cam.ac.uk

neel.soumya@gmail.com

Office: FC01 (Computer Science and Technology Department)

https://sites.google.com/site/neelsoumya/Home

https://github.com/complexsystemslab/project_ideas/blob/main/project_ideas.md