Parts of an AI Scientist

This is a collection of implementation notes as I work through a unified implementation of ideas from Model Discovery Agent and program-synthesis for SBI as well as a number of others. It explains ideas and parts of the code. The point of this work is to ultimately create a framework to assess the performance of LLM-driven systems where the boundary between the agent and the tooling can be drawn at different levels; how much modular, external formal machinery is required and how much can we subsume inside a reasoning system, while still achieving the same performance. In Kevin Murphy's work, the LLM is dropped in to specific subcomponents, whereas others allow a reasoning model to take over most work that does not involve the external world. I hope to make something that allows all the ideas around this to be benchmarked on an equal footing, with the different subcomponents made clear, and consequently allow diagnosis of where there are failures and what needs to be improved. Code referred to is found at github.com/megstanley/modeldiscoveryagent.

This introduction will be extended in time! For now, to be clear these are just rough implementation notes as I worked through the components 🙂