Publications and talks
Drive my dead thoughts over the universe
Like wither’d leaves to quicken a new birth!
— Percy Bysshe Shelley, Ode to the West Wind
Book
-
Radical AI Interpretability
Cambridge University Press, with Ben Levinstein
DraftarXivAbstract
We develop a framework for interpreting AI systems as agents, drawing on the philosophical tradition of radical interpretation and the tools of mechanistic interpretability. The core question is: given the computational facts about a system, how do we solve for its beliefs, desires, and meanings? This matters increasingly for safety. We want to be able to trust the systems we deploy, whether by understanding their goals or, more modestly, by reliably detecting deception. Interpretability researchers are building tools to read beliefs and desires off a model’s internals, but there is no settled account of when such a tool has succeeded. This book supplies one. We propose criteria on both representationalist and interpretationist approaches, and tie each to tests current interpretability methods can carry out. A central lesson is that these attributions cannot be made piecemeal. Beliefs, desires, and the propositional structure they presuppose are jointly constrained, and a method that fixes one while measuring the others inherits whatever distortions that introduces. This holism becomes pressing for AI systems, which may not share the interpreter’s concepts. However, it also provides leverage: a system’s attitudes constrain its propositional structure, that structure constrains which attitudes can be attributed, and mechanistic interpretability can help us measure both.
Cite
@book{herrmann_forthcoming_radical, author = {Herrmann, Daniel A. and Levinstein, Benjamin A.}, title = {Radical {AI} Interpretability}, publisher = {Cambridge University Press}, series = {Elements in Philosophy of {AI}}, year = {forthcoming}, eprint = {2606.26523}, archivePrefix = {arXiv}, note = {Forthcoming} }
Articles
-
Why Are the Social Sciences Hard? Two New Hypotheses
Studies in History and Philosophy of Science, with Aydin Mohseni and Gabe Avakian Orona
Blog versionAbstract
Debates about whether the social sciences are harder than the natural sciences rarely analyze what discipline-level hardness amounts to. We offer a simple formal framework that models disciplines as sets of prediction tasks with associated difficulty, and links judgments of hardness to a discipline’s best-achieved performance on such tasks and to the psychological impressiveness of those successes. Within this framework we develop two hypotheses about why the social sciences appear hard. The rigid demands hypothesis holds that, in the social sciences, we are precommitted to a narrow class of prediction tasks, which restricts the usual strategy of changing the question to a more tractable one and thereby lowers expected best performance. The fruit in the hand hypothesis holds that, due to evolutionary and cultural pressures, humans already solve many low-difficulty prediction tasks about human behaviour, so these do not register as scientific successes and harder residual tasks dominate our impressions. We show that, even if task-level difficulty distributions were identical across domains, these mechanisms would make the social sciences appear harder than the natural sciences. This undermines straightforward inferences from differences in headline successes to claims about the intrinsic difficulty of the social domain. This framework opens new lines of inquiry regarding perceptions of the difficulty of the sciences.
-
Naturalizing Natural Salience
British Journal for the Philosophy of Science, with Jacob VanDrunen
Free versionAbstract
Grice, Lewis, and Skyrms proposed similar distinctions between kinds of meaning. The meaning of terms in human language, as Lewis and Skyrms had it, is “conventional.” Skyrms presented models showing how it is possible for conventional meaning to evolve in a population without reliance on pre-existing meaning. But one might think of conventionality as coming in degrees, based on whether the evolutionary process begins with “natural saliences.” We propose a theory of natural salience and several extensions of Skyrms’s models to capture this notion. These models reveal that natural saliences can hinder, as well as help, the evolution of language.
Cite
@article{vandrunen2026naturalizing, author = {VanDrunen, Jacob and Herrmann, Daniel A.}, title = {Naturalizing Natural Salience}, journal = {The British Journal for the Philosophy of Science}, year = {2026}, volume = {77}, number = {2}, pages = {413--439}, doi = {10.1086/725654} } -
A Decision-Theoretic Approach for Managing Misalignment
Proceedings of the Second Conference of the International Association for Safe and Ethical Artificial Intelligence, with Abinav Chari, Isabelle Qian, Sree Sharvesh, and Ben Levinstein
arXivAbstract
When should we delegate decisions to AI systems? While the value alignment literature has developed techniques for shaping AI values, less attention has been paid to how to determine, under uncertainty, when imperfect alignment is good enough to justify delegation. We argue that rational delegation requires balancing an agent’s value (mis)alignment with its epistemic accuracy and its reach (the acts it has available). This paper introduces a formal, decision-theoretic framework to analyze this tradeoff precisely accounting for a principal’s uncertainty about these factors. Our analysis reveals a sharp distinction between two delegation scenarios. First, universal delegation (trusting an agent with any problem) demands near-perfect value alignment and total epistemic trust, conditions rarely met in practice. Second, we show that context-specific delegation can be optimal even with significant misalignment. An agent’s superior accuracy or expanded reach may grant access to better overall decision problems, making delegation rational in expectation. We develop a novel scoring framework to quantify this ex ante decision. Ultimately, our work provides a principled method for determining when an AI is aligned enough for a given context, shifting the focus from achieving perfect alignment to managing the risks and rewards of delegation under uncertainty.
Cite
@inproceedings{herrmann2026decision, author = {Herrmann, Daniel and Chari, Abinav and Qian, Isabelle and Sharvesh, Sree and Levinstein, Benjamin A.}, title = {A Decision-Theoretic Approach for Managing Misalignment}, booktitle = {Proceedings of the 2nd {IASEAI} Conference ({IASEAI}'26)}, year = {2026}, volume = {2}, number = {1}, pages = {217--227}, doi = {10.1609/iaseai.v2i1.43026} } -
Deference and Decision
Theory and Decision, with Gerard Rothfus
Free linkAbstract
Consider two principles of rational decision making. One says that you should never use the value of events that are inconsistent with your evidence to assess what to do. The other says that, if you know what a more informed version of yourself would do, then you should already do likewise with your current information. We show that no decision theory which agrees with evidential decision theory (EDT) and causal decision theory (CDT) whenever they agree can satisfy both principles. In particular, assuming that agents update via Bayesian conditioning, EDT satisfies the former and violates the latter, while CDT violates the former and satisfies the latter. We show that the causalist can satisfy the first (at the cost of then violating the second) if she either uses a learning rule that changes her beliefs about counterfactuals in a particular way or uses a decision rule more in line with a rule proposed by Hitchcock (2016). We then employ these results to illuminate a core tension at the heart of the debate between evidentialists and causalists.
Cite
@article{herrmann2025deference, author = {Herrmann, Daniel A. and Rothfus, Gerard J.}, title = {Deference and Decision}, journal = {Theory and Decision}, year = {2025}, volume = {101}, number = {2}, pages = {401--427}, doi = {10.1007/s11238-025-10036-0} } -
Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks
Philosophical Studies, with Ben Levinstein
Free linkAbstract
We consider the questions of whether or not large language models (LLMs) have beliefs, and, if they do, how we might measure them. First, we consider whether or not we should expect LLMs to have something like beliefs in the first place. We consider some recent arguments aiming to show that LLMs cannot have beliefs. We show that these arguments are misguided. We provide a more productive framing of questions surrounding the status of beliefs in LLMs, and highlight the empirical nature of the problem. With this lesson in hand, we evaluate two existing approaches for measuring the beliefs of LLMs, one due to Azaria and Mitchell (2023) and the other to Burns et al. (2022). Moving from the armchair to the desk chair, we provide empirical results that show that these methods fail to generalize in very basic ways. We then argue that, even if LLMs have beliefs, these methods are unlikely to be successful for conceptual reasons. Thus, there is still no lie-detector for LLMs. We conclude by suggesting some concrete paths for future work.
Cite
@article{levinstein2025still, author = {Levinstein, Benjamin A. and Herrmann, Daniel A.}, title = {Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks}, journal = {Philosophical Studies}, year = {2025}, volume = {182}, number = {7}, pages = {1539--1565}, doi = {10.1007/s11098-023-02094-3} } -
Sifting the Signal from the Noise
British Journal for the Philosophy of Science, with Jacob VanDrunen
Free versionAbstract
Signaling games are useful for understanding how language emerges. In the standard models the dynamics in some sense already knows what the signals are, even if they do not yet have meaning. In this paper we relax this assumption, and develop a simple model we call an attention game in which agents have to learn which feature in their environment is the signal. We demonstrate that simple reinforcement learning agents can still learn to coordinate in contexts in which (i) the agents do not already know what the signal is and (ii) the other features in the agents’ environment are uncorrelated with the signal. Furthermore, we show that, in the cases in which other features are correlated with the signal, there is a surprising trade-off between learning what the signal is, and success in action. We show that the mutual information between a signal and a feature plays a key role in governing the accuracy and attention of the agent.
Cite
@article{herrmann2025sifting, author = {Herrmann, Daniel A. and VanDrunen, Jacob}, title = {Sifting the Signal from the Noise}, journal = {The British Journal for the Philosophy of Science}, year = {2025}, volume = {76}, number = {3}, pages = {745--758}, doi = {10.1086/720805} } -
Standards for Belief Representations in LLMs
Minds and Machines, with Ben Levinstein
arXivAbstract
As large language models (LLMs) continue to demonstrate remarkable abilities across various domains, computer scientists are developing methods to understand their cognitive processes, particularly concerning how (and if) LLMs internally represent their beliefs about the world. However, this field currently lacks a unified theoretical foundation to underpin the study of belief in LLMs. This article begins filling this gap by proposing adequacy conditions for a representation in an LLM to count as belief-like. We argue that, while the project of belief measurement in LLMs shares striking features with belief measurement as carried out in decision theory and formal epistemology, it also differs in ways that should change how we measure belief. Thus, drawing from insights in philosophy and contemporary practices of machine learning, we establish four criteria that balance theoretical considerations with practical constraints. Our proposed criteria include accuracy, coherence, uniformity, and use, which together help lay the groundwork for a comprehensive understanding of belief representation in LLMs. We draw on empirical work showing the limitations of using various criteria in isolation to identify belief representations.
Cite
@article{herrmann2025standards, author = {Herrmann, Daniel A. and Levinstein, Benjamin A.}, title = {Standards for Belief Representations in {LLMs}}, journal = {Minds and Machines}, year = {2025}, volume = {35}, number = {1}, pages = {5}, doi = {10.1007/s11023-024-09709-6} } -
Prediction with Expert Advice Applied to the Problem of Prediction with Expert Advice
Abstract
We often need to have beliefs about things on which we are not experts. Luckily, we often have access to expert judgements on such topics. But how should we form our beliefs on the basis of expert opinion when experts conflict in their judgments? This is the core of the novice/2-expert problem in social epistemology. A closely related question is important in the context of policy making: how should a policy maker use expert judgments when making policy in domains in which she is not herself an expert? This question is more complex, given the messy and strategic nature of politics. In this paper we argue that the prediction with expert advice (PWEA) framework from machine learning provides helpful tools for addressing these problems. We outline conditions under which we should expect PWEA to be helpful and those under which we should not expect these methods to perform well.
Cite
@article{herrmann2022prediction, author = {Herrmann, Daniel A.}, title = {Prediction with Expert Advice Applied to the Problem of Prediction with Expert Advice}, journal = {Synthese}, year = {2022}, volume = {200}, number = {4}, pages = {315}, doi = {10.1007/s11229-022-03809-5} } -
PAC Learning and Occam’s Razor: Probably Approximately Incorrect
Abstract
Computer scientists have provided a distinct justification of Occam’s Razor. Using the probably approximately correct framework, they provide a theorem which they claim demonstrates that we should favour simpler hypotheses. The argument relies on a philosophical interpretation of the theorem. I argue that although, when supplemented with additional unstated assumptions, the proof of the theorem is mathematically sound, the standard interpretation of the result in the literature is misguided and that a better reading does not, in fact, support Occam’s Razor at all. In support of this, I state and prove a very similar theorem which, if interpreted the same way, would justify the contradictory Anti-Occam’s Razor, the principle that we should favour more complex hypotheses.
Cite
@article{herrmann2020pac, author = {Herrmann, Daniel A.}, title = {{PAC} Learning and {Occam}'s Razor: Probably Approximately Incorrect}, journal = {Philosophy of Science}, year = {2020}, volume = {87}, number = {4}, pages = {685--703}, doi = {10.1086/709786} }
Book reviews
-
Review of Cameron J. Buckner’s From Deep Learning to Rational Machines
-
Review of Wayne C. Myrvold’s Beyond Chance and Credence
Philosophia Mathematica, with David Peter Wallis Freeborn
Free linkCite
@article{herrmann2022myrvold, author = {Herrmann, Daniel A. and Freeborn, David Peter Wallis}, title = {{Wayne C. Myrvold}. \emph{Beyond Chance and Credence: A Theory of Hybrid Probabilities}}, journal = {Philosophia Mathematica}, year = {2022}, volume = {30}, number = {2}, pages = {273--279}, doi = {10.1093/philmat/nkac006} }
Under review & in progress
-
Subjective Naturalism and Act Probabilities
Available on request.
Abstract
Can an agent assign probabilities to her own acts while she deliberates? This question splits decision theorists into two broad camps: those who believe deliberation crowds out prediction, and those who believe that deliberation welcomes prediction. I argue that philosophers are divided because they respond in different ways to a threat posed to agency for a certain kind of naturalized agent. I introduce two conditions that capture this kind of subjective naturalism: richness and austerity. I show that taking these two conditions poses a dilemma, and that both camps respond to the dilemma by violating one of these conditions. I provide a novel account of agency, called desirability tracking, which allows us to avoid the violations of richness that the crowding out camp commits, while also avoiding the violations of austerity that the welcoming camp commits. The account also allows us to extract the decision problems an agent takes herself to face from her own attitudes, as opposed to imposing them from the outside.
-
A Bayesian Reduction of Causation
With Aydin Mohseni, Ben Levinstein, and Bruce Rushing
PreprintAbstract
De Finetti conjectured that causation, like chance, reduces to patterns in subjective probability. Recent results on the Pearl Causal Hierarchy seem to foreclose this possibility, proving that causal claims are irreducible to probabilistic ones within structural causal models (SCMs). We show that the reduction succeeds after all. The apparent irreducibility arises from an implicit restriction on the agent’s representational resources: once the variable set is enriched to include naturalized intervention variables, interventional propositions reduce to ordinary probabilistic conditioning. To complete the subjectivist picture, we prove a de Finetti-style representation theorem establishing that an agent with exchangeable credences can be represented as simultaneously learning causal structure and the probabilities within that structure via standard Bayesian updating. We then show how this framework bears on the debate between evidential and causal decision theory, arguing that causal decision theory functions as a useful approximation of evidential reasoning over the full algebra. Within SCMs, causal reasoning reduces to subjective probability.
-
Grounding Human-Machine Decision-Making
With Thomas Grote and Claire Vernade. Available on request.
Abstract
Machine learning systems now participate in many high-stakes decisions alongside human experts, yet the conceptual terrain remains fragmented. Inconsistent terminology and underspecified frameworks hinder study design and obscure when findings transfer across research contexts. We develop a unified decision-theoretic framework capturing four types of human-machine decision-making: human-in-the-loop (ML systems decide while humans monitor), decision-aid (humans decide with ML predictions), complementarity (division of labor by respective strengths), and collective decision-making (independent judgments combined by aggregation). Grounding these types in common formal apparatus reveals structural similarities often missed when studied in isolation, and predicts which failure modes can arise in each type. The framework connects the decision-aid literature to results from formal epistemology that characterize exactly when deference to a machine is rational, and it identifies two intervention points: modifying what information the machine communicates versus how humans respond. We extend these foundations to sequential settings where both parties learn over time, and discuss applications to attributing responsibility for erroneous decisions in human-machine systems.
-
Benign Interpolation and Occam’s Razor
With Tom F. Sterkenburg and Jan-Willem Romeijn
arXivAbstract
Contemporary deep learning methods generalize well even when they fit their training data perfectly, a phenomenon known as benign interpolation. This phenomenon cannot be accounted for by classical statistical learning theory and has prompted a range of attempted new explanations in the statistics and machine learning literature. A common feature of these new proposals is an appeal to a simplicity preference among interpolating models, often presented as a form of Occam’s razor. We clarify this debate for a philosophical audience and argue that this new appeal to simplicity creates an explanatory gap. The classical theory offers theorems which connect the simplicity of model classes to good generalization, thus underwriting methodological simplicity norms. The new accounts instead appeal to properties of individual models, which they interpret as a kind of simplicity. Lacking a provable connection to generalization, it is the name “simplicity” that does the work a theorem used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.
Cite
@misc{sterkenburg2026benign, author = {Sterkenburg, Tom F. and Herrmann, Daniel A. and Romeijn, Jan-Willem}, title = {Benign Interpolation and {Occam}'s Razor}, year = {2026}, eprint = {2608.03386}, archivePrefix = {arXiv}, note = {Manuscript} } -
Cartesian Frames
With Scott Garrabrant and Josiah Lopez-Wild
arXivAbstract
We introduce a novel framework, the theory of Cartesian frames (CF), that gives powerful tools for manipulating sets of acts. The CF framework takes as its most fundamental building block that an agent can freely choose from a set of available actions. The framework uses the mathematics of Chu spaces to develop a calculus of those sets of actions, how those actions change at various levels of description, and how different agents’ actions can combine when agents work in concert. We discuss how this framework might provide an illuminating perspective on issues in decision theory and formal epistemology.
Cite
@misc{garrabrant2021cartesian, author = {Garrabrant, Scott and Herrmann, Daniel A. and Lopez-Wild, Josiah}, title = {Cartesian Frames}, year = {2021}, eprint = {2109.10996}, archivePrefix = {arXiv}, primaryClass = {math.CT} } -
Measure for Measure: Problems with Probability in Decohering Worlds
Available on request.
Abstract
Quantum mechanics is one of our most successful physical theories. One prominent interpretation of quantum mechanics is the Everett interpretation, also called the “Many-Worlds Interpretation.” However, the Everett interpretation gives rise to a puzzle: probability seems to be absent from the theory. This is a problem, because the evidence we have for quantum mechanics is that its probabilistic predictions agree with the statistics of measurement outcomes. David Wallace has argued that we can use decision theory, via a representation theorem, to solve this probability puzzle. I argue that providing a representation theorem in which an agent is representable as maximizing a utility function with respect to what is mathematically a probability function is not sufficient to derive probability in the sense of credence, and thus that this solution to the problem fails.
Talks
Talks since 2024, most recent first. Other venues for the same talk are listed together.
-
Panel: The Language of AI Safety for Science
Invited AISciK Workshop, NeurIPS, Atlanta, December 2026 (upcoming)
-
A Bayesian Reduction of Causation
Philosophy of Science Association Biennial, San Diego, November 2026 (upcoming); Philosophy of Science Meets Machine Learning, Munich, 2026
-
Splitting the Atom: Atomless Semantics for Counterfactuals
Philosophy of Science Association Biennial, San Diego, November 2026 (upcoming); PPE Society Annual Meeting, New Orleans, 2025
-
Margins of Misalignment: Delegation to AI Beyond the Bayesian Idealization
PPE Society Annual Meeting, New Orleans, November 2026 (upcoming)
-
Why Are the Social Sciences Hard?
PPE Society Annual Meeting, New Orleans, November 2026 (upcoming)
-
Radical AI Interpretability
Invited Minds, Meaning, and Machines, Hong Kong, November 2026 (upcoming); AI Agency and Interpretability Workshop, Pittsburgh, 2026
-
Intrapersonal Comparisons under Preference Uncertainty
PPE Society Annual Meeting, London, 2026; Meeting of the Society for Social Choice and Welfare, Tokyo, 2026
-
Can We Attribute Beliefs and Desires to AIs? A Radical Approach
Invited Bennington Center for AI Speaker Series, Bennington, 2026
-
The Metaphor Problem for Belief Representation in Large Language Models
Invited Pacific APA, online, 2026
-
A Decision-Theoretic Approach for Managing Misalignment
International Association for Safe and Ethical AI, Paris, 2026
-
Machine Minds and the Propositional Projection Problem
Invited Central APA, Chicago, 2026
-
Comments on Cappelen and Dever’s Making AI Intelligible
Invited Eastern APA, Baltimore, 2026
-
Standards for Belief Representation in LLMs
Invited Bennington Science Workshop, 2026; Invited MCMP Research Seminar in Decision and Action Theory, Munich, 2024; Philosophy of Science Association Biennial, New Orleans, 2024; Invited InDeep Consortium Meeting, Groningen, 2024; Philosophy of Science Meets Machine Learning, Tübingen, 2024
-
Naturalizing Causation in Causal Models
Invited Agent Foundations Workshop, Pittsburgh, 2025
-
Pragmatics, Empiricism, and Self-Assembling Games
Invited Self-Assembling Games Workshop, Irvine, 2025
-
Deference and Decision
Invited Agent Foundations Workshop, Pittsburgh, 2025; Invited Pittsburgh Formal Epistemology Workshop, 2025; Invited UCI Game Theory Lab, Irvine, 2025
-
Manageable Misalignment
Invited Agent Foundations Workshop, Pittsburgh, 2025; PPE Society Annual Meeting, New Orleans, 2024
-
Decision Theory Foundations
Invited ILIAD Conference, Berkeley, 2024
-
Measuring the Beliefs of LLMs
Invited Purdue University Philosophy Colloquium, West Lafayette, 2024; Invited AI and Logic Talk, Groningen, 2024