Video summary
Occam's Razor, a fundamental principle in science attributed to William of Ockham (referred to here via Marcus Hutter), posits that among competing hypotheses or models that equally well describe observed data, one should select the simplest one. While this is not an absolute law but rather a heuristic for finding understanding and predictive power, its effectiveness can be grounded in Solomonoff induction. This theory rigorously demonstrates that if we assume the world operates under simple rules, Occam's Razor emerges as the optimal mechanism for discovering them. The principle suggests that simpler models possess greater reliability because they are less likely to overfit data or explain phenomena without genuine predictive capability, whereas overly complex models often fail to generalize beyond their specific training sets. The appeal of simplicity extends into evolutionary biology and human cognition; finding patterns in nature is a survival mechanism essential for avoiding danger, such as identifying predators like tigers among bushes. However, humans are prone to seeing patterns where none exist, leading to phenomena like the "Mars face" illusion. Solomonoff induction formalizes this search for regularities by treating scientific inquiry as an attempt at data compression. In this framework, a model is represented as a computer program or mathematical theory that reproduces observed data sequences. The goal is to find the shortest such program—the one with minimal Kolmogorov complexity—which serves as the most efficient description of reality without discarding potentially valid but more complex hypotheses entirely. Kolmogorov complexity quantifies this simplicity by measuring the length of the shortest self-extracting archive that can reproduce a specific dataset, effectively defining its information content. While the entire universe might theoretically be described by a very short program comprising fundamental laws like quantum electrodynamics and general relativity plus initial conditions, local subsets often appear highly complex due to noise or chaotic systems. This "noise," whether arising from quantum mechanics or deterministic chaos in dice rolls, complicates scientific modeling but may also provide the illusion of free will. A striking analogy illustrates how a simple global rule can generate immense complexity locally: just as a library containing every possible book requires only a short program to define (listing all combinations), any specific subset within that library appears information-rich and complex despite originating from a concise source code. Cellular automata, such as Conway's Game of Life, serve as powerful examples where simple rules give rise to rich, emergent behaviors including Turing-complete computation and chaotic patterns. Although humans may not intuitively grasp the full behavior of these systems immediately, familiarity allows for deeper understanding through mathematical analysis, much like deriving properties of fractals or the Mandelbrot set. In principle, one could reverse-engineer such complex structures by running all possible programs in parallel until finding those that reproduce the data; however, practical limitations and resource constraints make this approach theoretical rather than feasible. Consequently, while we can prove no fast algorithm exists to detect certain deterministic sequences as random (pseudo-random numbers), our intellectual pursuit remains an ongoing effort to compress the universe's complexity into simple, understandable theories.
Read the full video transcript
what is Occam's razor so Occam's razor
says that you should not multiply
entities beyond necessity which sort of
if you translate it to proper English
means and you know in a scientific
context means that if you have two
series or hypotheses or models which
equally well describe the phenomenon
your study or the data you should choose
the more simple one so that's just the
principle you're sort of that's not like
a provable law perhaps perhaps we'll
kind of discuss it and think about it
but what's the intuition of why this
simpler answer is the one that is likely
to be more correct descriptor of
whatever we're talking about
I believe that Occam's razor is probably
the most important principle in science
I mean of course we need logical
deduction and we do experimental design
yeah but science is about finding
understanding the world finding models
of the world and we can come up with
crazy complex models which you know
explain everything but predict nothing
but the simple model seem to have
predictive power and it's a reliquary
why and the two answers to that you can
just accept it that is the principle of
science and we use this principle and it
seems to be successful we don't know why
but it just happens to be or you can try
you know find another principle which
explains or comes racer and if we start
with the assumption that the world is
governed by simple rules then there's a
bias toward simplicity and planning
Occam's razor is the mechanism to
finding these rules and actually in a
more quantitative sense and we come back
to that later and tears of Solomon
reduction you can rigorously prove that
usually assume that the world is simple
then Occam's razor is the best you can
do in a certain sense so I apologize for
the romanticized question but why do you
think outside of its effectiveness why
do we do you think we find simplicity so
appealing as human beings why does it
just why does e equals MC school
word seems so beautiful to us humans I
guess mostly in general many things can
be explained by an evolutionary argument
and you know there's some artifacts and
humans which you know are just artifacts
and not an evolutionary necessary but
with this beauty and simplicity its I
believe at least the core is about like
science finding regularities in the
world understanding the world which is
necessary for survival right you know if
I look at a bush right and I just seen
Norris and there is a tiger right and
eats me then I'm dead but if I try to
find a pattern and we know that humans
are prone to find more patterns in data
than they are you know like the you know
Mars face and all these things but these
buyers towards finding patterns even if
they are not but I mean it's best of
course if they are yeah helps us for
survival yeah that's fascinating I
haven't thought really about it I
thought I just loved science but they're
indeed from in terms of just survival
purposes there is an evolutionary
argument for why why we find the work of
Einstein is so beautiful maybe a quick
small tangent could you describe what
Solomonov induction is yeah so that's a
theory which I claim and recent long
enough sort of claimed you know a long
time ago that this solves the big loss
of the problem of induction and I
believe the claim is essentially true
and what it does is the following so
okay for the picky listener induction
can be interpreted narrowly and wildly
narrow means inferring models from data
and widely means also then using these
models for doing predictions or
prediction is also part of of the
induction so I'm little sloppy sort of
as a terminology and maybe that comes
from ray solomonoff you know being
sloppy maybe I should say that
we can't complain anymore so let me
explain a little bit this theory yeah in
simple terms so assume we have a data
sequence make it very simple the
simplest one say 1 1 1 1 1 and you see
if 100 ones yeah what do you think comes
next the natural order IRA speed up a
little bit the natural answer is of
course you know 1 ok and question is why
okay well we see a pattern there yeah
okay there's a 1 and we repeat it and
why should it suddenly after hundred
ones be different so what we're looking
for is simple explanations or models for
the data we have and now the question is
a model has to be presented in a certain
language in which language to be used in
science we want formal languages and we
can use mathematics or we can use
programs on a computer so abstract me on
a Turing machine for instance or can be
a general-purpose computer so and they
of course lots of models of you can say
maybe it's 100 ones and 100 zeros and
hundred ones that's a model right but
there are simpler models there's a model
print one loop and it also explains the
data and if you pushed it to the extreme
you are looking for the shortest program
which if you run this program reproduces
the data you have it will not stop it
will continue
naturally and this you take for your
prediction and on the sequence of ones
it's very plausible right at print one
loop it's the shortest program we can
give some more complex examples like 1 2
3 4 5 what comes next the short program
is again you know counter and so that is
roughly speaking how salamat of
induction works the extra twist is that
it can also deal with more easy data so
if you have for instance a coin flip say
a biased coin which comes up head with
60% probability then it will predict if
you learn and figure this out and after
a while it predict or the next point Li
will be head with probability 60% so
it's the stochastic version of that but
the goal is the dream is always the
search for the short program yes yeah
well in Salamone of induction precisely
what you do is so you combine so looking
for the shortest program is like
applying a purse raiser like looking for
the simple
theory there's also a Pecos principle
which says if you have multiple
hypotheses which equally well describe
your data don't discard any of them keep
all of them around you never know and
you can put it together and say ok have
a buyer's - her simplicity but I don't
rule out the larger models and
technically what we do is we weigh the
shorter models higher and the longer
models lower and you use a Bayesian
techniques you have a prior and which is
precisely 2 to the minus the complexity
of the program and you weigh all this
hypotheses and take this mixture and
then you get also based elasticity in
yeah like many of your ideas that's just
a beautiful idea of weighing based on
the simplicity of the program I love
that that that seems to me may be very
human centric concept seems to be a very
appealing way of discovering good
programs in this world you've used the
term compression quite a bit I think
it's a beautiful idea sort of we just
talked about simplicity and maybe
science or just all of our intellectual
pursuits is basically the attempt to
compress the complexity all around us
into something simple so what does this
word mean to you
compression I essentially have already
explained it so it compression means for
me finding short programs for the data
or the phenomena at hand you could
interpret it more widely as you know
finding simple theories which can be
mathematical theory so maybe even
informal you know like you know just
inverts compression means finding short
descriptions explanations programs raw
data
do you see science as a kind of our
human attempt at compression so we're
speaking more generally because when you
say programs kind of zooming in a
particular sort of almost like computer
science artificial intelligence focus
would you see all of human endeavor as a
kind of compression well at least all of
science I see as a endeavor of
compression at all of humanity maybe and
well there are also some other aspects
of
like experimental design right I mean we
we create experiments specifically to
get extra knowledge and this is there
isn't part of the decision-making
process but once we have the data to
understand the data is essentially
compression so I don't see any
difference between contrast compression
understanding and prediction so we're
jumping around topics a little bit but
returning back to simplicity a
fascinating concept of coma girl of
complexity so in your sense the most
objects in our mathematical universe
have high coma girl of complexity and
maybe what is first of all what is coma
graph complexity ok Kolmogorov
complexity is the notion of simplicity
or complexity and it takes the
compression view to the extreme so I
explained before that if you have some
data sequence just think about a file in
a computer and best sort of you know
it's just a string of bits and if you
and we have data compresses likely
compress big files into sip files with
certain compressors and you can also pre
you self-extracting are curves that
means as an executable if you run it it
reproduces the original file without
needing an extra decompressor it's just
a decompressor plus the archive together
in one and now they are better and worse
compressors and you can ask what is the
ultimate compressor so what is the
shortest possible self-extracting
archives you could produce for a certain
data set yeah which reproduces the data
set and the length of this is called the
Kolmogorov complexity and arguably that
is the information content in the data
set I mean if the data set is very
redundant or very boring you can
compress it very well so the information
content should be low and you know it is
low according to this difference this is
the length of the shortest program that
summarizes the data yes yeah and what's
your sense of our sort of universe when
we think about the different the
different objects in our universe that
we each are concepts or whatever the
at every level do they have higher or
local girl complexity so what's the hope
do we have a lot of hope and be able to
summarize much of our world that's a
tricky and difficult question so as I
said before I believe that the whole
universe based on the evidence we have
is very simple so it has a very short
description the whole sorry did you
would you linger on that the whole
universe what does I mean do you mean at
the very basic fundamental level in
order to create the universe yes yeah so
you need a very short program when you
run it to get the thing going you get
the thing going and then it will
reproduce our universe there's a problem
with noise we can come back to the later
possibly no is a problem or a fee is it
a bug or a feature I would say it makes
our life as a scientist really really
much harder I didn't think about without
noise we wouldn't need all of the
statistics but that maybe we wouldn't
feel like there's a free will
maybe we need that for the odd physics
this is an illusion that noise can give
you free reason that way it's a feature
but also if you don't have noise you
have chaotic phenomena which are
effectively like noise so we can't you
know get away with statistics even then
I mean think about rolling a dice and
you know forget about quantum mechanics
and you know exactly how you you throw
it but I mean it's still so hard to
compute a trajectory that effectively it
is best to model it you know as you know
coming out with a number of this
probability one over six but from from
this set of philosophical como go of
complexity perspective if we didn't have
noise then arguably you could describe
the whole universe as well as standard
model plus general relativity I mean we
don't have a theory of everything yet
but sort of assuming we are close to it
or have it here plus the initial
conditions which may hopefully be simple
and then you just run it and then you
would reproduce the universe but that's
followed by noise or by chaotic systems
or by initial conditions which you know
may be complex so now if we don't take
whole universe but just a subset you
know just take planet Earth planet Earth
cannot be compressed you know into a
couple of equations this is a hugely
complex just so interesting so when you
look at the window like the whole thing
might be simple when you just take a
small window then it may become complex
and that may be counterintuitive but
there's a very nice analogy the the book
the library of all books so imagine you
have a normal library with interesting
books and you go there great lots of
information and you quite complex yeah
so now I create a library which contains
all possible books say of 500 pages so
the first book just has a aaaa over all
the pages the next book aaaa and ends
with B and so on I create this library
of all books I can write a super short
program which creates this library so
this library has all books has zero
information content and you take a
subset of this library and suddenly have
a lot of information in there so that's
fascinating I think one of the most
beautiful object as magical objects that
at least today seems to be under study
or under talked about is cellular
automata what lessons do you draw from
sort of the game of life for cellular
automata where you start with the simple
rules just like you're describing with
the universe and somehow complexity
emerges do you feel like you have an
intuitive grasp on the behavior the
fascinating behavior of such systems
where some like you said some chaotic
behavior could happen some complexity
could emerge some it could die out and
some very rigid structures do you have a
sense about cellular automata that
somehow transfers maybe to the bigger
questions of our universe there's a
cellular automata especially economist
game of life is really great because
this rule are so simple you can explain
it to every child and reading by hand
you can simulate a little bit and you
see this beautiful patterns emerge and
people have proven you know that is even
Turing complete you cannot just use a
computer to simulate game of life but
you can also use game of life to
simulate any computer that is truly
amazing and it's it's the prime example
probably to demonstrate that various
the rules can lead to very rich
phenomena and people you know sometimes
you know how can how is chemistry and
biology is so rich I mean this can't be
based on simple rules yeah but no we
know quantum electrodynamics describes
all of chemistry and and become later
back to that I claim intelligence can be
explained or described in one single
equation this very rich phenomenon you
asked also about whether you know I
understand this phenomenon and it's
probably not and this is saying you
never understand really things you just
get used to them and pretty using used
to settle out your mother so you believe
that you understand now why this
phenomenon happens but I give you a
different example I didn't play too much
with this converse game of life but a
little bit more with fractals and with
the Mandelbrot set and it's beautiful
you know patterns just just look
Mandelbrot set and well when the
computers were really slow and I just a
black and white monitor and programmed
my own programs on are in assembler -
Wow Wow
to get these vectors on the screen
and it was mesmerised and much later so
I returned to this you know every couple
of years and then I try to understand
what is going on and you can understand
a little bit so I try to derive the
locations you know there are these
circles and the Apple shape and then you
have smaller Mandelbrot sets recursively
in this set and there's a way to
mathematically by solving high order
polynomials to figure out where these
centers are and what size there are
approximately and by sort of ant
mathematically approaching this problem
you slowly get a feeling of why things
are like they are and that sort of isn't
you know first step to understanding why
this rich phenomena do you think as
possible what's your intuition you think
it's possible to reverse engineer and
find this short program that generated
the these fractals sort of by what
looking
at the fractals well in principle yes
yeah so I mean in principle what you can
do is you take you know any data set you
know you take these fractals or you take
whatever your data set whatever you have
say a picture of conveys game of life
and you run through all programs you
take your programs 1 2 3 4 and all these
programs run them all in parallel in so
called dovetailing fashion give them
computational resources first one 50%
second 1/2 resources and so on and let
them run wait until they halt give an
output compare it to your data and if
some of these programs produce the
correct data then you stop and then you
have already some program it may be a
long program because it's faster and
then you continue and you get shorter
and shorter programs until you
eventually find the shortest program the
interesting thing you can never know
whether it's a short this program
because there could be an even shorter
program which is just even slower and
you just have to wait yeah
but asymptotically and actually after
finite time you have this shortest
program so this is a theoretical but
completely impractical way of finding
the underlying structure in every data
set and that of a solo mode of
interaction dollars and Kolmogorov
complexity in practice of course we have
to approach the problem more
intelligently and then if you take
resource limitations into account
there's friends the field of
pseudo-random numbers yeah and these are
random that must so these are
deterministic sequences but no algorithm
which is fast fast means runs in
polynomial time can detect that it's
actually deterministic so we can produce
interesting I mean random numbers may be
not that interesting but just an example
we can produce complex looking data and
we can then prove that no fast algorithm
can detect the underlying pattern which
is unfortunately is it that's a big
challenge for our search for simple
programs in the space of artificial
intelligence perhaps yes it definitely
is wanted vision intelligence and it's
quite surprising that it's I can't say
easy army
I worked really hard to find theories
but apparently it was possible for human
minds to find these simple rules in the
universe it could have been different
right it could have been different it's
it's a it's inspiring
you