The Incredible Story Of Randomness

In this comprehensive exploration of randomness, we delve into its perplexing nature, historical journey, statistical interpretations, and pivotal role in various domains, particularly cryptography. Randomness, an enigmatic concept defying intuition, manifests through seemingly unpredictable sequences like coin flips or digits of pi, yet its true nature is only indirectly inferred through statistical tests.

The historical narrative reveals humanity's earliest encounters with randomness in gaming across ancient civilizations, progressing through Greek philosophy, Roman personification, Christian teachings, and mathematical analysis by Italian scholars and luminaries like Galileo, Pascal, and Fermat. Entropy, introduced in the 19th century, unveiled the limits of predictability, especially in complex systems like celestial mechanics.

Statistical randomness, derived from probability theory, relies on uniform distribution and independence of events in a sample space. However, its limitation lies in perceivable unpredictability, as exemplified by the digits of pi or coin flips, which exhibit statistical randomness yet remain reproducible given precise initial conditions.

Information theory, notably Claude Shannon's work, established entropy as a measure of uncertainty and information content, showcasing randomness as the opposite of predictability in a system. Algorithmic randomness, introduced by von Mises and refined by Kolmogorov, measures randomness through compressibility but faces challenges due to computability. Martin-Löf's work extends this notion by defining randomness based on null sets.

The integration of randomness into computer science led to the emergence of randomized algorithms, divided into Las Vegas and Monte Carlo categories, offering computational advantages. Encryption, crucial in modern communications, relies on randomness for secure key generation, facing challenges due to vulnerabilities in pseudorandom algorithms and hardware random number generators.

The evolution of cryptography, from DES to AES and asymmetric-key algorithms like RSA, emphasizes the critical role of randomness in securing digital communications. While hardware random number generators harness inherent physical unpredictability, they face challenges regarding auditability and potential vulnerabilities.

The future of randomness lies in embedded quantum random number generators, promising heightened security, while encryption algorithms adapt to counter emerging threats posed by quantum computing's properties.

This in-depth exploration captures the historical, theoretical, and practical dimensions of randomness, highlighting its significance in diverse fields and its pivotal role in securing modern communications.

SUPPORT NEW MIND ON PATREON

[scottterry]

(standing ovation) ... GREAT JOB!

I've spent over 40 years developing models and doing statistical analytics. During that time, two concepts have always fascinated me ... 1) the notion of "randomness" and 2) "nothing". After all this time and in the big picture, I'm convinced that randomness does not and never will exist because of cause and effect. Even entropy is predictable. Isn't that strange?

P.S. If I were still teaching, I'd require all my students to watch this video.

[Beb]

0:45 Mind already blown : "Randomness is not a measurable property, but a generative process"
We can't infer randomness once the generation has happened.

[misterdeedeedee]

5:55 i have never in my life seen a person flip a coin like this

[Bob-kein]

There is so much information in this video I'm going to save it and go over it again. My "back of the envelope" definition of randomness is a sequence or pattern for which we cannot currently develop a process to duplicate it. Implied is the idea that future knowledge may indeed be able to develop a predictive process so patterns considered random today may not be so in the future. But given the inherent uncertainty of quantum mechanics, at the bottom, there are patterns that can never be reconstructed and thus can be considered truly random. Youtube constantly amazes me with the vast depth of its content.

[kieranhosty]

the expertise that goes into these videos continues to impress me. I am only halfway into my second year of software dev, just getting my first taste of RSA encryption and basic statistical techniques to surpass encryption on text messages.

It is super interesting to see how much thought goes into making something unpredictable. Thank you for all these videos!

[dancoroian]

6:40 were these faces animated with AI? (I don't know what I'm asking, of course HD B&W portraits of dudes from the first half of the 19th century don't just exist out there...🤦🏻‍♂️)
Anyway, they looked really weird. I really hope this doesn't take off, just be glad for the quality photos and show them as they are, please!

[klikkolee]

It's important to note that the primary function of a hardware random number generator is to generate the seed of a cryptographic PRNG. It's not normal to wait for a hardware rng for every random number that is needed.

[JoseLopez-hpoo]

The casino gaming industry uses chi-squared distribution as a means test for the RNG. Also secure random numbers are a thing in modern SoC such as Snapdragon. The SoC or CPU as most call it contains a hardware root of trust (HROT) that performs a secure random as well as some basic secure transforms. This is completely independent of the CPU or GPU accelerators and the memory they use. The Trusted Operating System (TEE-OS) takes ownership of the TPU once the system starts up. Also another feature is PUF memory used for making unique and immutable ID for each SoC used for encryption processing.

[williamstearns]

Awesome video as always! But having Rudolph Clausius’ eyes move to stare at me was unexpected and kind of freaked me out. 😂

[snigdharahman]

The title got me here.

Edit: I got here too early and commented before completing the video. After watching the video, I can say the title is spot on.

Using quantum probability of photon arriving is hardcore. We came to this from gaming mind you. That really escalated.

Great great video. This video touched on security, probability using a fundamental concept. Amazing man

[Anirossa]

Looking forward to my class on statistics next semester, will be a nice change after being done with my calculus classes, always been extremely intrigued by how you can take into account factors to make more accurate assumptions that will statiscally speaking be more often current, than wrong, with a large enough sampe space.
Thanks for a great little teaser.

[rapauli]

Jeekers this is exhausting. Now you are changed. "Information causes change. If it doesn't, then it's not information." -- Claude Shannon

[Litwinel]

a day with a new video from New Mind is a good day

[cykkm]

‧ 0:23: “normal numbers” – how many of these do you know? I aware of just two, the Champernowne and Erdős constants. There's a proof that _almost all_ reals are normal, but it doesn't let you turn the thing around on its head and say “π is normal because probability of otherwise is negligible”)
‧ 5:29: From i.i.d., a random sequence inherits unpredictability — sorry, you've lost me here.
‧ 6:16: “Objective randomness” is completely determined by _your own_ operational definition of “objective.“
‧ 7:44: **Super, thank you!** Randomness, however it be defined, is immeasurable. Could you please clout that into the heads of the probability class students?
‧ 7:48: The split is a non-split. I believe the lecture would benefit from even a short proof of equivalence of the two, superficially incompatible views on probability. There is no “maths“ or “phys” probability. There's only probability.
‧ 8:20: This is somewhat unconventional. We don't usually associate an _interpretation_ of probability with the _domain of knowledge._
‧ 9:03: As a physicist for 40+ years, I don't believe in a (the?) “fundamental nature of reality” (FNR). The FNR experiences significant changes every 30 years or so. (My undergrad textbooks are good as the fireplace kindle tho.) Either the reality is prone to such quick changes, or... You see the direction I'm pointing to. It's we, not the _it._
‧ 9:10: No, no, no, no, no! No mathematical concept is the fundamental part of “existence” — I read “existence=“the real world.” By this bold exclamation, I invite all Platonist mathematicians to Platonise me, slowly and utterly unpleasantly, up to the fundamental nonexistence, or, worse, existence in the Platonic domain, with no space or time, but only truths. Who wants truths for the taking, not for a lifetime struggle with the having of discovering them?
‧ 12:25: Nothing “fatal“
in the K. complexity. It'd just pure maths for ya mates!
‧ Null covers. It's important, as the set of null covers of an infinite non-enumerable set is finite.
‧ 14:22 What happened to the Shor algorithm, the third of the two?
‧ 13:49 Digital Oopsie!
‧ 16:46: There is NO difference between pseudo-randomness and chaos.
‧ 17:40: The transistor “thermal” noise is an inherently quantum process. Treating it separately from the following quantum optics sources of randomness is unfounded in the current physics.

Here, I fixed thait fer ya! 💖 Nice vid, indeed; I watched it with an ultimate pleasure. Don't take my comments to seriously... But, uhm… .On the second though, I take that back. do. It won't harm you, even if I'm wrong on all points. Peace, mate, no intention to object, just that to complement your presentation!

[ronking]

Great video, and I'm glad to see that Shannon was given proper due. I would have liked to seen his ideas further fleshed out in the ideas of Conway's Game of Life, cellular automata, and more so into computational irreducibility as presented by Wolfram. To me, this is the most accurate way to view informational randomness. It allows for finite states that lead to unpredictable outcomes, due to nothing more than iteration and an inability to compute it, intractability. Still, great video!

[thedolphin]

0:09 Randomness *does not defy intuition* . It defies logic, rationality, presumptions, expectations. Intuitively, I completely get randomness, luck, the unexpected, disorder, irregularity, entropy -- simply because THEY are the basis of nature. I do not want or expect them to be the norm.

[circusitch]

Every time I flip a coin, it rolls underneath furniture.

[Lyra]

I stopped playing online Scrabble games when, as a long time Scrabble player, playing against a good friend, I could soon tell that the computer generated racks produced letter combinations that real life (picking out of a shaken bag) racks rarely did.

And, in fact, my friend and I then engaged in a long conversation about the nature of randomness and whether or not computers could ever generate random patterns without reference to some external non-linear function. I was convinced this was not possible. Not sure if this video suggests I'm wrong, because I'm no mathematician.

[bodhisoma]

The game shown at 1:30 is Go, which, like chess or checkers, has no random elements.

[sassulusmagnus]

Great topic. Whether randomness is an objective property of the world or whether it is the subjective experience of one's limited ability to discern objective order is a fascinating question. How can we tell the difference? How do we find whatever boundary there might be between the two?