This is a Slack conversation between my partner and I regarding our views on language, the human condition and AI:
Me:
11:52 AM
The real ‘learning’ here is that what humans – from Shakespeare to Eminem to me – have done with language is really not all that difficult or unique given a relative static set of words. There’s only so many ways to mix and match them to create meaning. It’s not until we get to the “Arrival”-type language where we’re doing something truly unique with language (Great book – ok movie). Playing the game “Connections” there are some words with 10 or more meanings – words that are nouns and verbs and adjectives each with various meanings – yesterday was “BUFF”, to shine, strong, fan of, etc. – and then there are words with very, very specific meanings – “LEDE” as in to bury the lede. Only used in that phrase. Not crazy to think that you could map all of that and then use it correctly.
Partner:
11:56 AM
Yea on your comment
To me, the fundamental delta with humans is that upon reading/hearing/speaking the language humans have an associated subjective experience. These LLM are not. (edited)
Shakespare and Eminen invoke a flavor of experience.
Me:
12:00 PM
Fascinating. Right – like when you’re listening to a story you are thinking about responding based on YOUR subjective experience. (I know you’re not supposed to do that – you are supposed to ask follow-up questions, but it’s pretty human – they are talking about their trip to Rome and you want to throw in that you fell into the Trevi fountain, etc.)
Partner:
12:01 PM
Yup… LLMs don’t have that all… all they are doing is basically regurgitating stuff people have written.
Then you get to talking about the fact that the LLM are going to be ingesting their owns stuff… and what will the result of that be?
But LLMs do a really good job a regurgitation.
Me:
12:03 PM
Now THAT seems like taking the average of an average. That seems bad.
Already LLMs output is pretty ‘soul-less’.
Partner:
12:07 PM
The stochastic element is really important… and seems to me to be a critical component of what makes LLMs work well… the LLM output for the next word is actually multiple words with probabilities attached to each word. If you set the temperature to 0, you ALWAYS get the highest probability word. If you turn it up, you basically rolling the dice to pick the next word.
Me:
12:08 PM
I’ve played with that. 0 is SUPER boring. Like almost toddler boring.
Partner:
12:09 PM
But basically picking the next word from a set of choices is such a simple calculation, but the “magic” seems to come from a large extent from rolling the dice.
Me:
12:09 PM
true
Image Source: The Met Collection.
Title: The Creation of the World and the Expulsion from Paradise
Artist: Giovanni di Paolo (Giovanni di Paolo di Grazia) (Italian, Siena 1398–1482 Siena)
Date: 1445
Medium: Tempera and gold on wood