Idiosophy

High-dimensional space is weird

On May 8, 2022

There was a mathematically unsatisfactory bit in the last post about measuring the relationships among mentions of color in The Lord of the Rings. When I used the Euclidean distance between the 62-dimensional vectors to calculate the relationship between color mentions, the dendrogram had some connections in it that don’t make much sense visually or textually, e.g., brown was clustered with black and red. The connections with a linear “Manhattan” distance measure made much more sense. I asked Digital Tolkien about it, as one does, and he assured me that the L1 metric was better. But why?

It turns out this is something that mathematicians know: in high-dimensional spaces, using the Pythagorean Theorem causes near neighbors and far-away neighbors to be all about the same distance apart! ¹ In fact, the choice of which of your neighbors is nearest isn’t even stable. The unavoidable numerical errors that come from using digital computers can dominate the real differences in the input data.²

Effect measured by L1 distance is more detectable at high dimension

Relative effect as a function of measure dimension

Of course, now that I’ve read a couple of papers about it, it’s obvious. Simplest possible case: suppose a book mentions one word once per chapter, and another word twice in one chapter and once in all the others. The relative difference between those two vectors, as a function of the number of chapters, looks like this.

62 dimensions counts as “high-dimensional”. Both ways of measuring distance have dropped a lot from our 3-dimensional experience, but the effect in our test case is twice as easy to compare when we use the Manhattan distance measure.

Notes

The Colors of the Forests

By Joe

On April 16, 2022

In Digital Humanities, fantasy

As previously discussed, black is the color mentioned most often in The Lord of the Rings, and white is right behind it. But grey is #3. Take that, Edwin Muir!

I fed the list of X11 color names into a text-processing program and collected all the color mentions I could find. With one exception: “tan” is a part of so many English words that it would be unfair to expect a computer to pick out which words containing that trigram were colors and which were not, so I deleted it from the list. This is what came out.

Figure 1. Frequency of color mentions

There are ten colors mentioned more than ten times in the text. Their relative frequency is in the pie chart in Figure 1. Oddly, none of the top-notch Tolkien illustrators has used this palette. I wonder why.

The places colors are most-often found are sometimes surprising. The chapter in which black is mentioned most is “The Siege of Gondor”. White, “The King of the Golden Hall”. Grey, “The Great River”. Red, “The Tower of Cirith Ungol”. Green and brown are mentioned in “Treebeard” more than any other chapter. Blue, yellow, and gold are mentioned most in “In the House of Tom Bombadil”; sometimes the place is not surprising at all.

Silver is most mentioned in “Lothlorien”. That chapter is #3 for “gold” instead of #1, because when a character has a color in her name, that tends to skew the distribution. Gold and silver are strongly present in all three chapters involving Lorien, though.

If we make a vector out of the fraction of each color’s mentions that happen in each chapter, we can test which colors tend to form clusters in the narrative. The dendrogram is in Figure 2. (I’ve inflicted dendrograms on you before.) As we trace a line from one color to another, the further left we have to go, the less-related the colors are in their occurrence in the text.

Figure 2. Which colors go together in the text

But what do we do with all these measurements? With an Idiosopher’s well-trained eye for the most significant thematic content of a work, I zeroed in on the disagreement between Celeborn and Treebeard. “Yet they should not go too far up that stream, nor risk becoming entangled in the Forest of Fangorn,” said Celeborn. “Do not risk getting entangled in the woods of Laurelindorinan!” said Treebeard. What’s the subject of their disagreement?

Figure 2 gives us an insight: brown is used to describe Fangorn more than any other place. Gold and silver are dominant in Lothlorien. The two forests agree on green, but to get from brown to gold and silver, we have to go all the way to the left edge of the diagram. These are the furthest-apart pair of colors in the text. So here is our answer: the source of the ancient enmity between the two forests is interior decorating. When Galadriel sang the woods of Lothlorien into existence¹, she may have had an idea of the kind of forest she didn’t want, and Fangorn may have been it.

Coda: Boring Details

Sometimes a color word is also a noun. Olive dropped out of the analysis because it’s only mentioned twice, one of each. That was an easy one. I tried to separate mentions of gold and silver into the color and the metal, but quickly discovered any partition I could make would be arbitrary. Tolkien doesn’t clearly separate them. He rarely mentions the metals without the colors being important, so I left them all in.

The method: First, all the color words were pulled from the text. Then they were classified into a standard color-word. Usually that was straightforward. The exception was “scarlet”, which got absorbed into “red”. Then each instance of a color was collected into a histogram by chapter or whatever.

Instances of a color by chapter form a vector in a 62-dimensional space. Vectors were normalized so the elements of each color’s vector were the fraction of mentions that were in that chapter. The distance between two vectors was computed using the linear distance between elements. (This is not the Euclidean distance between unit vectors; I re-did the analysis with those and got similar results, but not as easy to interpret them in a way that made sense with respect to the text. Linear differences seem more relevant to text analysis, but it’s always good to check.) The vectors were clustered using the R hclust function with complete linkage.

Environmentalism from Fiction

By Joe

On April 2, 2022

In Academy and Society, literature

The paper I didn’t present at TexMoot

The theme of this year’s TexMoot was “how fictional worlds teach us to care for this one”. For once, I was not the designated curmudgeon. That honor went to Joe Ricke, who started off the first talk by expressing uncertainty that the theme of the conference was something that even existed. He was referring to an immediate connection: that a reader would read a work of speculative fiction and come away with ideas about what to do the next day (month, year…) to save the planet. And he’s certainly right.

It’s always nice to hear about an intellectual error of which I’m not guilty. That’s not at all the way I interpreted the theme. A more likely effect that speculative fiction has on the reader is to get us used to thinking on a scale of parsecs of distance and centuries of time.

Such an attitude is in direct opposition to the demands of everyday economics. Most people are trying to make a living¹, which leads to a short-term focus. The value of gains and losses is time-dependent — money now is worth more than money some time in the future. If you don’t exploit all the things you have for profit right now, they’ll be taken over by someone who will. That impulse in the market economy caused most of the environmental destruction we’ve perpetrated in the modern era.

Speculative fiction can turn the reader’s relative valuation of possibilities away from the short-term, market-driven default. When we look at a tree, we’re not seeing just the fruit it produces, or its lumber value, but also Ents, and Yggdrasil, and all the other trees we know from literature. We see with different values. When I mentioned a half-baked version of this idea in class, Sørina expanded it with, “because we’re adding love to the calculation.” Which is an extraordinary thought. Apparently, among the powers of literature is to catalyze the reaction of love and mathematics.

As it turns out, the source of this idea is something I read long ago and forgot about. There’s a very similar thought from E.T. Jaynes, in his book ² that launched Bayesian statistics to its current prominence. Jaynes is talking about the concept from decision theory of the “loss function” — a way to quantify what we stand to gain or lose from each possible choice we can make.

Failure to judge one’s own loss function correctly is one of the major dangers that humans face. Having a little intelligence, one can invent myths out of his own imagination, and come to believe them. Worse, one person may persuade thousands of others to believe his private myths, as the sordid history of religious, political, and military disasters shows.

As the near-solecism “private myth” indicates, Jaynes is using the word to mean “falsehood”. Writing in the mid twentieth century, his thoughts were naturally going to be dominated by the negative implications of acting on fictional grounds, but “myth” has more meanings than that one. We now know that our standard way of thinking about nature is leading to disaster, so now myths can also be an opportunity to improve outcomes by changing our loss function. Expanding the elements of the computation to include all the lives involved, and the billions of years it took to bring them about, and the global (at least) results of our actions, are exactly the way speculative fiction has brought about its share of the change in attitudes to the environment that we’ve seen in the past 50 years.

Writers of both science fiction and fantasy know they’re doing this. Arthur C. Clarke said (several times), “If you take me too seriously, you’ll go broke. But if your children don’t take me seriously enough, they’ll go broke.”³ Those children are whom we call Generation X; the richest of us seem to have taken Clarke very seriously indeed.

Notes

Religious Tolerance

By Joe

On February 19, 2022

In Vive la France

It’s frustrating how poorly served Voltaire is by the World Wide Web. I needed the citation for one of his epigrams. It was actually faster to go down to the basement, get the two books it could have come from off the shelf, and leaf through them to find it.

It turns out that the reason I couldn’t remember which book it was from is that it’s in both the Lettres Philosophiques (1734) and in the Dictionnaire Philosophique (1764), prefaced in the latter by “it’s been said before, and I can’t say it better, …”

Once I had the right book, the Gallica app from the National Library of France was able to point me to the actual text.

Modern Transcription¹

S’il n’y avait en Angleterre qu’une Religion, le despotisme serait à craindre, s’il y en avait deux, elles se couperaient la gorge ; mais il y en a trente, & elles vivent en paix heureuses.

Translation

If there were only one religion in England, they should fear tyranny. If there were two, they’d slit each others’ throats. But there are thirty, and they live happily in peace.

Noted in Passing

By Joe

On February 18, 2022

In fantasy, squibs & crackers

Tom Hillman has a wonderful meditation on The Passing of Arwen Evenstar at “Alas, not me”.

One tangential thought struck me at the end. Bilbo also wrote, “I sit beside the fire and think of how the world will be / When winter comes without a spring that I shall ever see.” Which didn’t come true — Bilbo passed over the Sea in the autumn. But that couplet matches perfectly with Arwen’s death “when the mallorn-leaves were falling, but spring had not yet come”.

Arwen must have heard Bilbo sing that song. Using a good poem only once is not how Bilbo does things. Perhaps she was struck by the poignancy of that line, just as I was. Maybe Bilbo was one of the people from whom Arwen learned how to be mortal.

Old English in Modern

By Joe

On January 15, 2022

In trivial

Another line from Puck of Pook’s Hill that has no relevance to literary influence so it gets its own post:

“He sent for my second son, whom being unmarried he had ever looked upon as his own child…”

The speaker is Sir Richard Dalyngridge, a Norman knight who came over with William the Conqueror. He had become fond of the English and adopted their ways. I love that sentence for the way the reader grinds to a crawl on the eighth word. Word order won’t help us now; time for the Great English Verb Hunt we learned as the way to decipher what those old guys meant.

Kipling has done a beautiful job of recreating the feel of a modern English speaker trying to read Old English. There aren’t many inflections left these days, but the one we’ve got is effective. That sentence might easily have come from someone who’s thinking in Old English, and therefore doesn’t think of word order as an important part of grammar.

And yes, it’s intentional. Later in the book, we’re told a character is educated because he knows “the Leech-Book of Bald”.

Did Tolkien Kipple?

By Joe

On January 15, 2022

In fantasy

I have just finished reading Puck of Pook’s Hill (1906) by Rudyard Kipling, in the Project Gutenberg edition.¹ It’s a book for children. J.R.R. Tolkien was 14 when it was published, so he was a bit old for it, but I get a distinct feeling that either he read it or he was told about it. Holly Ordway‘s new book, Tolkien’s Modern Reading, has a table of works that she knows for certain Tolkien read. It’s very thorough, and Puck isn’t among them. That may be just for lack of evidence — the book wasn’t obscure ² and there’s no specific reason it would ever have come up in a surviving written source. Nonetheless, I heard echoes of Tolkien’s stories all through Kipling’s book.

Tom Bombadil

The eponymous character Puck is the first and last fairy in England. He was there first: “I’m Puck, the oldest Old Thing in England,” he introduces himself (p. 8), which I have to put next to “Eldest, that’s what I am!” from Bombadil. Puck is the last because he is a nature spirit, with a source of strength that could resist drab Protestant conformity.

The People of the Hills have all left. I saw them come into Old England and I saw them go. Giants, trolls, kelpies, brownies, goblins, imps; wood, tree, mound, and water spirits; heath-people, hill-watchers, treasure-guards, good people, little people, pishogues, leprechauns, night-riders, pixies, nixies, gnomes and the rest—gone, all gone! I came into England with Oak, Ash, and Thorn, and when Oak, Ash, and Thorn are gone I shall go too. (p. 10)

The way Bombadil put it sounds almost the same:

Tom remembers the first raindrop and the first acorn. He made paths before the Big People, and saw the little People arriving. He was here before the Kings and the graves and the Barrow-wights. When the Elves passed westward, Tom was here already, before the seas were bent. (LotR, I, vii)

The common reference to oaks and acorns is also good for us dendrophiles. Tolkien’s Middle-earth is connected to the prehistory of our world, which we can see here because Puck tells us the water-spirits are all gone, whereas Bombadil knows exactly where he can find one.

Also, not such a strong connection, but the story “Dymchurch Flit” is narrated by Puck himself, in disguise under the name of Tom.

Pictish Hobbits

Two chapters of the book are stories from the Roman Empire, narrated by a centurion on Hadrian’s Wall. North of the Wall were the Picts. Our narrator Parnesius befriended a Pict named Allo. There’s a Pictish Song before the story begins, which contains the lines,

We are the Little Folk—we!
Too little to love or to hate (p. 201)

This has countless echoes in The Lord of the Rings; the one that came to my mind was,

‘… one poor hobbit coming in from the battle is easily overlooked.’
‘It’s not always a misfortune being overlooked’, said Merry. ‘I was
overlooked just now by…’ (LotR V, viii)

The part that inspired this whole post is when Allo takes Parnesius and his comrade Pertinax hunting north of the Wall:

You are quite safe so long as you are his guest, and wear a sprig of heather where it can be seen. If you went alone you would surely be killed, if you were not smothered first in the bogs. Only the Picts know their way about those black and hidden bogs. (p. 158)

I couldn’t help thinking of Gollum leading Frodo and Sam through the Dead Marshes.

Jewish Dwarves or Dwarvish Jews

A line I always wondered about in LotR: “Gimli was fingering gold in his mind, and wondering if it were fit to be wrought into the housing of the Lady’s gift.” (II, ix) Gold is an elemental metal. All gold should be the same. Maybe Dwarves have a different sense, though, like a master vintner who can tell apart grapes from two adjacent vines, just by taste. Kipling’s character can do that. Kadmiel is a Jewish moneylender whose choices forced King John to sign Magna Carta. Kadmiel tells us,

I know the Golds. I can judge them in the dark; but this was heavier and redder than any we deal in. Perhaps it was the very gold of Parvaim. (p. 271)

(Actually, it was gold from West Africa.) This parallel doesn’t appear in any of the accounts I’ve read that explore how Tolkien’s Dwarves are influenced by stories of Jews in Europe.

The Departure of the Elves

The departure of the fairies, by Rackham

Tolkien’s Elves are leaving Middle-earth because their time is over. They’re boarding ships and sailing into the West. Kipling’s fairies feel the same way: “Fair or foul, we must flit out o’ this, for Merry England’s done with, an’ we’re reckoned among the Images.” (p. 242). They’re referring to the Protestant Reformation. But these are tiny, Tinkerbellish fairies. How can they get out?

A boat to be sure. Their liddle wings could no more cross Channel than so many tired butterflies. A boat an’ a crew they desired to sail ’em over to France, where yet awhile folks hadn’t tore down the Images. (p.242)

Puck has always maintained relations with us humans, so he could find them a couple of unlikely lads and a boat just barely large enough. Of course, these elves sailed into the South. There were still Catholics in France, and maybe some remnant of the Forest of Brocéliande had escaped the loggers’ axes.

Conclusion

The Hobbit and The Lord of the Rings are many things, but first they are adventure stories. Kipling was a master of adventure stories from the previous generation. When Tolkien sat down to write long-form fiction, common tropes of the genre were well established and ready to hand. This is a partial list of the ways Tolkien could play off of Kipling’s tropes, whether they were things he liked, such as nature spirits who are invulnerable to the changes in human society, or things he didn’t, such as Pigwidgeon fairies.

Any reader who does not like this essay can procure one leaf each from an oak, an ash, and a hawthorn, and forget the whole thing instantly.

Coda: Intertextuality goes both ways

Totally irrelevant to the rest of this post, but Kipling wasn’t above recycling a good thought, either:

“All good families are very much the same” (p. 131) vs. “Happy families are all alike…” -Tolstoy, Anna Karenina (1878).

Gimli’s Opium Dream

By Joe

On January 8, 2022

In Digital Humanities, fantasy, mythgard academy

I’m currently taking the mini-course “Tolkien’s Ents and the Environment” from Signum University’s SPACE program, taught by the unwiðmetenlic Sørina Higgins. We were discussing Gimli’s speech about the glories of the Glittering Caves (III, viii) and how it parallels the (more frequent) references to trees and plants as the object of environmentalist sympathies. Sørina challenged us to a close reading of Gimli’s speech.

by Massupa Kaewgahya

Surprising no one, I zoomed in on the extraordinary number of French-derived words in the passage. I’ve never counted them before. Time to fire up the OED Text Annotator! This analysis focused on Gimli’s direct speech, from “Strange are the ways of men…” to “It makes me weep to leave them.” This passage is 14% derived from French. As we have established, the threshold of madness in Tolkien is 7%. In this passage, Gimli leaves behind even the suicidal Denethor. It’s the second-highest French density I’ve identified so far, just behind Gollum’s pre-taming peak of 15%.

Then Sørina pointed out something fascinating: Gimli’s speech sounds a lot like Coleridge’s poem “Kubla Khan.” The speech and the poem are roughly the same length. (381 words to 349.) They share words like measureless caverns, underground rivers, domes, caves, towers, round, cover, hill, sea, music, deep, wall, war… (omitting the boring, common words). Of course the quantitative metrics kind of miss the point. The feeling is similar. Both are gushing over a beautiful place from which they’ve been untimely taken away.¹ As Sørina put it, the caverns cause Gimli not just to switch languages, but also centuries.

“Kubla Khan” is famously the result of an opium dream. There’s only one conclusion to draw here. There’s some kind of narcotic in the Glittering Caves of Aglarond. Sauron missed a trick when he tried to snare Dwarves with Rings of Power. That kind of addiction² doesn’t work on them. However, limestone caves apparently emit a gas that humans don’t notice, which acts like a drug on Dwarves. Even short-term exposure leads to monologuing, Romanticism, and French.

Hyphens and Colors

By Joe

On January 1, 2022

In fantasy

Sparrow Alden has published a paper on hyphens in the latest issue of Mallorn, which rejoices in the erudite title “Hyphens as Sub-Lexical Morphemes in The Hobbit“.¹ If you’re not a member of the Tolkien Society, you’ll be able to read it for free in a few years, or maybe you can reconstruct it from the pieces she’s already published on her blog.

It’s not easy to make the jump from a quantitative analysis full of numbers and graphs to the level of discourse that humanists expect. Sparrow makes it, and sticks the landing. Her idea is that Tolkien, as translator of a text originally in Westron, needed to come up with English equivalents to highly specific words in hobbit-speech for which we’ve never needed an equivalent. (e.g., “hobbit-speech”) The pattern, therefore, is that hyphenated words are focused in parts of the text that deal with things known well to Bilbo, but not to us. Where things are commonly familiar or commonly strange, the hyphens aren’t necessary to the translation. Very nice!

Let’s see where that idea can take us. I was looking for color-names in The Lord of the Rings the other day. It has ten cases of colors with hyphens in them.

Color	Mentions
grey-green	7
silver-grey	3
golden-red	2
blue-grey	1
brown-green	1
black-grey	1
green-white	1
green-yellow	1
red-golden	1
silver-green	1

“Grey-green” is used for everything from fields of grass to Ents. “Black-grey” is tree bark in Fangorn; “brown-green” is oak trees just about to bud. “Golden-red” is a rowan-tree or a fire. If Sparrow’s idea is correct, the vegetation in Middle-earth is of a slightly different color from anything we’re familiar with, and maybe burns differently. “Silver-grey” is purely Elvish; it’s perfectly reasonable to suppose Elves make things in a color we don’t have. “Silver-green” is Goldberry’s dress; ditto. The two “green-” compounds are Gollum and Shelob; I’m glad to be unfamiliar with them. “Red-golden” is Gandalf’s fireworks, which I regret never seeing. “Blue-grey” is smoke. That’s the only weak spot; smoke ought to be familiar.

All together, the idea that hyphenated words are translations from things we don’t exactly have holds up well.

Pensée d’escalier

The word “orange” does not appear in LotR. “Red-golden” and “golden-red” must be Tolkien’s attempt to come up with an Old-English equivalent of the color. (I don’t know why — according to the OED, its use as a color name was used as early as 1557 so it sneaks in before the deadline.) This is a case where we know the concept perfectly, but Bilbo and Frodo didn’t.

He got better

By Joe

On December 4, 2021

In Digital Humanities

A Mr. C. Hostetter (!) who is highly knowledgeable about these matters, points out an error in the post I derived from the Times Literary Supplement. Tolkien had nothing to do with Early English Lyrics: Amourous, Divine, Moral, and Trivial. I’ve fixed the old post.

Digitizing old magazines is a tricky business. In this case, the review was of two different books. The scanner put the two titles together up front and the authors’ names together at the end. This happens often enough in that database that I’ve been unable to do a graph analysis like I did with the London Review of Books, by the way.

While I was fixing that, I chased down all the other pre-Hobbit references. That brought me to a later review¹ of what seems to be the same work. This reviewer was in a more eupeptic mood, I suppose, because “above average” has been superseded.

Mr Tolkien’s vocabulary affords an excellent handmaiden, not standing in the way, while decidedly more than an appendix. The scholar will enjoy the explication of, inter plurima, ‘knacke,’ ‘lay,’ ‘sentence.’

Now I find myself wanting to track down those explications.