Archive | October, 2018

Measure your words with text mining Python scripts

Python text mining scripts, count your words

Ursula K. Le Guin said she wrote 40 stories before the first of her stories was published. This is a collection of her first published stories.

Text mining scripts in Python

I wrote a set of Python scripts to run numerical analysis on my writing. You can gain valuable insight into your writing by measuring your words. The measurement of your writing can be part of your writing practice, and sets the basis for creating tests when validating hypothesis about your writing.

You can find the text mining scripts in my GitHub repository. The repo contains the following scripts:

  • Concordance (word count)
  • N-Gram (word pair count)
  • Entity extraction (keyword count)
  • Search engine keyword analysis (scores keywords)
  • Parts-of-Speech (part-of-speech frequency)
  • Reading level (reading grade level)

These scripts were partly inspired by a job I had as a social media analyst in 2008. And the desire to create this set of scripts is one of the reasons I started learning Python. Social media analysis mixes two different disciplines: social network analysis and text mining. Social network analysis is the kind of thing that epidemiologists use to find patient zero. For my job, I was using network maps to track who was talking about products and who was listening to them talk about products. For example, I looked at who was talking about salsa. To quickly assess what they were talking about I used some crude forms of text mining. My methods at the time typically involved Excel.

These scripts use the Natural Language Toolkit (NLTK) which has been around for a while.

GitHub Repo – Measure words

Count words

I have always counted the words in my writing, or in a handwritten journal, the time.

When I was sixteen years old and facing what seemed an essential choice of what I wanted to do with my life, this doing seemed more than a way of making money, but a choice that had to be a vocation. In my sophomore and junior year in high school, the school gave us a military skills test and career tests. I think both tests determined I should be a clerk.

I had planned on going to school to become an electrical engineer. In my family at the time the only white color jobs anyone had ever had been as engineers. I went to school were some of the parents were engineers for Boeing and some of the parents worked in the factories assembling airplanes. My grandfather had worked in a factory working on airplanes. My other grandfather worked on nuclear submarines. My father and uncles, however, had driven cabs and worked in restaurants. They weren’t chefs, but worked in the kitchen. And the thing about restaurant work is that no one thought of themselves a chef who worked in a Seattle dinner preparing Crab Louie. They thought of themselves as people who lived their lives and paid the bills by working in a Seattle diner preparing Crab Louies.

The engineers, however, identified themselves as engineers as if being an engineer was an existential condition. In the spring of sophomore year contemplating a profession of being an engineer where I would be an engineer rather than a guy who works on engineering to pay the bills, I decided if that as the deal I would rather be a writer.

I had no idea what this meant. Since engineers could earn a living in the state of being an engineer couldn’t writers get buy by being in a state of being a writer?

I didn’t really realize at the time that being a writer was more like being a clerk, and I was just affirming the effectiveness of the vocational tests.

I had a vague notion that a writer was someone who wrote and somehow things like housing and food were not that big of a deal. They just came with the gig. I may have been basing this on Jack Torrance from The Shining or Garp from The World According to Garp. I had three ideas that ended up being helpful to me.

One idea was that I had to write every day and that I was beginning from nothing and would have to learn by writing if I wanted to be a writer.

The second idea was that I had to finish stories. Garp wrote a story a month while in high school. And I had read in the introduction to a collection of Ursula K. Le Guin’s firs stories, The Winds Twelve Quarters that she had written stories as a regular practice and sent them out. She wrote 40 stories before she was published.

The third idea was that I had to send my finished stories to magazines to get published. This meant I had found out where these things were and who to send them to.

Every night beginning that spring in 1987 I sat down to write. I thought about my work like it was a homework assignment, and so learned I could write 500 words with some degree of concentration. Even 500 words meant hat in a week I had a number of words that indicated a length of a story.

In a year I had finished 10 stories. In two years I had finished about 20 stories and written what I thought was a novel which was about twenty thousand words. One of the surprising aspects of this habit was that it didn’t require that much time. I could write 500 words in less than an hour. After I learned to type, I could write that in less than half an hour.

I kept writing when I enlisted in the Army Reserve and went through Boot Camp and skills training at Fort Sam Houston. I didn’t have time to write in Boot Camp, but at Fort Sam, the base library had typewriters I could use and I bought onion skin typing paper and typed on the IBM Selectric typewriter they had, 500 words or more. And then learned that onion skin is not good to type on because the ink flakes off. So I retyped my stories.

Word count was a familiar metrics to me. It was like miles are to a long distance runner, or laps to a swimmer.

I thought of the count of finished stories as proof that I was progressing toward being a writer thinking there would be a state change at some point. I would be published, and thereafter be a writer.

I also sent stories out. I found The Writer’s Digest Writer’s Market. I learned to send a story with a return postage, and began to collect rejection slips. I expected rejection slips at first and then became used to it. Early on in 1988 I got a shock when I sent a story to a local magazine edited by writer I had read about in The Seattle PI, Jessica Amanda Salmonson. She sent me back a letter and said something to the effect that based on the strength of the title of my story she had read my entire manuscript. And this had been her mistake. She suggested I get serious psychological help as quickly as possible not only for my own safety by the safety of everyone around me. My story had been called, “Leave Shatter Like Skulls.” I was reading a lot of L. Sprague de Camp, Robert Howard, Micheal Moorcock, and HP Lovecraft at the time. I was thrilled that my story had been read and that it has struck a nerve.

Based on this letter that saw my writing not as writing but as the symptom of a deranged mind I kept at for years. As a college freshman in 1992 I won a prize but not publication in STORY magazine in a year that saw a writer named Benjamin Anastas from the University of Iowa MFA program winning the first prize. The next year I published a story the Bellevue College magazine Arnazella.

It was working in that it was less of a state change from not a writer to being a writer. Being a writer was more like being a runner. While actively running, I am a runner. While putting in a regular word count, I am writer. Jack Torrance is probably a good model. I think most people may think of Torrance as a proxy for Stephen King. And in terms of making a living as a writer, who wouldn’t want to be Stephen King? But in fact it is much more like Torrance putting in words in the lobby of the Overlook Hotel, just be nice to your family and don’t go to Room 237.

Comments { 0 }

Down Comes the Other Leg


The Bogeyman, Slender Man, or Struwwelpeter

My seventeen year old daughter Riley said something horrifying to her three year old cousin, Ila. Riley said that if Ila wandered away from where they were playing outside, a man who would find her and user bones as blocks.

“You said that?” my wife asked. “Why would you say that?”

Riley was confused that we were horrified. “Did anyone hear you say that?” my wife asked.

It struck us as both a weird thing to say to a child, and yet it was the kind of thing that people in my family said to their children. It was the kind of thing adults have always said to children. These things had been said to me. And in hearing her say it, I wondered if I had said similar things to my daughter? Why wouldn’t I have? Riley seemed to think this is what you do with children: Scare the hell out of them. You tell them about the Bogeyman, Slender Man, or Struwwelpeter. You engage in ambient child abuse through mythology.

Riley seemed confused. She said she had been told as a child if she didn’t behave a man would come and put her into a hole in his back. She said this in a matter of fact way. She had been told this, and we tell kids that monstrous things will happen to them at the hands of supernatural forces because that is what happens when you misbehave.

Yet what Riley had said about the man using Ila’s bones as blocks was clearly made up on the spot. Why would she make something like this up on the spot? Who told tell her a story about a man with a hole in his back that would imprison children in his body cavity if they did bad things?

Riley said that was us. It was some adult. We had told her this. My wife said,  “I didn’t say anything like this.”

In the moment of round robin denial, I said, “I didn’t say something like that. It’s horrifying.” But later I thought about it. Had I? I mean, I could have. Before I had, had a child I had a very fuzzy idea of what it meant to be a child. My own parents felt that childhood was a dangerous myth that put young people in danger. In their world, a child was just an adult who as three years old. Growing up, I had never seen the kind of nurturing, protective, (maybe too-protective) attitude that parents had toward child and even the concept of childhood. The idea that something like Free Range Children needed to exist was absurd where I grew up. We were told to get out from under foot. Get out of the house and play. If we stuck around the house, we ended up doing chores. I was expected to turn up at dusk.

When my daughter was born, my attitude about childhood and adulthood changed because I was forced to examine what my attitudes even were. Part of this was due to the parenting classes offered through my Health Co Op that taught a form of parenting called attachment parenting, an approach used by Dr. Sears. Sears promoted attentive engagement with your child and close physical contact. Anticipate your child’s needs and provide a nurturing ting safe environment for the child to gain the confidence their needs would be met, and they were fundamentally safe. His approach seemed very logical. My parents practiced, I guess, a kind of proximity parenting, and a discipline inspired as much by Max Ernst as John B. Watson’s Behaviorism. Even though they were self-professed hippies, they weren’t into Dr. Spock. They were in the barn. I was in the barn. There was food in the pantry and the orchard. I could eat if I was hungry. I was raised in the barn.

As soon as my daughter was born, this changed. She had not existed. Then she was a kind of diaphanous figure in the amniotic void of an ultrasound in the physical reality of my wife’s womb. It took her nine months to grow a person inside of her body. The unreality of pregnancy turned very real. When our daughter came out into the world, she was coated in fluid, her eyes were closed, and she needed everything short of her own capacity to pump blood through her heart and air through her lungs. Anything short of nurture from that point forward seemed cruel. I was quick to yield completely to the myths of childhood, to the social contract, to the demands of Dr. Sears, to the responsibilities of adulthood and at times the crushing conformity that is required by a parent who needs to provide medical care, nutritious food, opportunity to their child. Nine months was just the warm up to the decades required.

Why then tell a small person who is still sorting out that airplanes fly, clouds are fluffy, and so that there are also men who kill you and use your bones for play things? Why tell them men would place your entire body into their bodily cavities? Who had told my daughter this disgusting and horrifying lie? This freaky fairy tale?

It was likely a someone from my side of the family. And among my side of the family it was likely a Briggs. My mother’s side of the family made up psychological torments, but only in the lines of what was materially possible. My father, however, would tell an old story he had heard on a Alfred Hitchcock radio show as a child about a fireplace in a cabin in the woods. A travel comes into the cabin and builds a fire. As he is warming myself by then fire, a voice calls out, “down comes the leg,” and then there was a leg dancing in the fire. And then a voice calls out, “downs come the other leg!” This became a catch phrase when I was growing up, “Down comes the other leg!” This would be the alarm when something surreal in real life was happening, which in my father’s world was often the case. During the cluster of the story the entire body comes down with the climax happening when “Down comes the head!”

There was in my father’s family a number of malevolent figures that my dad and uncle would talk about. That is the underpinnings of these figures were grossly allegorical and the crude allegory made them approachable as fantasy figures. We could just as easily make up own figures. (And we did later with figures like the Main in Plaid). Yet a the time I first heard of the Pernicorn or even the Sasquatch they seemed as real as any other thing adults told me.


A figure called Ngurp, that is is similiar to the Pernicorn, in my uncle’s journal.

For a six year old, they were no less real than other figures, such as Santa Claus, The Easter Bunny, Jack Frost, bobcats, bears, and other spirits and animals in the forest. These figures for as long as I could remember as I could remember where balanced between myth and reality. The question of their existence was part of their existence. They were at once musical and cynical, at once a product of the imagination and a product of critical inquiry.

These figures seemed at once made up and also like they have come form something or meant more than just pure whimsy. Whimsy in my father’s family was used as a kind of weapon. This is one of the comforts of Trout Mask Replica to me. Captain Beefheart’s masterpiece is partly a masterpiece because it embodies the drunk uncle on acid and PCP aesthetic that my father and uncle delighted in when I was a child in Snoqualmie in the 1970s. (This is aesthetic is not isolated to the Snoqualime in the 1970s.) There is in that album a sequence where Beefheart and his cohorts record their interactions with some relatively straight neighbors during the cult like period of rehearsing the album. “It’s a bush recording. We’re out recording the bush.”

Hair Pie: Bake 1

Girl: We just moved in around here. We heard you play, so we decided we’d come up and find out who it was

Captain Beefheart: O, ho, yes. Captain Beefheart and his Magic Band.

Boy: Really?

Captain Beefheart: Yeah

Captain Beefheart: Where did you move here from?

Girl: Oh, just from–

Boy: Reseda

Girl: Yeah

Captain Beefheart: Reseda?

Girl: Yeah

Captain Beefheart: She’s nice…

Captain Beefheart: What do you think?

Visitors together: Sounds good.

Captain Beefheart: It’s a bush recording. We’re out recording the bush. Name of the composition is “Mea… Neon Meat Dream of an Octafish.”

Boy: Hum

Girl: Um

Captain Beefheart: No! It’s: ‘Hair Pie’

Girl: Looks you don’t find the drummer now.

Boy: Huh!

For some reason we make up bogeyman for our children. I don’t know if is the children that need them however.

Comments { 0 }