Microsoft Word - ds2500_lab_nlp.docx
DS 2501: Intermediate Programming with Data / Lab Practicum
Prof. Rachlin, Northeastern University
Hambot
In natural language processing, n-grams refer to all of the n-length contiguous words occu
ing
over the text. For example, if our input is: “She sells sea shells by the sea shore” the 2-grams
would be (ignoring case): she sells, sells sea, sea shells, shells by, … etc. and our 3-grams would
e: she sells sea, sells sea shells, sea shells by, shells by the, … etc.
We’re going to try to use 2-grams and 3-grams to create a lyrics-generating bot for the musical
Hamilton.
Here’s the general approach:
1. Read the file containing Hamilton lyrics. Remove punctuation, headers that indicate the
speaker such as [HAMILTON] or [HAMILTON & LAURENS] and convert all words to
lowercase.
2. Catalog all the words that begin a sentence.
3. Catalog all the words that terminate a sentence
4. Catalog all the 2-grams as a dictionary where the key is the first word, and the value is a
list of all the possible follow-on words. For example, in the tongue twister provided
above we might have:
start_words: [she]
stop_words: [shore]
2-grams: {she: [sells], sells: [sea], sea:[shells, shore], … }.
5. Now generate random Hamilton lyrics by doing the following:
a. Pick a random start word
. For each cu
ent word, pick a random next word from your 2-gram dictionary by
looking up the possible follow-on words and choosing one at random.
c. If the chosen word is a stop word, your sentence is done, otherwise go to step b.
6. Does this produce anything reasonable or complete gi
erish? I suspect maybe the
latter. Submit your code and your best lyrics.
7. To earn a 5 on the lab: Modify your code to work with 3-grams. Now the start and stop
words become start / stop word pairs, and the 3 grams map pairs of words (tuples) to
lists of possible 3rd words. This should produce sentences that are more coherent. Does
it work?
[HAMILTON]
I am not throwing away my shot!
I am not throwing away my shot!
Hey yo, I'm just like my country
I'm young, scrappy and hungry
And I'm not throwing away my shot!
I'ma get a scholarship to King's College
I prob'ly shouldn't
ag, but dag, I amaze and astonish
The problem is I got a lot of
ains but no polish
I gotta holler just to be heard
With every word, I drop knowledge!
I'm a diamond in the rough, a shiny piece of coal
Tryin' to reach my goal. My power of speech: unimpeachable
Only nineteen but my mind is olde
These New York City streets get colder, I shoulde
Ev'ry burden, ev'ry disadvantage
I have learned to manage, I don't have a gun to
andish
I walk these streets famished
The plan is to fan this spark into a flame
But damn, it's getting dark, so let me spell out the name
I am the—
[HAMILTON/LAFAYETTE/MULLIGAN/LAURENS]
A-L-E-X-A-N-D
E-R—we are—meant to be…
[HAMILTON]
A colony that runs independently
Meanwhile, Britain keeps shittin' on us endlessly
Essentially, they tax us relentlessly
Then King George turns around, runs a spending spree
He ain't ever gonna set his descendants free
So there will be a revolution in this century
Enter me!
[LAFAYETTE/MULLIGAN/LAURENS]
(He says in parentheses)
[HAMILTON]
Don't be shocked when your hist'ry book mentions me
I will lay down my life if it sets us free
Eventually, you'll see my ascendancy
[HAMILTON & LAURENS]
And I am not throwing away
My shot (my shot)
I am not throwing away
My shot (my shot)
Hey yo, I'm just like my country
I'm young, scrappy and hungry
And I'm not throwing away my shot
And I'm not throwing away my shot
[HAMILTON/MULLIGAN/LAURENS/LAFAYETTE]
I am not throwing away my shot
I am not throwing away my shot
Hey yo, I'm just like my country
I'm young, scrappy and hungry
And I'm not throwing away my shot
It's time to take a shot!
[LAFAYETTE]
I dream of life without a monarchy
The unrest in France will lead to 'onarchy?
'Onarchy? How you say, how you say, 'anarchy?'
When I fight, I make the other side panicky
With my—
[HAMILTON/LAURENS/LAFAYETTE/MULLIGAN]
Shot!
[MULLIGAN]
Yo, I'm a tailor's apprentice
And I got y'all knuckleheads in loco parentis
I'm joining the rebellion 'cause I know it's my chance
To socially advance, instead of sewin' some pants!
I'm gonna take a—
[HAMILTON/LAURENS/LAFAYETTE/MULLIGAN]
Shot!
[AARON BURR]
How does a bastard, orphan, son of a whore and a
Scotsman, dropped in the middle of a forgotten
Spot in the Cari
ean by providence, impoverished, in squalo
Grow up to be a hero and a scholar?
[JOHN LAURENS]
The ten-dollar Founding Father without a fathe
Got a lot farther by working a lot harde
By being a lot smarte
By being a self-starte
By fourteen, they placed him in charge of a trading charte
[THOMAS JEFFERSON]
And every day while slaves were being slaughtered and carted
Away across the waves, he struggled and kept his guard up
Inside, he was longing for something to be a part of
The
other was ready to beg, steal, bo
ow, or barte
[JAMES MADISON]
Then a hu
icane came, and devastation reigned
Our man saw his future drip, dripping down the drain
Put a pencil to his temple, connected it to his
ain
And he wrote his first refrain, a testament to his pain
[BURR]
Well, the word got around, they said, “This kid is insane, man”
Took up a collection just to send him to the mainland
“Get your education, don’t forget from whence you came, and
The world's gonna know your name. What’s your name, man?”
[ALEXANDER HAMILTON]
Alexander Hamilton
My name is Alexander Hamilton
And there's a million things I haven't done
But just you wait, just you wait...
[ELIZA HAMILTON]
When he was ten his father split, full of it, debt-ridden
Two years later, see Alex and his mother bed-ridden
Half-dead sittin' in their own sick, the scent thick
[FULL COMPANY EXCEPT HAMILTON (whispering)]
And Alex got better but his mother went quick
[GEORGE WASHINGTON & COMPANY, BOTH]
Moved in with a cousin, the cousin committed suicide
Left him with nothin' but ruined pride, something new inside
A voice saying
“Alex, you gotta fend for yourself.”
He started retreatin' and readin' every treatise on the shelf
[BURR & COMPANY, BOTH]
There would have been nothin’ left to do
For someone less astute
He woulda been dead or destitute
Without a cent of restitution
Started workin', clerkin' for his late mother's landlord
Tradin’ sugar cane and rum and all the things he can’t afford
Scammin' for every book he can get his hands on
Plannin' for the future see him now as he stands on (ooh)
The bow of a ship headed for a new land
In New York you can be a new man
[COMPANY & HAMILTON, WOMEN, MEN]
In New York you can be a new man (Just you wait)
In New York you can be a new man (Just you wait)
In New York you can be a new man
In New York, New York
Just you wait!
[COMPANY]
Alexander Hamilton (Alexander Hamilton)
We are waiting in the wings for you (waiting in the wings for you)
You could never back down
You never learned to take your time!
Oh, Alexander Hamilton (Alexander Hamilton)
When America sings for you
Will they know what you overcame?
Will they know you rewrote the game?
The world will never be the same, oh
[BURR, MEN, & COMPANY]
The ship is in the ha
or now
See if you can spot him
Just you wait
Another immigrant
Comin’ up from the bottom
Just you wait
His enemies destroyed his rep
America forgot him
[MULLIGAN/MADISON & LAFAYETTE/JEFFERSON]
We fought with him
[LAURENS/PHILIP]
Me? I died for him
[WASHINGTON]
Me? I trusted him
[ELIZA & ANGELICA & PEGGY/MARIA]
Me? I loved him
[BURR]
And me? I'm the damn fool that shot him
[COMPANY]
There's a million things I haven’t done
But just you wait!
[BURR]
What's your name, man?
[COMPANY]
Alexander Hamilton!
[COMPANY]
1776. New York City
[HAMILTON]
Pardon me. Are you Aaron Bu
, sir?
[BURR]
That depends. Who’s asking?
[HAMILTON]
Oh, well, sure, si
I’m Alexander Hamilton, I’m at your service, si
I have been looking for you
[BURR]
I’m getting nervous
[HAMILTON]
Sir…
I heard your name at Princeton. I was seeking an accelerated course of study when I got sort of out of sorts with a buddy of yours. I may have punched him. It’s a blur, sir. He handles the financials?
[BURR]
You punched the bursa
[HAMILTON]
Yes!
I wanted to do what you did. Graduate in two, then join the revolution. He looked at me like I was stupid, I’m not stupid
So how’d you do it? How’d you graduate so fast?
[BURR]
It was my parents’ dying wish before they passed
[HAMILTON]
You’re an orphan. Of course! I’m an orphan
God, I wish there was a war!
Then we could prove that we’re worth more
Than anyone bargained for…
[BURR]
Can I buy you a drink?
[HAMILTON]
That would be nice
[BURR]
While we’re talking, let me offer you some free advice
Talk less
[HAMILTON]
What?
[BURR]
Smile more
[HAMILTON]
Ha
[BURR]
Don’t let them know what you’re against or what you’re fo
[HAMILTON]
You can’t be serious
[BURR]
You wanna get ahead?
[HAMILTON]
Yes
[BURR]
Fools who run their mouths off wind up dead
[LAURENS]
Yo yo yo yo yo!
What time is it?
[LAURENS/LAFAYETTE/MULLIGAN]
Show time!
[BURR]
Like I said…
[LAURENS]
Show time! Show time! Yo!
I’m John Laurens in the place to be!
Two pints o’ Sam Adams, but I’m workin’ on three, uh!
Those redcoats don’t want it with me!
Cuz I will pop chick-a pop these cops till I’m free!
[LAFAYETTE]
Oui oui, mon ami, je m’appelle Lafayette!
The Lancelot of the revolutionary set!
I came from afar just to say “Bonsoir!”
Tell the King “Casse toi!” Who’s the best?
C’est moi!
[MULLIGAN]
B
ah
aaah! I am Hercules Mulligan
Up in it, lovin’ it, yes I heard ya mother said “Come again?”
[LAFAYETTE & LAURENS]
Ayyyyy
[MULLIGAN]
Lock up ya daughters and horses, of course
It’s hard to have intercourse over four sets of corsets…
[LAFAYETTE]
Wow
[LAURENS]
No more sex, pour me another
ew, son!
Let’s raise a couple more…
[LAURENS/LAFAYETTE/MULLIGAN]
To the revolution!
[LAURENS]
Well, if it ain’t the prodigy of Princeton college!
[MULLIGAN]
Aaron Bu
!
[LAURENS]
Give us a verse, drop some knowledge!
[BURR]
Good luck with that: you’re takin’ a stand
You spit. I’m ‘a sit. We’ll see where we land
[LAFAYETTE/MULLIGAN]
Boooo!
[LAURENS]
Bu
, the