How to Use R to Scrape Tweets: Super Tuesday 2016

By Kris Eberwein

Super Tuesday 2016 has come and gone, we have most of the election results, but what were the American public saying on Twitter?

The twitteR package for R allows you to scrape tweets from Twitter’s API and use them to form sentiment analysis. The Plotly chart below shows what the Twitter-verse was saying about the candidates during last night’s poll results.

A basic tutorial is below, but if you want to see my full code including the charts, you can check it out on my GitHub page. For the chart below, I scraped 20,000 tweets that mentioned each candidate and then ran them through a dictionary of positive and negative words to calculate the sentiment score, excluding neutral comments.

Basic Method for Using twitteR

First, you’ve got to have a Twitter account, (with your phone number attached). After you’ve got that, just go to Twitter’s apps page and create an application. Add the name and description of your app along with a website name. The website can be a test website. There’s also a field for callback URL, but that’s optional.

Sentiment Analysis with R

Grab your API keys and access tokens from Twitter, you’ll need them for the R script.

library(twitteR)
library(ROAuth)
library(httr)

# Set API Keys
api_key 
By now you should be in. Now time to grab some data.
# Grab latest tweets
tweets_sanders [email protected]', n=1500)

# Loop over tweets and extract text
library(plyr)
feed_sanders = laply(tweets_sanders, function(t) t$getText())

Now you've got a bunch of text data for Bernie Sanders, so how do we decide what's a “good” tweet and a “bad” tweet? This is where I turned to the Hu and Liu Opinion Lexicon, a list of 6800 positive and negative words compiled by Bing Liu and Minqing Hu of the University of Illinois at Chicago.
Unpack the Opinion Lexicon into your working directory and you should be ready to roll.
# Read in  ...read more
Source:: r-bloggers.com