How to Learn Programming – A Programmer’s Track Preview, Witchcraft Sourcing with Twitter and Ruby

I will be presenting at Sourcecon in Austin about sourcing using Bash, Ruby, Python, and R.

Here is how I look on a happy day. Feel free to say hi if you see me around. I am often friendly 🙂

You can also connect with me on Linkedin.

 

Tom Ordonez at Sourcecon

 

 

Check it out on Wednesday, September 27 • 2:25 pm – 3:00 pm under the Programmers track

 

Witchcraft Sourcing at Sourcecon

This is a preview of witchcraft sourcing with Twitter and Ruby.

What you will learn from this awesome story:

  • How to create a CSV file from any Twitter list
  • The CSV file has valuable info about a user
  • Use the data for automating personalized DMs

This is a preview of what this CSV looks like.

 

Twitter Sourcing with Tom Ordonez

 

Twitter has about 100 million tweets a day.

Those are a lot of tweets.

100 million.

Actually is about 250 million.

Or maybe more.

Just to make math easy. Let’s stick with 100 million tweets per day.

That’s like 1,000 tweets per second.

These numbers are hard to visualize in your mind.

But if you have used a social media tool like HootSuite you see how those tweets fly…

…at warp speed.

You cannot even read them. They appear on screen, they get pushed down your timeline and disappear fast.

So how do you keep up with so many tweets?

Drum roll…trrrrrrrr

Twitter lists!

But Twitter lists can be both your best friend and your worst enemy.

If you have a moderate number of users added to a Twitter list, then the tweets might not fly away as fast.

There is a good chance that you will be able to read your Twitter timeline using a Twitter list (with a social media tool).

Let’s say that a “moderate number” is 50 users.

You can easily scroll up and down a day’s worth of those 50 users.

What if your Twitter list has 500 users?

…feeling dizzy now.

What if it had 1,000 users?

…I need a bag…not feeling good.

So many users, so many tweets, so much data.

Why is Twitter data useful?

You can do sentiment analysis using Twitter.

Google this: “twitter sentiment analysis.”

Twitter sentiment analysis

You will get about 1 million results.

Sentiment analysis is used to find out if a piece of text is positive, negative or neutral.

Social media is all about emotions, right?

That’s why emoticons are such a big thing.

That’s why Facebook added emoticons to their “Like” button.

What if you could find the emotions from tweets. That would be awesome right?

What if you could find if someone is in a good mood. You contact them, and they reply back.

But what if they are in a bad mood. You contact them, and they block you.

It’s a gold mine to have that knowledge. Positive, negative or neutral.

We will leave sentiment analysis for another life. As this can become very technical.

1.21 Gigawatts. Great Scott!

What the heck is a gigawatt?

Yeah…that technical 🙂

For now, we are going to focus on getting data from Twitter. You have to start somewhere right?

Getting data from Twitter

Here is a summary for getting data from Twitter:

  • Install Ruby
  • Create a Twitter app
  • Find Twitter lists
  • Use magic potions to get Twitter data

0. Hardware

Let’s start with zero.

I previously used a MacBook Air, 2011, 4GB.

I also used a Mac mini, 2013, 16GB.

Article Continues Below

A cheap Thinkpad, 16GB with Ubuntu Linux. Then I moved to Fedora.

I have used Windows since Windows 98. OMG, I am feeling old right now.

I am a hardware geek.

Hardware Geek

What’s your favorite hardware?

Ideally, you are on Linux. Although there is a good chance, you are on Mac or Windows.

In Windows, it will be a little bit painful because you need a terminal to enter commands. And although Windows has one, it has different commands than Linux or OSX.

For Windows, I recommend that you install a Linux Virtual Machine. If you search on Youtube “install Linux VirtualBox on windows” you will find a lot of good tutorials to follow.

Linux Virtualbox

1. Installing Ruby

Ruby is a programming language. Twitter was initially built using Ruby on Rails. Now they use Scala and Java.

Installing Ruby is (sort of) easy depending on your computer abilities 🙂

You can install Ruby using two ways:

Ruby has a component called gems which are software packages with different features.

2. Installing the Twitter gem

There is a gem called Twitter CLI that uses the Twitter API and Ruby.

On your Terminal, it’s as simple as:

gem install t

3. Register an application on Twitter Developer

Go to Twitter Application Management.

  • Sign-in using your Twitter account
  • Click on Create New App
  • Fill out the form
  • Go to Permissions and set Access to “Read, Write and Access direct messages.”

Go to the command line and type:

t authorize

This will send you to an URL to authorize the application.

4. Use magic potions to get Twitter data

Get a list of all available commands:

t help

Twitter Sourcing with Tom Ordonez

Send a tweet from the command line:

t update "I am tweeting from the command line. So awesome"

Get details about a Twitter user:

t whois tomordonez

Twitter Sourcing with Tom Ordonez

Get data from a Twitter list:

I see that Sourcecon has five lists.

Twitter Sourcing with Tom Ordonez

 

 

The first list is called “SourceCon 2017 Speakers”. It has 26 members (as of this writing). I just realized I am not one of them. Sad face.

 

 

Twitter Sourcing with Tom Ordonez

This is the URL of this list:

https://twitter.com/SourceCon/lists/sourcecon-2017-speakers

What you need to get is the user name and the list name:

  • User name: SourceCon
  • List name: sourcecon-2017-speakers

Now let’s create a CSV file with all the users that belong to that list.

t list members -c SourceCon/sourcecon-2017-speakers > sourcecon-2017-speakers.csv

Twitter Sourcing with Tom Ordonez

These are some of the columns from the CSV file:

  • Date joined
  • Date of last tweet
  • Number of tweets, favorites, listed, following, followers
  • User name
  • Name
  • Bio
  • Location
  • URL
Twitter Sourcing with Tom Ordonez

What to do with this data?

  • Automate sending personalized DMs
  • Sentiment analysis
  • Data mining
  • Create a data warehouse

I love computers. My favorite tools are Linux, Python, Azure and AWS. I have two engineering degrees. Built teams at Sears. Now at WalmartLabs. Growing teams in Walmart eCommerce for machine learning, data science, data engineering and product management.

Topics