Advertisement

Using TweetBeaver to Extract Data from Twitter

Article main image
Oct 29, 2019

In the past I‘ve tried sourcing candidates from Twitter by manually looking at who followed targeted accounts. This was often tedious, not an effective use of my time, and it mostly involved a lot of aimless browsing hoping that a key word would catch my eye. Then I hit the jackpot with Tweetbeaver.

Tweetbeaver has 14 tools, all of which are free and can be used to extract data from Twitter to a CSV file. The tool I found to be most useful on TweetBeaver is the ability to download an account’s followers, which the site returns along with information such as location, bio and any links associated with their profile. This is a game changer.

You can get started by visiting the Tweetbeaver home page and clicking the button to “Download a user’s followers list.” From there enter the name of your desired account and click “Download as a CSV”.

Tweetbeaver screenshot showing how to download a user's followers

Instead of haphazardly browsing pages of followers for each Twitter account, I used the CTRL+F search function in Excel to find the information most relevant to my search. For example, you could search for “linkedin” or “github” to see how many users on the spreadsheet have those sites included in their Twitter profile URLs.

One drawback of this tool is that it only has the capacity to fetch data on up to 10,000 users. If you attempt to download followers from an account with more, it will only return the 10,000 most recent followers. You may get an error message if you try downloading larger lists in a Chrome browser, so I recommend using Firefox if you intend to pull data with the maximum amount.

I started using the tool in my search for professionals with active security clearance so naturally, I headed over to the @ClearedJobsNet. Out of their 5738 followers I found that 388 of those followers had a LinkedIn URL attached to their profile. Unfortunately, this account would not be the treasure trove of professionals with top secret clearance I was hoping for. 394 followers were recruiters, 380 were business accounts and very few were actual jobseekers. However, searching by keyword allowed me to quickly narrow the followers down to the viable candidates. Ultimately, I was able to find the LinkedIn URL of a full-stack developer who held TS/SCI with a full-scope polygraph clearance. The candidate is currently being scheduled for an interview so using this tool was more than worth my time.

This technique is also useful for other searches. I investigated an account called @pybites, a community of Python developers that sharpens their skills through coding challenges. Their account comes in just under the limit at 9539 followers, so I was able to pull data on all of their followers. Looking at the results, It was clear that this account was home to many of the developers that I was looking for – a genuine IT community.

Excel search

Here are a few points of useful data I was able to discover by using the search function in Excel.

  • 1409 list themselves as developers or software engineers in their bios
  • 368 have a GitHub URL in their profile
  • 195 accounts have LinkedIn URLs
  • 61 had links to their personal Instagram
  • 18 accounts have a Gmail addresses

Not bad!

Tweetbeaver has 13 other tools that I have yet to explore. I’m looking forward to digging in.