Using Twitter Data to Compare the Popularity of 3 Programming Languages

Twitter is a rich source of useful information. It is a great tool for:

  • Researching Needs (for early customer development),
  • Tracking Trends (in your industry),
  • Watching Competition,
  • Finding  Influencers in your industry segment

We have been dabbling in some tools for mining Tweets and I am always on the look out for more.

There are a few kindred spirits who seem to be interested in similar topics. Here is one Scooped by Jose C Gonzalez. An Introduction to Text Mining using Twitter Streaming API and Python  by Adil Moujahid

Twitter data constitutes a rich source that can be used for capturing information about any topic imaginable. This data can be used in different use cases such as finding trends related to a specific keyword, measuring brand sentiment, and gathering feedback about new products and services.

In this tutorial, I will use Twitter data to compare the popularity of 3 programming languages: Python, Javascript and Ruby, and to retrieve links to programming tutorials.

This tutorial teaches you:an approach to mining tweets, analyzing them and visualizing them using simple open source tools. You will learn:

  • How to use a  Twitter Library for Python to find tweets on specific topics (in this case Python, Ruby, Javascript)
  • How to decode JSON, returned by the Twitter searches
  • How to use a Python library called Pandas to analyze Twitter Streams
  • How to use another Python library (matplotlib) to plot the results of analysis












It is How Things Work

I liked this essay and I think it is a must read for every software professional. Here is a small snippet from Matt’s What Have You Tried?

it’s just how things work: you begin with a lack of understanding about a topic, and a need to solve a problem in that topic area. The honest, sustainable means of doing so is to improve your understanding. This is achieved by:

  1. Formulating a question which, when correctly answered, will improve your understanding in some way; then:
  2. Attempting to answer it.

Being active in a few forums, I can relate to the problem Matt is talking about.

Good essays/posts are a pleasure to read. When I find some, there is a strong desire to share. I just realized that I don’t have to always write an entire blog post to provide something of value.


Matt Gemmell – All This Stuff Scares the Hell Out of Me

All of this stuff scares the hell out of me:

  • Software patents, and their use as a financial weapon.
  • The walled garden of the various App Stores, with mysterious and ever-changing rules governing admittance, and the constant threat of capricious rejection.
  • The consequent relative invisibility of non-App Store software.
  • The incredibly crowded market, with imitations and duplicates of popular titles springing up overnight.

I don’t want to sound too negative. But a tweet from Tim O’Reilly got me reading a chain of posts.

Tim Bray · Discouraged Developer

Ed Finkler – “I find myself more and more concerned about my future as a developer. ”

Who, What and Where of IOT Jobs

IoT devices offer huge potential for electronic component manufacturers, but this is clearly not where the value will stop. Most of the added value in IoT solutions will come from the processing of the generated data. In fact, the ratio between electronic components and data processing can reach 1:50 in certain long-term cases!

Technologies & Sensors for the Internet of Things mentions three types of companies that will benefit from IOT initially.

The IoT is a multi-billion dollar market emerging from several different markets (i.e. industrial sensors, wearable electronics and home automation) which will see strong convergence in the next five years. Three industrial and service sectors will be integral to the valorization of this new market:

  • The electronics industry, which will manufacture the sensing devices
  • The communication and cloud data storage industry, which will handle data transmission, storage and processing
  • Service companies, which will valorize the data either through processing or by selling to a third party

This got me interested in finding jobs in the IOT space. Wanted to find out who is hiring, what kinds of jobs are being offered and where are they hiring. I did a simple Data Journalism style  experiment.

  1. Tried a job search in Indeed and SimplyHired for IOT jobs (Indeed and SimplyHired are job aggregators)
  2. Got the list of jobs (wrote a small Python script to extract the job list).
  3. Wrote another program to take the list of jobs  and extract entities using Open Calais API. Open Calais is a tool for extracting entities from text. They have a free API (with certain rate limits) that you can use to automate the entity extraction.
  4. A final tiny Python program took the entity file (generated from the previous step) and extracted the cities, job positions and companies.
  5. Here is the output of the first run from steps 1-4 and some minor edits to the output.


CA Software
Cypress Semiconductor Corporation
First Data
Honeywell International
Juniper Networks
LogMeIn Inc.
Product Development Company
Quirky Incorporated
Sasken Communication Technologies
Wink Inc.
iOT Company

Job Positions

Software Engineer
Product Manager
Senior Application Engineer
Manager – PTC
Host Protection Architect
Cloud Architect
IOT Solution Architect
Director of Evangelism
Princ Research Engineer
UX Designer
Manager – Internet
Test Engineer
iOS WiFi IOT Engineer
Legal Counsel
Home Research Scientist


San Jose
Mountain View
New York
Santa Clara
San Diego

This is a small sample and just a tiny peek into the industry.

The cool thing about languages like Python is that you can write about 10-20 lines of code to extract, clean and generate useful data. In this specific instance, I reused some code and wrote a couple of simple scripts. I need to clean up the code and turn it into a more usable tool.

Meta: Writing a data blog is fun. It took me a couple of hours of exploration  and writing and testing scripts, but I feel it is worth the time I spent. I also learned a bunch of things I did not know before.


Educational Technology for Schools

Technology in Education is one of my interests. With the advent of low cost tablets and ubiquitous cloud computing, the way we teach and learn is bound to change. But it is not enough to simply provide technology. We need innovative ways to harness technology.

A small team from KCG Social Causes Club tried a few experiment at a couple of public schools in Chennai. We observed a few things.

  • Kids are excited by tablets and smart phones. They seem to have an intuitive understanding of touch computing and require almost no training to discover their capabilities.
  • While there are lots of free games and content available in English, there is not much  for people in other languages. In Chennai at the middle school level students speak a mixture of Tamil and English.
  • There are huge gaps (and opportunities) in using tablets, smart phones and cloud connectivity in innovative ways to engage students. Our college students who were guiding the school kids found it a big challenge. So some of them started building apps for education.
  • We cannot simply leave the task of innovating to schools and teachers. Technology companies need to play an active role in translating teachers’ ideas into simple apps for kids.
  • Knowledge is required in areas of user centric design, app building, gamification and we need to experiment with lots of different ideas to engage students and learn how they learn.

It is nice to see NMC and CoSN Release the NMC Horizon Report  on the trends, challenges and emerging technologies for schools. Reports like these provide us a broader view and helps us understand the trends and challenges of using technology in education.

The report covers:

Six key trends, six significant challenges, and six emerging technologies are identified across three adoption horizons over the next one to five years

… identifies BYOD and cloud computing as technologies expected to enter mainstream use in the first horizon of one year or less. Games and gamification and learning analytics are seen in the second horizon of two to three years; The Internet of Things and wearable technology are seen emerging in the third horizon of four to five years.

More after I digest the report. I do need to work on an action plan for the next batch of the Social Causes team at KCG Tech.

Stories of Programming Language Use: Caml

Over time, the company has used Caml to build not only its core trading system, but also a concurrency library, a sophisticated publish-and-subscribe system, and a collection of system automation tools.

Overall, the company has “a huge number of systems maintained by a small team,” Minsky said.

Over 3 million lines of code, is no small matter.

From You won’t believe what programming language this Wall Street firm uses | ITworld

Snippets Worth Sharing: Deep Links

Deep links are a kind of hyperlink that points to a specific place or function inside a mobile app. A deep link on a mobile Web page or in an e-mail, for example, might take you to a specific product in a shopping app or song in a music app.

Support for deep linking has been built into Apple and Google’s mobile operating systems for some time. It allows app developers to give URL-like identifiers such as “iOSApp://location/123456” to specific sections within an app. However, uptake by mobile app developers was initially slow due to differences in how the feature worked on different operating systems and a lack of broad support for their usage.


The Hyperlink Gets Reinvented

Search Startup Quixey Aims to be the Google of the App Era


Get every new post delivered to your Inbox.

Join 7,863 other followers