Query API, Twitter wacky race using Highcharts

We have been – anxiously – following United States political scene lately. As it reveals pretty anxiogenic, I decided to play around, gamifying both JavasScipt learning and 21st century reality check…
You miss the old good days! When wacky racers were desperately trying to catch the little bird? Let’s see how American political leaders nowaday catch another bird and attract audience…

The challenges

I want to run a Twitter followers race. Contenders will be US political leaders.
But how do I display the actual race? After some sketches I decided to go with a basic bar chart using Highcharts. As I wanted to gamify the exercise, I will implement a special version of it, a “wacky race” version, using actual characters from the Great Old cartoon.

1. The data access issue

To have an up-to-date data my application needs to be plugged to an API. My first thought was to use Twitter API, as it allows me to access an immense amount of data. Nevertheless, I don’t need all this data, and the authentication process is a bit challenging to set up because It requires registering the application. So I decided to go with Yahoo query API, scraping technique, and to use publicly available data with just a user follower count on twitter.

The following URL fed with a Twitter account ID will provide me with the associated follower count:

url = "https://query.yahooapis.com/v1/public/yql?q=select%20content%20from%20html%20where%20url%3D%22https%3A%2F%2Ftwitter.com%2F" + ID + "%22%20and%20xpath%3D'%2F%2Fli%5Bcontains(%40class%2C%22ProfileNav-item--followers%22)%5D%2Fa%2Fspan%5Bcontains(%40class%2C%22ProfileNav-value%22)%5D'&format=json&callback="

This solves – temporary – my data problem

2. Scrapping or not scrapping…

Using a scaper rather than Twitter API raises 2 mains challenges:
Scrapping is time consuming, as it could take seconds to gather the data! to resolve this challenge, I use setTimeout function to allow the necessary time to collect all the data (from 16 users) before rendering the chart:

setTimeout(function() { // allow to wait for data to be loaded to draw the chart

//... Highcharts

}, 5000); //end of set time out function

I also created a little countdown, to entertain the users while waiting for data to be collected and processed.
The second drawback comes with scraped data. The collected data is a string rather than numbers; here are some examples “15,3 k”, “13,3M” or “371 k”
To deal with this challenge, I wrote a piece of code that identifies whether the number is expressed in Million or Thousands. Then I get rid of the string, split the number, and add the necessary number of zeros. Doing so, you have to forget about the precision.Hopefully, the gap between our contenders is such that it does not to affect the results of the race, so far.

Check below the procedure to use:

 

var thousandK = "K"
    var million = "M";
    var thousand_coma = ",";
    if (string.indexOf(million) !== -1) {
      var split = string.split('.', 2)
      if (split.length > 1) {
            var Million = parseInt(split[0]) * 1000000
            var Thousand = parseInt(split[1])
            if (Thousand < 10) { Thousand = Thousand * 100000 } else if (Thousand > 9) {
              Thousand = Thousand * 10000
            } else if (Thousand > 99) {
              Thousand = Thousand * 1000
            }
              var count = Million + Thousand
          } else {
            var Million = parseInt(split[0]) * 1000000
            var count = Million
                }
           } else if (string.indexOf(thousandK) !== -1) {
      var split = string.split('.', 2)
     //to deal with K or M with nothing after it...
      if (split.length > 1) {
        var Thousand = parseInt(split[0]) * 1000
        var Hundreds = parseInt(split[1]) * 100
        var count = Thousand + Hundreds
      } else {
        var count = parseInt(split[0]) * 1000
      }
      //dict_count.push(count);
    } else if (string.indexOf(thousand_coma) !== -1) {
      var split = string.split(',', 2)
      var Thousand = parseInt(split[0]) * 1000
      var Hundreds = parseInt(split[1])
      var count = Thousand + Hundreds
        //dict_count.push(count);
    } else {
      var count = parseInt(string)
    }

Since it also wanted to improving my skills in JavaScript, I decided to spice it up with few personalizations such as the ability to select different characters and assignments. My users can race with Barack, Donald, Hillary, Bernie, etc. and chose their favorite character.

3. Using Images in Highcharts

Last but not least, I needed to display wacky characters on the chart. I had in mind examples provided on Highcharts web site such as http://www.highcharts.com/demo/spline-symbols. Ultimately I used a formatter function in the data series to be able to use HTML within the chart:

 data: [{y: count_array[13],
      dataLabels: {enabled: true,useHTML: true,
        formatter: function() {
          return '<div><img src="http://www.stats4dev.com/wacky/' + char13 + 'ico.png">' + dict_count_formatted[count_array[13]] + '</div></div>';
            }
         }
     }]

The results: the twitter wacky race application

Ad the final result here: