Way to react to API failure

APIs can be described as a tap delivering data instead of water, as it provides a continuous stream of information. With APIs, developers send queries to a server and get access to data or even plug web applications to an up-to-date data-source.

In my field, International Development, The World Bank is the most important source of data to study economic and social development. World Bank’s API allows developers to take advantage of open data and automation; so that everyone can create up-to-date applications, as applications can be plugged directly to the data servers, and have access to live data.

But here is my issue, I am torn between a full automatization and a constant monitoring of an update. If I go with an automatization, there is a risk of a crash when major updates occur, such as indicators names revision, series discontinuity, etc. And if I don’t use automation, that means I have to constantly monitor any data update. As a developer and maintainer of a dataviz portal on international development for the French Development Agency; I have to provide users with a continuous service.

To provide that service and enjoy the holidays gained out of automation, I need a backup plan.

CONTINUOUS SERVICE: LOCAL BACKUP AND URL SWITCH

Hospitals have local generators that switch on in a case of a power grid crash. To allow a continuous dataviz service a switching mechanism and a local generator are required. But how this concept could be used in my case? If the API request fails, the chart will use a local data. The local generator will be our local JSON file, and the switch will be probably a script in javascript.

Let’s do it.

1ST STEP: CREATE A LOCAL JSON FILE

Click on the following link to download the JSON file.

2ND STEP: A SWITCHING MECHANISM

The following javascript send the initial request to the API with the method getJSON. If it succeeds;  the chart will use the requested URL data. Otherwise, local file will be selected:

// Request to the API
var url_API ="http://api.worldbank.org/countries/CIV;
LMC;SSA;WLD/indicators/EG.USE.COMM.FO.ZS?
per_page=80&MRV=20&format=jsonP&prefix=?"
// Local folder where the file created in step 1 is stored
var url_local ="data/CIV_EG_USE_COMM_FO_ZS.json"

// variable that will take the value 1 if the call is successful
API_success =0

//------- Try 1st url, if succeed, API_success=1
$.getJSON(url_API, function(d) {
API_success =1; // if failure, API_succes remains=0
url_success=url_API;
document.getElementById('bulb_success').style.color ="#6E9FC5";
});

//--------- if getJSON fails give a new URL ------------
setTimeout(function() {
if (API_success ==0) { // i.e. if failure
url_success = url_local
document.getElementById('bulb_success').style.color ="red";
};
}, 200);

LAST STEP: CREATE THE DATAVIZ WITH HIGHCHARTS

To create the interactive charts, I use Highcharts library; as It is very easy to implement and can be customized with CSS. In this example, I use IPython notebook to write the HTML file hosting the chart (click here)

To check the API connexion status, I am using a font awesome icon, the icon turns blue if the chart reaches the API and turns red if the chart gets the data from the local file. This way, I will know if there’s any issue and eventually update the script. As a demonstration, let’s try API request that doesn’t succeed.

var url_API = "http://FAKE_API_request/countries/CIV;
LMC;SSA;WLD/indicators/EG.USE.COMM.FO.ZS?
per_page=80&MRV=20&format=jsonP&prefix=?"

The following chart uses this URL:

HTML('<iframe src="Dataviz_with_FAKE_API_request.html" scrolling="no" 
frameborder="0" width=100%" height="475px"></iframe>')

As shown in the above chart, the icon turns red, following a failed API request, indicating that the chart is fetching data from the local file. The data is not automatically updated, but at least it is not broken, and I can work to adapt the script, and manually update the local data file if necessary. Now, I  have time to maintain the service and I can enjoy my holidays.