Loading millions of points in Highcharts

How do you load millions of points in Highcharts?

The browsers have limitations when it comes to great amounts of data. Think about loading 1 million points of OHLC data – with three and four values each point, it will mean megabytes of data and heavy looping to process it. Perhaps some modern browsers can cope with it, but legacy browsers won’t get anywhere near it.

Enter the server. Backed by a database, we can load just the resolution we want to look at. In our new lazy loading demo, we set up a backend in PHP and MySQL. The original data is minute data for the AAPL stock, in total 1.7 million records. We wanted to display a candlestick chart with open, high, low and close data values, so we hit a limitation in MySQL. Since it lacks the concept of first and last in groups (open/close), we needed a nested select, which made it slow. So instead, we created four data tables holding data for minutes, hours, days and months respectively. If we wanted to display a simple line chart with average values, we wouldn’t need more than the original minute table.

Furthermore, to prevent the Highstock navigator from adjusting to the new data and sending off another setExtremes event, we needed to introduce a new option, navigator.adaptToUpdatedData. Now we can do whatever we want with the data in the main chart without affecting the navigator series, which always displays the entire data range.

Now as you can see in the source code below, the client server connection happens through jQuery.getJSON. We hook up to the X axis afterSetExtremes event, which unlike setExtremes fires after the new extremes have been adjusted for the X axis minRange and other variables. In the event handler we show a loading screen, then send off the new extremes to the server. The server then chews through this information, decides which resolution it should go for, selects the data and sends it back. Client receives it, hides the loading screen and sets new data to the series.