We know how important the storage, processing, analysing and visualising of large data sets can be, especially in the fast-paced world of Science and Research where speed, accuracy and flexibility are all significant factors.
At Spicule we have the knowledge and experience to design, implement and support robust data processing platforms tailored to the specific needs of our clients across a wide range of Science and Research sectors. Here’s how we used Apache Druid to solve one particularly unique challenge.
When a satellite is launched into space, getting it into orbit is only the beginning of the journey. Once it’s up there the satellite will receive, generate and transmit a monumental amount of data, all of which needs to be securely downloaded and stored so that the scientists back on Earth can analyse what it’s telling them in real time, instantly compare it to historical trends, and respond to any changes or anomalies as they happen. The data can take many forms depending upon the satellite’s purpose, but it will also include crucial details like spacecraft positioning and orientation and the functional status of onboard hardware and software. After all, if something’s close to failure or your satellite’s orbit is decaying, you’ll want to put another satellite up there as quickly as possible to fill the gap.
When you consider the petabytes of data a single satellite could produce over the course of its lifetime, think how much more complicated and data-heavy the problem becomes when a constellation of satellites is launched into space and each of those satellites has to be tracked and analysed individually, historically, and in relation to all the other satellites within that constellation.
Our challenge was to design and build a platform that could a) store the massive quantities of data each satellite was producing on a second-by-second basis, b) analyse the data in real-time while simultaneously comparing it to historical trends, and c) generate the results in a format that was quickly and clearly understandable via an intuitive custom-user interface.
Apache Druid is infinitely scalable – both up and down – which means it can comfortably ingest any amount of streaming data the satellites send down to it, regardless of volume and without any negative impact on Druid’s speed and performance. Also, because Druid enables fast analytical queries at high concurrency on a sub-second basis, the users can interrogate and begin gathering results from the data as soon as it enters the platform. Since Druid combines the key characteristics of timeseries databases, OLAP/analytic databases and search systems, it’s the fastest and most effective way to run rapid analytics on real-time and historical data stores, identify trends and iterate on queries to answer questions. And, if a server fails, Druid’s fault-tolerant architecture instantly re-routes around it with no loss of data or service.
As far as generating the results was concerned, we designed a bespoke front-end interface which is easy-to-operate and provides the users with a variety of different reporting options including graphs, charts and tables. We also ensured that the platform we built integrates seamlessly with the client’s existing systems.
As a result, Druid provided the perfect cutting-edge solution for our client. It gives them the ability to store and stream limitless amounts of data, monitor and query satellite operations second-by-second in real time, run experiments, and formulate and confidently predict how their satellites will perform today, tomorrow and into the future.
From satellites to genomics and everything in-between, Druid’s speed and flexibility makes it the perfect platform for any area of scientific research – whether you’re processing large amounts of historic data or providing real-time analytics on a brand-new data stream.
Whatever your requirements may be, we’ll work with you to develop a solution which meets your needs and will help you find the answers to your questions.
Spicule have offices in Norwich and Cambridge, and we work with clients in the UK and around the world. We pride ourselves on designing and delivering the best possible solution to meet each client’s specific need, and we are able to provide 24/7 on-site and remote support whenever required.
Whatever your real-time data management problem may be, we’ll work hard to ensure the results you’re looking for.