Wildfires are one of those natural disasters we cannot control. However, by applying modern technologies, we have a chance to predict them and reduce the consequences for both individuals and businesses.
We want to share our experience of working on GlideFinder, a platform that uses images from NASA satellites to analyze information about wildfires in real-time, and alert people nearby.
Once we got an email from Dmitry Kryuk, GlideFinder Founder and CTO, who was looking for experienced Google Cloud developers. He wanted us to improve the product by making several changes to the product's architecture. In particular, he needed to integrate streaming data analytics tools and connect additional NASA satellites for more accurate and precise wildfire information.
GlideFinder is a platform that locates wildfires, alerts subscribers, and provides analytics on time. Initially, GlideFinder was developed for the California region, where fires are especially dangerous and annually destroy hundreds of homes, resulting in multi-billion dollar damages, business bankruptcies, and lives lost. Now, the company wants to help people all over the world to reduce the consequences of wildfires.
The GlideFinder platform leverages both historical and real-time data sources. For real-time alerts, the platform gathers data from:
To make predictions about the spread of wildfires, the product analyses data from:
Once GlideFinder detects a wildfire, it alerts users about the danger via SMS. Users can also monitor the fire's movement, direction, speed, and size on real-time maps integrated to the GlideFinder website.
GlideFinder users can also see how far the wildfire is from their homes, their children's school, parents' homes, and offices.
Most of the data analytics happens on the Google Cloud Platform, which reduces latency and provides all the necessary data analytics tools.
Dmitry needed a team to upgrade the product. For this task, he had the following requirements:
To integrate streaming data analytics, we developed an ETL (extract, transform, load) pipeline that builds on Google Cloud components such as Dataflow, PubSub, Cloud Functions, BigQuery, etc. We also materialized the views with the required business logic and used a Dataflow job to process that data and write into JSON files in Cloud Storage and Cloud Firestore.
During Data Flow integration, we needed to use Python SDK, because the entire architecture was developed using this programming language. Since Python architecture is less documented (unlike Java, which we apply more often), we dedicated a significant amount of time on the investigation process.
To add open-source geostationary operational environmental satellites, we had to dig deeper into how to work with geo-data, so we conducted further research. We took images received from the satellite and split them into geographical zones using geospatial transformations.
At this stage, we also applied to the Geospatial Data Abstraction Library, a computer software library for reading and writing raster and vector geospatial data formats.
It took us about four months to complete all these tasks.
For this project, we created new Buckets (essential containers that hold the project's data) on Google Cloud Storage. Then, to upgrade the product's infrastructure, we gradually migrated batch data components to streaming infrastructure.
The integration of GOES-16 and GOES-17 took us through the following actions:
Google Cloud Components:
We empowered GlideFinder with a new infrastructure best suited for streaming data analytics. We also integrated geostationary satellites GOES-16 and GOES-17 that make new scans every 5-15 minutes, so the platform's users can receive even more relevant data about nearby wildfires. You can try the platform on the link.
We are proud to have participated in GlideFinder's development because we had the unique opportunity, not only to enhance our Google Cloud Platform expertise, but also to help others save lives.