Custom Affiliate Marketing System
Project details
This particular Custom Affiliate Marketing System provides effective set of tools to navigate and operate within large quantities of diverse information coming in real time. Main purpose of the product is to earn money on providing a viable platform for ads designed for mobile app installations.
Our client had an existing CPA product and they were preparing to expand their business, so they needed to transform it into white-label marketing platform so that the system would be capable of handling planned traffic growth. Therefore, our client approached us with a request to build a custom affiliate marketing system that was aimed at mobile conversions.
Our task was to expand and refine toolset for data processing — make it scalable, add new parameters and metrics, make the design easy to follow.
Challenge: Move the System to Cloud
One of the biggest challenges that we dealt with during the development of the application was migration to the new system built on Google Cloud Platform. We needed to ensure smooth transition and scalability. It involved transferring several terabytes of data from a prior platform. Not only we needed to design effective inclusive architecture from the ground up, but we also needed to make its scalability flexible and highly operative.
Challenge: Create Data Analytics Feature
That challenge was combined with adding numerous new features to the mix. In order to make targeting and data analysis more precise and efficient we added several new types of parameters and diversified the metrics. This improved and stabilized operation of traffic control and targeting components.
Because of continuous addition of new features we needed to ensure possibility of continuous deployment without operational loss on the client’s side. Primary purpose of these additions were broadening of the scope and making the big picture more detailed.
Challenge: Develop Data Tracking for High-Load
Another challenging task was handling the tracking component. In order to improve performance and secure scalability we rewrote the tracking component so that it was able to handle large amount requests.
Challenge: Make It Usable for People
We also needed to provide the system with user-friendly interface that would not be too winding to master. With a little help of UX principles we have managed to make it as intuitive and comfortable as possible.
Our Solutions
Our first step was transferring of the system to Google Cloud Platform. While working on scaling we rewrote tracking component and data processing pipeline. This increased the stability of the system and made it more reliable.
In its current form the system combines high efficiency and intuitive design. From a monolith product, we are moving toward services, which offers more flexibility. We have a powerful admin panel with various levels of user permissions.
The system itself is targeted at conversions, instead of traditional aim at impressions and clicks. Therefore, the monetization scheme is built to maximize the conversions. Business features include complex and precise targeting as well as numerous instruments to make campaign and data management convenient for the end users.
At the moment the system is able to process more than 100 million events per day with about 5000 queries per second at the peak load.
The continuous integration is organized through blue-green deployment scheme which allows seamless addition of new functions with no downtime. This achieved by using two sets of servers for the product. One set is in active operation while the other set is kept on stand-by as release candidate. When the new version of the product is ready — servers are switching places.
Technology stack
In order to provide the best possible way of developing the application we have tried out several software options before settling on a definitive set. We wanted to create application whose maintenance would be completely independent of additional software.
The following tools were used to ensure smooth, uninterrupted operation in Android platform:
- Tracking:
- Symfony 4 and PHP 7.1
- Google Cloud for autoscaling features and cost reduction
- Google Stack Driver for monitoring;
- Redis Cluster for quick access to operational data.
- Analytics and data processing:
- Apache Beam on top of Google Cloud Dataflow for scalable streaming data processing;
- Java — is used for data processing;
- Initially we were using MySQL for interactive analysis of massive sets of data. Later we switched to BigQuery because of its integration.
- Google Cloud PubSub — real-time messaging service.
- Terraform, Packer, Ansible – toolset for automation of infrastructure
- PagerDuty – integration with this service provided alert system and 24/7 support for system health monitoring;
Our team consisted of:
- 1 Project Manager
- 3 Backend Developers
- 1 Frontend Developer
- 1 QA Engineer
- 1 DevOps Engineer
Summary
This project was huge technical achievement for our team. The entire architecture of the project was revised, upgraded, and the project was moved to the Google Cloud in order to simplify scaling procedures and reduce costs. considering the complexity of the project and requirement for easy scalability, we opted for Google Cloud services.
The backend also is optimized to process 2,000 queries/second on average (the peak load so far was around 5,000 queries/second). For this project, we have implemented an innovative solution for targeting ads with ability to select different IP database (IPDB) realizations per white-label (Maxmind and Digital Element.) This decision also positively affected the cost efficiency.
As the result of our cooperation, the client got a platform where his clients could register new CPA targeting systems. The analytics now feature over 50 parameters with 20 metrics, describing each impression, click and conversion. All this is presented in a simple, intuitive interface.
