Fraudulent and malicious activity is a big problem all over the Internet. It is especially a sensitive subject in the realm of digital advertisement.
In many cases, fraudulent activity can throw a wrench into business operation and seriously disrupt the workflow. For those who are managing ads and ad spaces - detecting and eliminating any notion of fraud is matter of life and death.
A few ad fraud statistics for you:
- 1 out of 5 ad-serving websites is only visited by fraud bots
- 20% of pay-per-click conversions were fraudulent in 2017
- Mobile display ad fraud: 30 billion fraudulent impressions per minute
- Facebook disabled a total of almost 1.3 billion fake accounts in 2018 (and deleted 865 million spam posts)
The problem of fraud is critical in cases of real-time bidding. Advertisers can get into hot water if some fraudulent ad content will slip through and cause a fuss. Such instances must be prevented wholly.
The custom ad fraud detection system, that the APP Solutions team has developed, is designed to detect and report fraudulent and malicious adverts before they cause any damage.
The fraud detection system is a twofold project, consisting of:
- Ad data processing and analytics
- Ad crawler
In order to do that, the system uses a multi-layered system of monitoring activity and reporting anything resembling anomalous activity or malicious content before it started rampaging around.
Another element of the system is the crawler engine. Its purpose is to check the credibility of the ad publishers. Ad Crawler goes through adverts for fraudulent and malicious ad content and assesses its potential of being such. The method includes scanning ad publishers websites, analyzing incoming requests and reporting on it in cases of fraud detection.
The system also uses serverless tracking pixel to process information coming from mobile tracking systems.
This particular Fraud Detection System counters the following types of Fraud:
- Cookie Stuffing
- Click fraud:
- Click bots
- Click farm
- Click spamming
- Click injection
- Lead fraud:
- Lead Bots
- Lead Farms
- Impression Fraud
Among the prevention methods we have implemented were:
- Digital Footprint / Signature-based - this method uses predefined patterns to detect suspicious activity.
- Anomaly-based - this method uses statistical analysis and historical data to inspect suspicious kinds of content and determine whether it is malicious.
- Credential-based - this method is used by a web crawler to assess potential fraud activities.
Fraud Detection is a resource demanding operation that requires high scalability of the system. Given the fact that the system processes a large amount of incoming information — it is critically important to make it able to scale according to the workload.
In order to provide smooth and reliable scalability in cases of processing large quantities of incoming information, we’ve used Google Cloud Platform and its autoscaling features.
Also, we applied serverless tracking pixel to secure smooth scalability and balanced data processing from multiple sources.
The centerpiece of the Fraud Detection System is an intricate set of data analysis algorithms that monitor and assess content and activity.
However, in order to make it work effectively, we needed to refine the algorithms of fraud detection. This required a substantial study of various types of fraudulent and malicious ad content and activity. This research became a foundation of the algorithms.
Data Analysis itself is rather demanding process on the resources side. In order to keep data processing workflow uninterrupted no matter the workload, we used Apache Beam.
Another significant challenge came during the development of the crawler engine. We needed to refine its working process and include every type and variation of fraudulent and malicious ad content in order to make the assessment precise.
In order to secure efficient monitoring - we needed to gather a database of references. This was achieved through extensive research of various fraudulent and malicious ad content.
Also, in order to make the crawler identifiable as a real mobile visitor, we have developed a specialized app and used real actual Android / iOS mobile devices.
- Google Cloud Platform
- Apache Beam
This team was a unique one because this project's DevOps engineer was at the same time the Project Manager (it was due to the fact that this was an existing client and so most of the tech details were known and processes set up.)
- 3 Senior Developers
- 2 QA Specialists
- 1 DevOps Engineer
This project was a real test of skill for our team. Because of the numerous technical challenges we learned a lot about the subject during the development. We performed thorough studies on fraudulent and malicious ad content. Also, we made a research on the subject of how fraud content tries to counter fraud detection.
That broadened the scope of our fraud detection solutions and allowed us to develop extremely efficient fraud detection system that is able to operate under significant workload without slowing down or crashing.