Presentation of Crawlino, a next-generation distributed crawling system.
What it’s about
Crawlino is a distributed crawling system I designed to scout and analyze web applications at scale.
I explain the limits of traditional crawlers, its distributed and scalable architecture and how it does intelligent crawling with learning. I apply it to pentesting and reconnaissance use cases, at millions of URLs per day.
Why it’s relevant
Traditional crawlers don’t scale. When you have to analyze thousands of domains or millions of URLs, you need a different approach.
Crawlino solves it with a distributed architecture, a message queue and intelligent crawling.
Impact
Security teams adopted it for massive reconnaissance of attack surfaces in large organizations.