ApacheCon 2016

Most users who consume data from the web are concerned with a subset of documents in the crawlers output. Though crawlers like Nutch offer flexible configuration mechanisms to make the crawl focused on interesting pages, it is almost impossible to isolate the less interesting data from the more important information that the crawler should be focused on. In this presentation, Thamme Gowda and Chris Mattmann will describe a useful clustering technique they formulated by combining various similarity measures on DOM structure and CSS styles of web pages including Tree Edit Distance and Jaccard Similarity. The clusters can be thus used for extracting interesting data and applying special analysis based on cluster content. They also showcase an implementation of this technique which is planning to contribute to Apache Tika and shows how this can be scaled to web scale using Spark's MLlib.