Web Crawler is a utility designed for testing and demonstration of the WebEngine library features. It gathers information about resources of a specified web server by analyzing references in the HTML markup, text, and. Additionally, a query is sent to the Web Of Trust knowledge base to obtain information about the analyzed site. Download Crawler v0.2 (Crawler_v0_2.2) here. A very good feature offered by this tool is that you can implement your own plug-ins to improve the functionality of this application.
Source: https://thehackernews.com/2010/11/web-crawler-web-application-crawler-new.html