How the crawler works

SAEM uses our in-house built crawler which gives us many advantages. One of them is the ability to modify the tool for our customer's needs and report error types that are important for their business.

Our users can select any location they want the crawls to be done from and even choose multiple locations. You also have different user agents to choose from to use in the crawls.

When you define the start URL the crawler will begin navigating your website from that exact start URL and will never leave the path you input in the settings. For example if you set your start URL as “www.mysite.com/path1” SAEM will analyze all those URLs within the “path1” and nothing else.

SAEM is also smart enough to understand any domain changes and always follows the new domain. This means that SAEM is combining new crawled information with the information from the previous domain crawls and you will always have all the data under one project even if the website's domain changes.