myWebby Crawler

myWebbyBot is myWebby's web crawling bot. Crawling is the process by which our bot discovers new and updated pages to be added to our search engine index.

robots.txt

We fetch the robots.txt file before crawling your site.
Before we save any website to our index, we also search for the robots meta tag, if we aren't allowed to index the site or to follow the links we won't do it.
If you don't want us to crawl your website or a specific path on your website, add the following rule into your robots.txt file: User-agent: myWebbyBot
Disallow: /privatePath

User agent

The user agents of our bot are currently the following: myWebbyBot/2.1 (+http://my-webby.eu/bot) myWebbyBot/1.0 (RSS Bot; +http://my-webby.eu/bot) myWebbyBot/0.1 (Jobs Bot; +http://my-webby.eu/bot)

Add/remove your website

If you want to add your website to our index or if you want to remove it, then notify us please.

Your name:

Your email:

Your website:

Action: Add Remove