This will include /articles/howto-set-a-rss-feed/ and similar links that are not actually RSS feeds and it will hammer those pages time and time again it is not smart enough to realize something that's not actually an RSS feed isn't.īrandwatch does not appear to care about the Robots Exclusion Standard. That being said, it's also not very abusive since it only grabs that it thinks may be some kind of feed. There's no benefit provided by it at all. It's function is to notify big corporations when they are mentioned in an article. This spider identifies itself as magpie-crawler/1.1 (U Linux amd64 en-GB +) and it will fetch anything it thinks may or may not be some kid of RSS feed. No benefit unless you pay for their services. This crawler is basically useless and should be blocked.īLEXBot is just like AhrefsBot, it gathers data for "SEO analysis" for paying customers. It uses the user-agent "Mozilla/5.0 (compatible Barkrowler/0.9 +)" It's a subscription service that will "Thanks to Babbar’s data and metrics you can uncover the strong and weak points of your sites and their competitors.". This crawler is used for a service called "Babbar" which describes itself as "SEO is made easier". Their bot used to identify itself as Attentio/Nutch-0.9-dev Barkrowler They are still around but we have not seen their bot since 2010 so blocking them may not be very important. Their bot used to be hostile and annoying. There is no benefit in having this waste bandwidth unless you are willing to pay for their services - in which case you need to allow it to get the data they collect about your site.Īttentio from Belgium describes themselves as "a corporate intelligence service". This belongs to a company offering SEO analytic services to paying customers. Allowing them may actually hurt you since many use them to setup sites with garbage content carrying the keywords similar to those used on your site in order to gain search engine traffic. These services are not at all useful if you're not one of their customers. They may be useful if you are one paying them and using them. Basically, you can register at these companies and pay them to tell you what web pages are on your website (along with other data). They are all run by different companies who all provide the same class of service: "Research" and "Analysis" to paying clients. The most well-known ones are AhrefsBot, BLEXBot, mj12bot and SemrushBot. These robots are used by closed services which are only available to paying customers. "SEO", advertisement other "research" robots 4.2 htaccess: For Less Friendly Yet Identifiable Bots.4.1 robots.txt: For Worthless But Conforming Bots.1 "SEO", advertisement other "research" robots.
0 kommentar(er)
