Crawler
SkiftlabBot.
The crawler behind our price comparison services. Everything you need to recognise it and let it through.
SkiftlabBot visits the online stores we compare and reads publicly available product pages: price, stock status, and the product details shown on the page. The information appears on our price comparison pages, where every store is listed with its price and a link to its own product page.
The crawler is operated by Billod AS (Norwegian business registry no. 924 686 812), the company behind the skiftlab brand. If you see SkiftlabBot in your logs, that is us.
The sites it crawls for
One and the same crawler serves all four of our services. If you sell goods in one of these categories, that is why we are visiting:
How it identifies itself
All crawling identifies itself with this user agent:
Mozilla/5.0 (compatible; SkiftlabBot/1.0; +https://skiftlab.no/bot)Every request we make carries this one user agent, whatever it is fetching. A rule matching the substring SkiftlabBot therefore covers all our traffic.
Letting the crawler through
If your store uses Cloudflare or another WAF and you want to let us through, create a rule that allows user agents containing the substring SkiftlabBot. That covers all our traffic. If you need to restrict the rule to our IP addresses, ask us and we will send them over.
Crawl rate
The crawler keeps its volume low and runs mostly at night, with plenty of margin to what a normal site handles. If you do notice load from us, tell us and we will turn it down.
Would you rather not be visited?
Email us and we will stop crawling your site and remove the store from our services. No firewall rule needed.
Contact
Questions about the crawler, access, or partnerships? Write to hei@skiftlab.no. We usually reply within 24 hours.