Crawler
CIQRA-SiteImport
If you saw CIQRA-SiteImport in your server logs, one of our merchants asked us to read their own site so they can move it to CIQRA. This page says exactly what it does.
Who we are
User agent
CIQRA-SiteImport/1.0 (+https://www.ciqra.com/bot)
Operated by
CIQRA OÜ, the company behind CIQRA Sites and CIQRA Commerce. It is not a search engine and it does not build an index.
When it runs
Only when a merchant pastes the address of their own site into CIQRA and confirms that the site is theirs or that they are allowed to migrate it. It never crawls on its own, and it fetches one site at most once an hour across the whole of CIQRA.
What it fetches
The home page and the sitemap
It reads your home page and your sitemap, and looks at no more than 500 addresses listed there.
A small sample of pages
At most 40 pages in one run, none larger than 2 MB, to read your colours, fonts, menu, a few products and a few posts.
A few images
At most 25 images, such as your logo, favicon, hero and product pictures.
Public addresses only
It follows only public web addresses and never reaches into private networks.
How it behaves
It honours robots.txt
It reads your robots.txt before it fetches anything and obeys the rules written for CIQRA-SiteImport.
It honours Crawl-delay
If you set a Crawl-delay it waits at least that long between requests, and never less than half a second.
How to block it
Add two lines to robots.txt
Add the line User-agent: CIQRA-SiteImport and, directly below it on its own line, Disallow: / . From then on the crawler finds nothing it is allowed to fetch on your site.
Ask us to stop
If it reached your site and you did not expect it, write to info@ciqra.com with your domain and we will add it to our block list.
Questions about the crawler?
Write to info@ciqra.com or use the contact form and we will answer.