Crawler

CIQRA-SiteImport

If you saw CIQRA-SiteImport in your server logs, one of our merchants asked us to read their own site so they can move it to CIQRA. This page says exactly what it does.

Who we are

  1. User agent

    CIQRA-SiteImport/1.0 (+https://www.ciqra.com/bot)

  2. Operated by

    CIQRA OÜ, the company behind CIQRA Sites and CIQRA Commerce. It is not a search engine and it does not build an index.

  3. When it runs

    Only when a merchant pastes the address of their own site into CIQRA and confirms that the site is theirs or that they are allowed to migrate it. It never crawls on its own, and it fetches one site at most once an hour across the whole of CIQRA.

What it fetches

The home page and the sitemap

It reads your home page and your sitemap, and looks at no more than 500 addresses listed there.

A small sample of pages

At most 40 pages in one run, none larger than 2 MB, to read your colours, fonts, menu, a few products and a few posts.

A few images

At most 25 images, such as your logo, favicon, hero and product pictures.

Public addresses only

It follows only public web addresses and never reaches into private networks.

How it behaves

  1. It honours robots.txt

    It reads your robots.txt before it fetches anything and obeys the rules written for CIQRA-SiteImport.

  2. It honours Crawl-delay

    If you set a Crawl-delay it waits at least that long between requests, and never less than half a second.

How to block it

Add two lines to robots.txt

Add the line User-agent: CIQRA-SiteImport and, directly below it on its own line, Disallow: / . From then on the crawler finds nothing it is allowed to fetch on your site.

Ask us to stop

If it reached your site and you did not expect it, write to info@ciqra.com with your domain and we will add it to our block list.

Questions about the crawler?

Write to info@ciqra.com or use the contact form and we will answer.