Collecting Web Data for Your App Without Getting Blocked

robots.txt and the terms of service are the site telling you what it wants. Read both before your first request, not after your first block. The Robots Exclusion Protocol became a standards-track document in September 2022 as RFC 9309, authored by Martijn Koster with Gary Illyes, Henner Zeller and Lizzi Sassman of Google. It defines how user-agent matching works, how allow and disallow rules resolve by longest match, and two rules worth knowing: crawlers should not use a cached robots.txt for more than 24 hours, and if robots.txt is unreachable because of a server error, a crawler should assume complete disallow. Crawl-delay is not part of that standard. Google's robots.txt documentation states it supports user-agent, allow, disallow and sitemap, and that fields such as crawl-delay are not supported. Its absence is not permission to go fast. Pick your own delay. Set a real User-Agent string with your project name and a contact address. If you are causing a problem, you want the site operator to email you rather than ban your IP range. The direction of travel is towards explicit permission. On 1 July 2025 Cloudflare announced it would block AI crawlers by default for new domains and launched a Pay Per Crawl beta, letting publishers charge for access.

Two bodies of Australian law matter most when you collect data from the web: copyright and the Privacy Act 1988. This is general information, not legal advice. If money or personal data is involved, talk to a lawyer. On copyright, facts are treated differently from the way facts are expressed, and Australian courts have set a high bar for copyright in factual compilations. In Nine Network v IceTV [2009] HCA 14, the High Court unanimously held that reproducing time and title information from television schedules did not infringe. In Telstra v Phone Directories [2010] FCA 44, the Federal Court found copyright did not subsist in the White and Yellow Pages. A price or a nutrition figure sits at a very different risk level from a reviewer's paragraph or a product photo. Do not assume an AI carve-out exists. On 26 October 2025, Attorney-General Michelle Rowland ruled out a text and data mining exception to the Copyright Act 1968, and said the government would consult on licensing instead. On privacy, publicly visible does not mean unregulated. In a determination dated 3 November 2021, then Information Commissioner Angelene Falk found Clearview AI breached the Australian Privacy Principles by scraping facial images, calling the covert collection "unreasonably intrusive and unfair". On 29 October 2024 the OAIC and sixteen international counterparts issued a joint statement confirming that publicly accessible personal data remains subject to privacy law. The cheapest way to stay clear of most of this is to not collect personal information at all. Product names and prices are not personal information. Reviewer names, profile photos and user handles are.

The fastest request is the one you never make. Cache every response you get, and never fetch the same page twice in the same day. Another buildDay registrant put the problem exactly: "I was hitting scraping rate limits and wanted to know how to maximize throughput while minimizing cost (ie prefer not to use a residential proxy)." The answer is rarely more capacity. It is fewer requests. Store the raw response body keyed by URL, with a fetched_at timestamp, before you parse anything. When your parser has a bug, and it will, you fix it and re-run over the cache. Rewriting a parser is free. Refetching 40,000 pages is not. Use conditional requests. RFC 9110, published June 2022, defines If-None-Match and If-Modified-Since as precondition fields, and 304 Not Modified as the response when nothing has changed. Save the ETag and Last-Modified values you receive and send them back next time. An unchanged page then costs you a tiny response with no body to parse. Then tier your refresh rates instead of treating every page the same: - Prices and stock levels: daily - Category and listing pages: weekly, to discover what is new - Product descriptions, ingredients and images: monthly, or only when a hash of the page changes Crawl the cheap listing pages to work out what changed, then spend your request budget only on those detail pages. On most catalogues that turns a daily full crawl into a crawl of a few per cent of the site.

Rate limit yourself before the server does it for you. One request every second or two from a single worker will out-collect ten parallel workers that get blocked in the first minute. Status code 429 was defined in RFC 6585, published April 2012 by Mark Nottingham and Roy Fielding, and means "the user has sent too many requests in a given amount of time". That RFC also notes the response may include a Retry-After header telling you how long to wait. Honour it exactly. If a server tells you 3600 seconds, waiting 3600 seconds is the whole answer. When you do have to retry, back off exponentially and add randomness. Marc Brooker's post on the AWS Architecture Blog, dated 4 March 2015, tested this directly and found plain exponential backoff was "the clear loser", while adding jitter "reduced our call count by more than half" with 100 contending clients. Without jitter, everything that failed together retries together. Two habits worth building in from the start: - Put the limiter in one place, as a single token bucket the whole job draws from, rather than a per-worker delay that multiplies as you add workers - Treat a 429 or a 503 as a standing instruction, not a blip. Halve your rate and leave it halved for the rest of the run A scraper that slows down when a site is struggling gets to keep running tomorrow.

A daily job has 24 hours to finish. Once you accept that, most rate limits stop being a problem at all. Run the numbers. Forty thousand product pages at one request every two seconds is about 22 hours of wall clock on a single worker. At one per second it is about 11 hours. Both fit inside a daily cycle, and neither needs a proxy pool. The urgency was self-imposed. This is also where the cost argument lands. Residential proxy services bill by the gigabyte, so every page you do not fetch is money you do not spend. Patience and caching reduce the bill and the block rate at the same time. Build it as a queue rather than a script. A table with url, status, attempts and next_attempt_at, plus a cron worker that leases a small batch each run, gives you a job that is resumable and idempotent. If it dies at 60 per cent, it restarts at 60 per cent. If a page fails three times, it gets parked instead of hammered. Run it off-peak in the site's local timezone. If the job genuinely cannot fit in a day at a polite rate, narrow the scope or email the site and ask for access. Plenty of operators will hand you a feed if you explain what you are building. If you want help applying this to a real project, that is the kind of thing we work on in buildAcademy.

Is web scraping legal in Australia? There is no single law banning it, but several apply at once: copyright, the Privacy Act 1988, contract law through terms of service, and Australian Consumer Law if you republish. Collecting public prices sits very differently to collecting personal information. This is general information, not legal advice. Does ignoring robots.txt actually matter? robots.txt is not itself a law, but ignoring it removes your best defence that you acted reasonably, and it usually breaches the site's terms of service, which is a contract question. Practically it also gets you blocked faster. Treat it as binding. How do I find out if a site has an API? Check the footer for a developer link, then open your browser network tab and reload the page. Many sites render from a JSON endpoint you can call directly. Also check sitemap.xml, any RSS feed, and whether an open dataset already covers the same ground. What request rate should I start at? Start at one request every two seconds from one worker, with a single shared limiter, and watch for 429 and 503 responses. If you see none across a full run, you can consider going slightly faster. If you see any, halve the rate and leave it there. Do I need a proxy service to scrape at scale? Usually not, and reaching for one first hides the real problem. If you are blocked at a polite rate, the site is telling you something. Fix caching, spread the job over hours, and ask the operator for access before you spend money on infrastructure.