After an unsuccessful conference submission, part of my goal to do more presenting, the reviewer feedback questioned the legal implications and ethics of scraping public listings. It said the talk "verges on web scraping which can have legal implications", but also that "the ethics of scraping public listings is something this audience would genuinely want to hear about". I've been thinking about that on and off since. So rather than another post in my Building Clusterflick series, this one is about that wider question, with Clusterflick as the case study.
Currently, Clusterflick scrapes listings for 488 London venues twice a day. I don't have an agreement with any of them, and some of those sites would clearly rather I didn't.
I've been building it long enough that I stopped asking whether that's OK. So I wrote down my gut position and argued it through, looking for the places it didn't feel right. This is where I've landed.
Where I started
My instinct was simple. Once you put something on the internet you've shared it, and you shouldn't expect to control where it goes. That feels especially true of facts: "the train leaves at 9am" isn't something anyone can own, and neither is "this film, at this cinema, at this time". Creative work felt different, because a restaurant review is personal and its author should get to decide where it shows up. Aggregated data, like "80% of people enjoyed their dinner here", sat somewhere in between.
On top of that sat a general rule, close to Mill's harm principle: if what you're doing doesn't harm anyone, you should be allowed to do it. Does the scraping put strain on the site? Does having the data somewhere else cause the site problems?
Clusterflick seemed to pass easily. It collects facts (the film, the venue, the time, the format, accessibility features) and leaves out blurbs, marketing copy and prices. It scrapes gently. And every listing links back to the venue, which is where the booking happens.
Where that was too broad
"Public means shared" does too much work. Putting something online makes it visible, which is different from agreeing to every possible reuse. A shop window is visible to anyone walking past, but I don't think that means you can photograph the display and sell prints of it. A narrower version holds up much better: once something is public, you can't reasonably object to uses that fit the context it was published in. Cinemas publish listings so people find screenings and buy tickets, and an aggregator that sends people to those screenings fits that purpose. Something harvesting the same listings to feed a ticket-reselling bot uses exactly the same data, and most people would see it very differently.
Facts are free, but collections of them can be protected. Copyright doesn't cover individual facts, but in the UK database right can protect a collection of them where someone has made a substantial investment in putting it together. So could a cinema claim its listings that way?
The case that sets the limit is British Horseracing Board v William Hill, decided by the European Court of Justice in 2004. The British Horseracing Board spent a lot maintaining its database of fixtures, runners and riders, and when William Hill used some of that information on its online betting service, the BHB claimed it was infringing its database right. The court disagreed, because the right only protects investment in obtaining and checking data that already exists, not investment in creating it. Drawing up the fixtures was part of organising races, so that spending didn't count. The UK's database right came from the same EU directive and was kept after Brexit.
As I read it, cinema listings are in the same position as those fixtures. They exist because the cinema programmes films, not because someone went out and gathered them. Where the right is more likely to apply is a dataset built by gathering, which is what an aggregator like Clusterflick does.
The better question is whose product the data is. Aggregated data felt like it sat between facts and creative work, but that turned out not to be the useful distinction. "80% liked it" isn't anyone's personal expression, but for a review aggregator the score is the thing it exists to make, so taking it means taking the product. Asking the same question of listings gives a different answer. For a venue, listings advertise the product, which is the screening, so redisplaying them works in the venue's favour. For a ticketing platform like Eventbrite or Dice, listings are much nearer to its product, though it still makes its money when someone books.
Collecting is neutral, use is judged
The idea that pulled this together is that the act of collecting is mostly neutral, and what gets judged is what you do with what you've collected.
Photographing a shop window is fine, while selling prints of the display isn't. Reading a library book is fine, while keeping a copy so you never need the library again isn't. Saving someone's film review in an RSS reader is fine, while republishing it on your own film site isn't. Copyright works roughly the same way, with reading free and redistribution and substitution being where it bites.
The act isn't entirely neutral, because load comes from the collecting itself. Load on its own isn't a harm, though. These sites exist to be visited, and they're built to serve their pages to anyone who asks for them. The harm is load beyond what a site is set up for, and a rate limit is how a site tells you where that line is. That's why rate limits feel legitimate in a way a blanket "no scraping" rule doesn't: a rate limit describes a harm, while a blanket ban describes a preference about who's asking.
The split also explains why sites restrict collecting anyway. A site can only see the act, while what you do with the data afterwards is invisible to it, so it polices what it can see. A blanket ban is often standing in for "no harmful use" because the site has no way to tell the two apart. That doesn't make the ban right, but it does mean a site and a well-behaved scraper might not disagree about much. Both could agree that sending people to screenings is fine and reselling tickets isn't. The site bans everything because it can't tell which scraper it's dealing with.
What is the restriction protecting?
Clusterflick honours rate limits. The BFI run (Southbank, IMAX and festivals) takes around two hours because it goes one page at a time to stay inside them.
It also gets round bot detection on purpose. Most requests are plain and go out with Node's default user agent, node, which is honest about being a script without saying whose. Some sites use the user agent as a cheap check, refusing node but serving the same request when it claims to be a browser, so for those I send a Firefox user agent. Anything that needs a real browser runs Playwright with a stealth plugin, which hides the usual signs of automation. And where a site still serves a bot challenge, it escalates to Camoufox, a Firefox build made to pass those checks.
I used to describe this as complying with the technical restrictions while ignoring the terms: the check asks for a browser, so I send one. That defends what I do on the letter of the check, though, and the check exists to keep automated access out. I'm overriding it.
A better question is what each restriction is there to protect:
- Restrictions aimed at harm, like rate limits, I honour, whether they're in the code or in the terms.
- Restrictions aimed at controlling facts, like a blanket ban on scraping public listings, I override, whether they're in the code or in the terms.
Bot detection on a page of public showtimes falls in the second group.
Checking myself against the rules
To see how this holds up in practice, I went through every place Clusterflick fetches from and recorded what its robots.txt says about the pages I request and what its terms say about automated access, as they stood on 30 October 2026.
There are 101 of them. Two, TMDB and IMDb, offer their data through an official API or download. I read the other 99 from their websites. At 15, what I do goes against the site's own published rules: robots.txt disallows the pages I fetch at four, and the terms ban automated access at 12, with Rotten Tomatoes in both.
The pattern fits the split between harm and control. Ten sites, including Eventbrite, Letterboxd, Vue and Curzon, allow every page I fetch in robots.txt and ban automated access in their terms. The file a crawler is meant to read says yes, and the legal text says no. The objections also cluster at the top. The big platforms and chains forbid scraping, while of the 42 venues I read directly, 15 link no terms from their homepage and 25 have terms that say nothing about automated access.
Two cases are less clear-cut than the counts make them look. Rotten Tomatoes disallows its search page in robots.txt, and that's the page I use to find each film. Disallowing search is common, and it's usually there to stop search engines crawling and indexing endless pages of results. Clusterflick makes one search per film and indexes nothing, so I don't think the rule is aimed at what I do, though it still counts against me by the site's own rules. Cinebur's listings can only be read signed in, so Clusterflick holds an account there. The terms linked when I signed up in the app go to a page that doesn't exist. The only copy I've found is a Google Drive file linked from the footer of its website, and it excludes data mining tools.
Does the benefit change anything?
Bringing the listings together makes things possible that no single venue's site can. You can search every screening in London at once, filter by format or by accessibility features like captioned and relaxed screenings, and follow a film club or festival that runs across several venues in one place.
I think that adds weight to the case for overriding a restriction aimed at control, because the restriction isn't only keeping out a scraper, it's keeping facts away from people who'd use them. It doesn't outweigh harm, though. A public benefit is the justification nearly every scraper offers, and if it could cancel out harm to the source, the rest of this would mean very little.
There's a second benefit in the data being open. Clusterflick publishes it under CC BY 4.0, and other sites, like filmhose.uk and culturecity.london, build on it. Without that, each of them would need its own scrapers fetching the same listings from the same venue sites, so one careful scraper serving several uses puts less load on the sources than several doing the same job. It does stretch the idea that use is what gets judged, because once the data is open, how others use it is out of my hands.
Harm, not preference
Should a venue be able to opt out? My gut says no, not for factual data.
Testing that against examples helped. Train times, opening hours and showtimes are published so people see them, and nobody is hurt by them spreading. The cases where an opt-out does feel right are things like a fact about a person rather than an organisation, such as a home address, or an event whose attendees could be put at risk by wider exposure. Every one of those turns out to be a harm case, where the objection rests on the harm rather than the publisher's preference.
So a request to stop is evidence of harm rather than a veto. I'd take one seriously because it might point at harm I can't see from my side, but "we'd rather you didn't" on its own isn't enough.
The weak spot in any harm-based rule is that the person doing the thing gets to decide what counts as harm. Treating requests as evidence is the answer to that, and it only works if people can reach me. The code and data are public on GitHub, the data licence page sets out what's published and how it can be used, the sources report lists every site I fetch from and how, and the site links to an email address and socials. A venue that finds its listings on Clusterflick can find me. A venue looking at its server logs mostly can't: only one request, a lookup to the Bechdel Test site, names Clusterflick in its user agent. I've accepted that trade-off so far, because identifying the scraper would most likely get it blocked without changing anyone's mind, but putting the sources report together has made it something I want to look at again.
Where I've landed
- Collecting is neutral; use is judged. Redisplaying facts in the context they were published for is fine. Substituting for the source's product or taking its creative work isn't. Clusterflick shows the facts of a screening and sends you to the venue to book.
- Leave the source better off. Attribution, a link to the booking page, and the source stays the authority. Listings are re-scraped fresh on every run, so anything a venue removes or changes is gone from Clusterflick by the next pass. I can't show the traffic it sends, though, because the links aren't tagged and no venue has said anything either way.
- Ask what a restriction protects. Honour those aimed at harm and override those aimed at controlling facts. Clusterflick sticks to rate limits and deliberately gets round bot detection. It goes against 15 of 101 sites' published rules.
- Harm, not preference, is the reason to stop. A request to stop is evidence worth taking seriously, and people need a way to make one. Clusterflick's code, data, sources and contact details are all public.
- Keep your footprint negligible. Clusterflick runs twice a day, goes one page at a time where a site needs it, and publishes its data openly so others don't need to scrape the same sites.
