I run a small AI/automation consultancy in Brazil, and a recent lead-research project needed the full follower list of a public Instagram profile — about 153,000 followers — plus enrichment (bio, public email/phone) to find business accounts worth contacting. The problem Pulling a list that size is never one API call. Instagram reports ~153,628 followers; you get them page by page, and any long-running extraction WILL hit a failed request eventually. If your pipeline can't resume, you start over from zero — which is expensive and slow. What I built The pipeline runs on n8n with Supabase as the datastore: Batched extraction — followers are downloaded in batches of up to 10,000 per cycle, on a schedule, instead of one giant run. Resume on error — every page cursor and count is persisted. When a request fails mid-run (in one run it stopped at 4,782 followers after 96 pages read), the job logs the error, emails me a status report, and picks up from the same point on the next cycle instead of restarting. Enrichment pass — a second workflow walks the stored followers and pulls profile details, flagging commercial accounts and any public email/phone in the bio. Personal/private accounts return no contact data, which the report counts separately. Email reports — each cycle sends me a summary: profile, followers reported vs. downloaded, pages read, batch name, and the exact error if one occurred. For the Instagram data layer I used HikerAPI — I tested a few other options first, and it won on pricing and rate limits for this volume. It handled the pagination fine: the run above made 100+ requests without me managing sessions or proxies myself. Tradeoffs / what didn't go perfectly Long extractions still fail sometimes (timeouts); resume logic is not optional at this scale, whatever API you use. Early days for me on this stack: so far it has worked well, but I'm still collecting more data before I'd call the pipeline battle-tested. I'll know more after a few full 150k-follower cycles. Enrichment yield is modest: most personal/private accounts expose no public contact data, so plan conversion expectations accordingly. Takeaway Treat large social extractions as a resumable batch job, not a script: persist cursors, cap batch sizes, and report every cycle. The datastore + scheduler matter more than the scraping API itself. Disclosure: I'm part of HikerAPI's user-rewards program — they credit my account for posts like this. Sharing my actual experience.