How to scrape Reddit and forum threads: what each source hands over and where it stops
Public discussion is the one source nobody hides. No login wall on the thread, no export contract, no sales call, just text any reader can open in a browser. I took a job collecting what people said about a hardware brand across 14 subreddits, three forums running two different engines, and a vendor support site on Discourse. The first pass reported 138,400 comments with a status column that was green the whole way down. It was short of the conversation by about a third.
The gap had nothing to do with errors. Every call came back with status 200, every selector matched, every stored row was accurate as far as it went. These sources answer narrow questions and stay quiet about the width of the answer: a feed replies about its most recent slice, a thread endpoint replies with the part of the tree it feels like sending, an old forum replies about a page offset that shifts under my feet while I walk it. Four shapes, four different edges, and none of the edges announces itself.
So this piece goes source by source. The official API, the JSON the web client reads, the HTML of a forum engine, the archive index. For each one I set down what it hands over, where it stops, and the reading I take to prove the stop is real. Every figure comes from that brand project: 2,860 threads, 30 days of revisits, a pool of 24 addresses.
What a discussion looks like once it lands in a table
Three objects show up in every source. A feed, which is an ordered list of threads. A thread, which carries a title, a body and a score. A comment, which carries a body and a pointer to its parent. That pointer is the whole difficulty, because a comment without its parent is a sentence with the subject removed, and analysis built on such rows produces confident nonsense.
My table has one row per comment and these columns: source, thread_id, comment_id, parent_id, author, body, body_hash, score, depth, captured_at, revision. The parent_id gets written even when the parent has not arrived yet. A nightly query counts rows whose parent is absent, and that orphan count became my first completeness reading. After the first pass I had 4,140 orphans, which meant 4,140 branches I had never asked the source about.
The second reading is the declared count against the stored count. Nearly every source states how many comments a thread holds: Reddit puts num_comments on the thread object, Discourse puts posts_count on the topic, phpBB prints the reply count above the first post. I store that number when I first see the thread and compare it against my own row count at the end. Agreement within 2 percent means the walk finished. My first pass agreed on 41 percent of threads, and on the rest my count sat lower by 8 to 60 percent.
The official API: signed calls with the quota printed on every answer
This is the shape I reach for first, and on Reddit it means an OAuth token from a script app: client id, client secret, a User-Agent string of my own that identifies the collector. The token lives for an hour. The listing endpoints hand over 100 items a call, which is the maximum, and the cursor is the fullname of the last item in the batch, something like t3_1a2b3c, passed back as after.
What it hands over is generous. Sorted feeds for a board, search across a board or across the whole platform with a time window, the full comment tree of one thread, the posting history of one author, and moderation flags on removed items. Everything arrives as typed JSON with stable field names, so the parser stays 20 lines long and stops changing when the site gets a new front end.
Where it stops is printed on every response, and I read it on every call. Three headers carry the quota: calls remaining in the window, calls used, seconds until the window resets. I pace off remaining directly, which removes the guesswork from throttling entirely.
import time, httpx
PROXY = "socks5://user:pass@node.example.net:1080"
UA = "discussion-collector/0.4 by u/my_account"
def listing(client, path, after=None, limit=100):
p = {"limit": limit, "raw_json": 1}
if after:
p["after"] = after
r = client.get(f"https://oauth.reddit.com{path}", params=p, timeout=25)
left = float(r.headers.get("x-ratelimit-remaining", 60))
reset = float(r.headers.get("x-ratelimit-reset", 60))
if left < 5:
time.sleep(reset + 1) # the window tells me how long to hold
d = r.json()["data"]
return d["children"], d.get("after")
def walk(client, path, floor=1000):
seen, after, calls = [], None, 0
while True:
batch, after = listing(client, path, after)
seen += batch
calls += 1
if not after or not batch:
break
if len(seen) >= floor: # the listing ended, the board did not
return seen, calls, True
time.sleep(1.1)
return seen, calls, FalseThat True on the third return value is the flag the whole pass hangs on. A listing that stops at exactly 1,000 items has told me nothing about the size of the board, and treating it as a finished feed is how a collection reports 2,860 threads for a board holding 11,000. When the flag comes up I split the query window and re-queue both halves.
The one thing the API declines to return is text that was removed or deleted. The row survives, the author and the score survive, the body reads [removed]. Getting that wording back is a job for the archive, and I come to it further down.
The public JSON the web client reads
Every one of these platforms ships a front end that talks to its own feed, and that feed answers a signed-out caller. Appending .json to any Reddit thread or listing URL returns the same structure the API returns. Adding raw_json=1 stops the ampersands and quote marks coming back as HTML entities, which saved me a whole unescaping stage. The thread endpoint takes limit, depth, sort and context as query parameters, so the shape of the tree I get back is something I ask for.
Discourse goes further and hands over the map before the territory. A call to /t/{slug}/{id}.json returns the topic with post_stream.stream, an ordered array of every post id in the topic, alongside the first 20 bodies. I know the exact count and the exact order before fetching a single body. The rest arrives through /t/{id}/posts.json?post_ids[]=... in batches of 20 ids.
def discourse_topic(client, base, tid):
head = client.get(f"{base}/t/{tid}.json", timeout=20).json()
ids = head["post_stream"]["stream"] # every post id, in order
have = {p["id"]: p for p in head["post_stream"]["posts"]}
rest = [i for i in ids if i not in have]
for k in range(0, len(rest), 20):
chunk = rest[k:k + 20]
q = "&".join(f"post_ids[]={i}" for i in chunk)
r = client.get(f"{base}/t/{tid}/posts.json?{q}", timeout=20)
for p in r.json()["post_stream"]["posts"]:
have[p["id"]] = p
time.sleep(0.8)
return [have[i] for i in ids if i in have], len(ids)The comparison at the end writes itself: len(collected) against len(ids). On 640 Discourse topics that check caught 11 walks where a batch had timed out and my retry had quietly moved on, and each one was fixed by refetching 20 ids. A source that publishes its own index is a gift, and I take the index as the reference count for the whole thread.
| Source shape | Cursor style | Items a call | Where the answer stops | What proves the stop |
|---|---|---|---|---|
| Signed API listing | item fullname in after | 100 | 1,000 items on any single feed | count of items equals the floor exactly |
| Signed API thread | stub nodes with child ids | 100 per expansion | folded branches past depth 10 | stub queue still holding ids |
| Public JSON thread | limit, depth, sort | 200 | truncation follows the sort order | declared count above stored count |
| Discourse topic | post_ids[] batches | 20 | none, the stream lists every id | stream length equals stored rows |
| Forum HTML page | numeric offset or page number | 25 per page | pages reorder as replies land | post ids repeat across two pages |
| Archive index | timestamp ranges in CDX | 1,000 lines a call | capture coverage is uneven | gaps between capture timestamps |
The signed route and the public feed differ in one property worth planning around: the signed route ties the quota to a token, the public feed ties it to the address. Which one I want depends on the pass. For a wide sweep across many boards I run the signed route from private addresses for collection passes so that a token and an address stay paired for the whole session and the quota headers describe one predictable stream.
Forum engines and the page that moves while you read it
Old forums answer in HTML, and the HTML is friendlier than modern front ends because the markup was written when tables were still fashionable. phpBB pages a topic with viewtopic.php?t=91&start=75, offsets in multiples of the posts per page setting. vBulletin uses showthread.php?t=91&page=4. XenForo puts it in the path: /threads/some-slug.91/page-4. Each post block carries an id attribute in the markup, id="p1284412" on phpBB, and that id is the only stable handle on the page.
The offset is where a walk goes wrong. Numeric offsets index a list that grows from the bottom, so a reply landing while I am on page 4 pushes everything down by one: one post gets read twice, one never gets read at all. Across a 40 page walk of an active support topic I measured 6 repeated posts and 5 that never arrived. Nothing in the response says so, and both numbers land inside the row count I would call healthy.
Anchoring on post ids removes the whole class of problem. I keep the set of ids seen so far, and a page whose ids are already known means the walk has caught its own tail and stops. Pages are requested from the last one backwards, since the tail of a topic is the part that moves. Two engine features shorten the walk further: phpBB serves a whole topic as one document at viewtopic.php?t=91&view=print, and vBulletin accepts a posts per page override that turns 40 pages into 4.
Quoted text is the trap under the parser. A blockquote inside a post repeats another post's wording verbatim, and one popular sentence on that hardware forum appeared in 34 rows before I stripped quote blocks. I drop every blockquote element and every line beginning with the quote marker before hashing, and the edit notice at the foot of a post goes into its own column, since Last edited by kolya on ... carries an edit timestamp no API on that forum exposes.
For forum work I bind one worker to one address for the length of a topic. Session cookies on these engines get issued on the first page view and checked on every later one, so IPv4 addresses pinned one per worker keep the cookie, the referer chain and the reading pace consistent from page 1 to the end of the topic.
The archive index: the source that remembers what was taken down
Archives answer a question the live sources cannot: what did this page say before somebody changed it. The Wayback CDX endpoint returns a line per capture, and it takes filters that turn it into a change log.
https://web.archive.org/cdx/search/cdx
?url=forum.example.org/viewtopic.php*
&output=json
&fl=timestamp,original,digest,statuscode
&filter=statuscode:200
&collapse=digest
&limit=1000The digest field is a hash of the captured body, so collapse=digest drops every capture identical to the one before it. For that hardware forum 2,140 raw captures collapsed to 61 distinct states, and fetching 61 documents gave me the entire visible history of the board's busiest topics. Each line also carries the capture timestamp, which becomes an upper bound on when the wording changed.
Recovering removed text was the payoff. Of the comments that read [removed] in my live rows, 380 had full bodies sitting in an earlier capture, and every one of them was a moderation removal on a thread the client specifically cared about. I write those into the same table with source set to the archive and a flag saying the body came from a snapshot, so nothing downstream mistakes an archived body for a live one.
What the archive stops at is coverage. Captures cluster around popular pages and thin out fast, timestamps record when the crawler visited, and the index answers slowly under pressure. I hold to one call a second from a single exit and pull the CDX lines first, then the documents, since the index call is cheap and the document fetch is not. Archived pages arrive as full HTML with images and scripts referenced inline, so a night of snapshot pulls moves real volume, and addresses with no traffic meter keep the last hours of that pass running at the same rate as the first.
Cursors, offsets, and the floor under every listing
Three pagination styles cover everything above, and each one fails in its own way. An opaque cursor is a token the source hands back, meaningful only to the source, usually bound to the session that received it. An item cursor is the identifier of the last row seen, which survives a restart and can be replayed from the log. A numeric offset counts into a list that keeps changing shape.
| Pagination style | Where I meet it | Survives a restart | Failure mode | What I store to replay |
|---|---|---|---|---|
| Item cursor | signed API listings | yes | none while the item exists | last fullname and item count |
| Opaque token | search endpoints, some feeds | no, expires in about 60 seconds | empty page with status 200 | query, sort and window |
| Numeric offset | forum engines | yes | repeats and skips as replies land | post id set of the last page |
| Explicit id list | Discourse topics | yes | none, the list is complete | topic id and stream length |
The floor sits under all three. Any single listing on the platform stops around 1,000 items, whatever the cursor keeps promising, so a board holding 40,000 threads gives up 1,000 per sort order and no more. Slicing the query is what gets under it. Search accepts a time window, so I walk a busy board a day at a time: 300 to 700 threads a day sits well below the floor, and 540 daily slices covered the whole period the client asked about. Slower boards get monthly slices. Flair and author filters do the same work on boards where search behaves oddly.
The rule I apply is short and it lives in the walker: a listing returning exactly the floor is unfinished, gets logged as unfinished, and its window gets halved and re-queued. Applying it turned 2,860 threads into 9,410 on the second pass, and the extra 6,550 came almost entirely from three high traffic boards where my daily windows had been weekly ones.
Walking a comment tree until the stubs run out
A comment tree never arrives whole. The thread endpoint returns the top of the tree and replaces the rest with stub nodes, each one carrying a list of child ids and a count of how many are hiding underneath. Past depth 10 the folding gets more aggressive, and a deep branch becomes a single pointer that has to be requested as a thread of its own.
Expansion takes up to 100 child ids a call and returns a flat list, which I re-link through parent_id on my side. Each expansion can produce fresh stubs, so this is a queue, not a loop.
def expand_tree(client, link_id, node):
rows, stubs = [], []
stack = [node]
while stack: # first sweep: what came inline
n = stack.pop()
if n["kind"] == "more":
stubs += n["data"]["children"]
else:
rows.append(n["data"])
stack += n["data"].get("replies", {}).get("data", {}).get("children", [])
rounds = 0
while stubs: # second sweep: what was folded
chunk, stubs = stubs[:100], stubs[100:]
r = client.post("https://oauth.reddit.com/api/morechildren",
data={"link_id": link_id, "children": ",".join(chunk),
"api_type": "json", "sort": "old", "raw_json": 1},
timeout=30)
for item in r.json()["json"]["data"]["things"]:
if item["kind"] == "more":
stubs += item["data"]["children"]
else:
rows.append(item["data"])
rounds += 1
time.sleep(1.2)
return rows, roundsOn one 4,800 comment thread that queue took 71 expansion calls and emptied on the fifth round. A walker that expands once and stops would have stored the first 380 comments and reported a finished thread, which is exactly what my first pass did across the board.
Sort order decides whether the walk is repeatable. Sorting by age gives a stable sequence, so two runs an hour apart return the same nodes in the same positions and the diff between them is meaningful. Sorting by score reshuffles between calls, and a stub expanded under one ordering returns children that already arrived under another. Every collection walk I run is sorted by age, and score ordering is a thing I read from the stored scores afterwards.
The second pass on the same 2,860 threads returned 214,900 comments against 138,400. Sorting the difference by cause gave the chart above, and stub nodes alone accounted for 38 percent of everything I had been missing. My completeness reading here is the stub queue itself: a thread whose queue emptied and whose stored count sits within 2 percent of the declared count is finished, and every other thread goes back in line.
The tunnel matters for these walks in a way it does not for single page fetches. An expansion call is bound to the session that received the stub ids, so I pin one thread walk to one exit from the first call to the last through a SOCKS5 endpoint for the whole tree walk, and the share of expansions returning an empty list dropped from 4.6 percent to 0.3 percent once the walk stopped hopping addresses mid-tree.
Folding four sources into one row per comment
Identity inside a platform is settled: the comment id is authoritative and nothing else is needed. My primary key is the pair (source, comment_id), and an upsert on that pair absorbs every retry, every overlapping window and every re-walk at no cost.
Repeated text is the harder half, and it arrives from four directions. Quote blocks reprint a neighbour's wording. Crossposts carry an entire thread into a second board. Some forums run a mirror on a second domain with different ids for the same posts. And my own retries after a timeout collect a page twice.
Normalisation runs before any comparison: lowercase, strip quote blocks and their markers, drop zero width characters and the soft hyphen, collapse whitespace, remove the trailing edit notice, then hash. Equal hashes fold. For near matches I run a 64 bit simhash over word shingles and flag any pair within a Hamming distance of 3.
import re, hashlib, unicodedata
QUOTE = re.compile(r"(?ms)^\s*(>|>).*?$|<blockquote.*?</blockquote>")
EDIT = re.compile(r"(?i)\s*last edited by .*$")
def norm_body(s):
s = unicodedata.normalize("NFKC", s)
s = QUOTE.sub(" ", s)
s = EDIT.sub(" ", s)
s = s.replace("", "").replace("", "")
s = re.sub(r"https?://\S+", " url ", s.lower())
return re.sub(r"\s+", " ", s).strip()
def body_hash(s):
return hashlib.sha1(norm_body(s).encode()).hexdigest()[:16]Across 214,900 rows the hash folded nothing at all inside a single platform, which is the answer I wanted, since it says the comment ids were doing their job. The simhash flagged 2,940 groups. Of those, 2,100 turned out to be quote fragments my stripping had missed on one engine that renders quotes as a styled div with no marker, and the remaining 840 were genuine reposts of the same complaint across three boards. Both findings were useful and only the first was a bug.
Joining a forum thread to a Reddit post about the same subject needs a different key, and the outbound link does the work. When both discussions point at the same article URL, I normalise that URL, strip its tracking parameters, and store it as a topic key on both threads. On this project 214 forum topics and 380 Reddit threads collapsed into 176 shared subjects, and that mapping was what the client actually wanted to read.
Keeping the history of a thread that keeps changing
A discussion is a document that keeps being rewritten. Scores drift, bodies get edited, moderators remove, authors delete their own text, and a thread that looked settled at midnight reads differently by breakfast. Storing one snapshot answers what a thread said once, and the client wanted to know how the conversation moved.
A revision row appears only when the normalised body hash changes, which keeps the table honest and small. Score is different: it moves constantly and carries no text, so I write it into a separate time series with the capture timestamp and leave the revision table for wording.
| Thread age | Revisit interval | Calls per thread in the band | Share of all edits caught |
|---|---|---|---|
| 0 to 6 hours | 15 minutes | 24 | 61 percent |
| 6 to 48 hours | 1 hour | 42 | 22 percent |
| 2 to 14 days | 24 hours | 12 | 14 percent |
| day 30 | one closing capture | 1 | 3 percent |
Across 30 days on 9,410 threads, 8.4 percent of comments changed at least once, 1.9 percent were removed by a moderator, and 0.4 percent were edited more than three times. Those last ones are worth watching by hand, since a body edited four times on a product thread is usually somebody walking back a claim, and the diff is the story.
Deletion is recorded as an event with the prior text kept. When a body comes back as [deleted] or [removed] I write a revision carrying the marker, a flag naming which of the two arrived, and the timestamp, while the previous revision keeps the wording. The row count never drops, so a chart of comments over time stays truthful even as the live thread loses text.
Revisits are the part of the job that runs for weeks, and address stability decides whether the diffs mean anything. A body hash that changes because a different exit got served a slightly different page is a false revision, and I chase those for hours before finding they were mine. Holding addresses reserved for a month of revisits keeps the same exit on the same board across the whole window, and false revisions on my last run sat at 3 rows out of 18,000.
What every call writes down, and the pool behind it
One line per call, written whether the call succeeded or failed, holding enough to replay the call from nothing.
{"src":"reddit","kind":"morechildren","thread":"t3_1a2b3c",
"cursor_in":"m_4471,m_4472","cursor_out":null,"sort":"old",
"items":97,"first_seen":91,"stubs_out":12,"declared":4812,"stored":4779,
"quota_left":41.0,"status":200,"exit":"a11","retries":0,
"started":"02:41:07","ms":1840}Five fields there earn their space. first_seen says whether the call brought anything new, and a run of calls reporting zero is a walk that has finished without knowing it. stubs_out drives the expansion queue, so a pass generates its own next pass. declared against stored is the completeness reading for the thread, sitting on the same line where I can see both. quota_left is what the pacing reads back. And exit answers the question that always arrives eventually, which is whether an odd result came from the source or from one address behaving unlike the other 23.
The pool arithmetic follows from the log. My pass ran 61,000 calls across the four source types in a 9 hour window, which is 113 calls a minute. Measured safe rates differed per source: 9 calls a minute per address against the forums, 14 against the archive index, and the signed API paced off its own quota headers at roughly 30. Dividing through gave 16 addresses at the minimum, and I ran 24, because expansion queues grow mid pass and retries take slots that were never in the plan.
I run these passes on a datacenter pool for long collection passes, since the arithmetic above only holds when the per address rate is a property I set. Addresses spread across separate subnets matter here too, because forum engines count per network as readily as per address, and private addresses with no shared history start a pass without carrying whatever a previous tenant did to the same board last week. Assignment is by hashing the thread id modulo the pool size, so one address never sweeps a single board in a straight line, and no address may carry above 10 percent of the pass.
Thirty days of these logs turned into a table of per source floors, per source safe rates and per engine pagination quirks that I now carry into every new discussion project. The second brand took four days against the three weeks the first one cost. The declared count, the stub queue and the body hash are the three readings that decide whether a discussion collection is finished, and once the log carries all three per call, a pass that looks complete and a pass that is complete stop being the same green status column.
Two neighbouring pieces cover the parts I moved through quickly here: building a scraper in Python that finishes its pass works through the request layer command by command, and reading a 403 before changing anything sorts the refusal codes these sources return. If you are sizing a pool for a discussion pass, start from the call arithmetic above and take server addresses sized for the thread queue with the expansion headroom already counted in.