Skip to navigation

How do I find worthwhile new articles when checking peer WeChat accounts each day?

To find new articles worth reading in peer WeChat accounts each day, request page one of each account’s article list with wechat_mp.v2.fetch_account_articles every day, and compare the returned articles with your saved records by account, app_msg_id and idx. Within one day’s scan, request the next page with the next_offset that the same scan returned; the next day, start on page one again. wechat_sync.py in step 8 writes the newly discovered articles to new-articles.csv and keeps the comparison state in state.json. Read article bodies only for the selected articles, with wechat_mp.v2.fetch_article_detail_h5. Share the discovered articles, the reason to read each one and the range you checked.

What does a daily check deliver?

For example, you check three pages and find a pricing announcement published today and an event notice from last month that your archive missed. Both articles go in new-articles.csv. In your review sheet, label the pricing announcement “Published today” and the event notice “Older article discovered today.” If the title of a known article changes, open the original article from changed-articles.csv and check what changed. Use the article key to link both CSV files to the source.

If the page budget ends at page three, report the three-page scope and the stop reason. On failure, keep the previous state and reuse the responses already saved. Use state.json for the next comparison, and keep reading decisions in a separate sheet. To schedule the check and notify a group, see the n8n and Feishu workflow.

1. Prepare the existing article table

Keep the account username, content tab, unique article key, title, URL, first-seen time and original response file for each article. Use a verified stable article ID or URL as the article key, not the title.

Call data:wechat_mp.v2.fetch_account_articles with an explicit raw:false. The documented simplified response contains data.biz_username, data.articles[], data.next_offset, and data.is_end (0/1). Each article has app_msg_id, idx, title, digest, url, create_time, and update_time. BeatAPI returns the simplified payload directly under data without remapping fields into another platform’s schema. The default raw=true response nests batch entries; do not feed a raw=true response into wechat_sync.py in step 8.

wechat_sync.py uses an account-scoped app_msg_id:idx key, so secondary articles in the same message stay separate. The table below lists local columns, not API output paths:

Local columnPurpose
article_keyStable article identity; include account identity when merging accounts
article_urlActual article URL for optional body reading
first_seen_atFirst local discovery, not publication time
last_seen_atMost recent observation in a scan
content_tabKeep article and other content-tab pagination separate

2. Inspect and bound the scan

curl -sS https://api.beatapi.io/v1/capabilities/inspect \
-H 'Content-Type: application/json' \
-d '{"reference":"data:wechat_mp.v2.fetch_account_articles"}'

Confirm the current inputs and price with the free Inspect request above. For a first run, limit the scan to one account, the article tab and at most three pages. Three pages is an example limit; three pages are not guaranteed to cover every account. Backfill older articles with the history guide.

3. Start every new scan at page one

Do not continue from the final cursor of an earlier history export to look for new posts. Omit offset in the first request:

curl -sS https://api.beatapi.io/v1/social-data/call \
-H "Authorization: Bearer $BEATAPI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"action":"wechat_mp.v2.fetch_account_articles","params":{"username":"<username>","item_show_type":0,"raw":false}}' \
-o wechat-sync-page-1.json

Give each run its own directory. Keep the original responses, request IDs, timestamps and inputs, and do not overwrite earlier runs. Process records only after checking business success and the documented response mapping; a parsing error does not mean “no new articles.”

4. Compare keys and decide whether to continue

Map the returned articles into the local table and compare account plus article key with the history table. Add unseen keys, and update last_seen_at on existing keys. A changed title on an existing key is a modification, not a newly published article.

For the next page, pass the current run’s actual next_offset unchanged. Do not increment next_offset, and do not reuse next_offset across accounts or tabs. Track the cursors already used; stop and record the reason on a repeated cursor, a business error or the three-page limit.

Do not stop at the first old article. The fetch_account_articles contract does not guarantee strict chronological order, so new articles can appear after old ones. Reaching the scan limit does not mean the list is exhausted. To reduce omissions, widen the controlled lookback, scan more often or check deeper history periodically, and always report the range you actually checked.

5. Read bodies only for selected new articles

Review the list of newly discovered articles first, then call wechat_mp.v2.fetch_article_detail_h5 only for the URLs that need full text. Each detail read is charged separately; ten new articles in the list do not mean you should read all ten bodies.

Reuse saved text for the same article key where appropriate. Detecting edits needs a separate refresh policy: a key seen before does not prove the article content is unchanged. See Read article text.

6. Persist progress and recover before scheduling

Export each validated batch of saved pages into a new directory. If writing the local files fails, replay the saved JSON with key-based deduplication before buying another query. A failed run must not erase the previous successful history table.

Log the page count, new-key count, stop reason and original-response locations. An automated setup needs its own persistence, concurrency control and retry rules. An external scheduler triggers the runs, and writing to Feishu or a database is a separate integration. The steps on this page do not provide a built-in monitoring service.

7. Count calls and verify outcomes

P list pages plus N selected article-body reads take P + N platform queries. Price each action with a current Inspect request; later model analysis is billed separately. The list ignores page_size, so do not use page_size to estimate the record count.

Processing the same local response twice must add no duplicate articles. Test four cases: an old article followed by a new one, failed pages, repeated cursors and scan-limit termination. Failed pages, repeated cursors and scan-limit termination must never report “no updates today.” Check the scheduler and the destination writes separately.

wechat_sync.py in step 8 runs locally with no network calls and validates saved responses against the documented raw:false mapping.

Completeness and missed updates · Page sizes and content tabs · WeChat question index

8. Run the saved-page incremental processor

Pass wechat_sync.py the account’s canonical gh_ username, confirmed earlier from article detail. The API also accepts custom usernames, but wechat_sync.py requires the canonical username so that the response biz_username can be compared directly. Keep old state files unchanged as evidence. On the first run, pass - instead of a prior state file; the first run records only the observed pages, not a complete history. Before you call later additions “new,” import existing rows into the same account/app_msg_id/idx state format, or replay the saved simplified pages of the existing rows.

Save pages.json with the actual parameters, HTTP status, timestamp and relative response filename. The pages.json example below is a placeholder, not a recorded request. To collect pages, run the step 3 curl with --max-time 30 and -w '%{http_code}\n', and record the real status. A transport timeout is not an empty response and does not prove the request was free: keep the attempt and check usage before any retry.

[{"action":"wechat_mp.v2.fetch_account_articles","params":{"username":"gh_example","raw":false,"item_show_type":0},"http_status":200,"collected_at":"2026-11-16T18:00:00Z","response_file":"wechat-sync-page-1.json"}]

Save the code below as wechat_sync.py. Initialize with python3 wechat_sync.py - pages.json scan-001. For the next daily scan, run python3 wechat_sync.py scan-001/state.json today-pages.json scan-002. Each daily scan starts with an empty offset, and each output directory must be new. wechat_sync.py never modifies the old state file, including on validation failure. wechat_sync.py processes at most three pages per scan and does not install a scheduler.

import csv, hashlib, json, pathlib, sys
from datetime import datetime
previous, manifest_path, target = sys.argv[1:4]
manifest_path, target = pathlib.Path(manifest_path), pathlib.Path(target)
entries = json.loads(manifest_path.read_text(encoding="utf-8"))
ACTION = "wechat_mp.v2.fetch_account_articles"
if not isinstance(entries, list) or not 1 <= len(entries) <= 3:
raise ValueError("Supply 1–3 saved pages for this bounded incremental scan")
state = {"version": 1, "username": None, "item_show_type": 0, "articles": []} if previous == "-" else json.loads(pathlib.Path(previous).read_text(encoding="utf-8"))
if state.get("version") != 1 or state.get("item_show_type") != 0 or not isinstance(state.get("articles"), list):
raise ValueError("Expected the article-tab state produced by this helper")
def number(v):
if type(v) is int and v > 9007199254740991:
raise ValueError("Unsafe numeric ID; require an originally string-encoded ID")
if isinstance(v, bool) or not isinstance(v, (int, str)) or not str(v).isdigit() or int(v) < 1:
raise ValueError("Expected positive article ID/position")
return str(int(v))
def stamp(v):
t = datetime.fromisoformat(v.replace("Z", "+00:00"))
if t.tzinfo is None:
raise ValueError("Collection times must include timezone")
return t
def key(row):
return number(row["app_msg_id"]) + ":" + number(row["idx"])
records = {}
for row in state["articles"]:
k = key(row)
if k in records:
raise ValueError("Duplicate article identity in old state")
stamp(row["first_seen_at"]); stamp(row["last_seen_at"])
records[k] = row
initial = set(records)
new, changed, observed, request_ids, cursors = set(), set(), set(), set(), set()
pages, raw_pages, observations = [], [], []
expected, username, last_time = None, state["username"], None
stop = "saved_pages_exhausted"
watermark = stamp(state["last_collected_at"]) if state.get("last_collected_at") else None
processed = state.get("processed_requests", {})
if not isinstance(processed, dict):
raise ValueError("Invalid processed-request ledger")
metadata = ("title", "digest", "url", "create_time", "update_time")
for index, entry in enumerate(entries):
if stop != "saved_pages_exhausted":
raise ValueError("Manifest contains pages after terminal response")
params = entry["params"]
if entry.get("action") != ACTION or params.get("raw") is not False or (type(params.get("item_show_type")) is not int or params["item_show_type"] != 0):
raise ValueError("Use the article tab and raw=false explicitly")
if set(params) - {"username", "raw", "item_show_type", "offset", "page_size"}:
raise ValueError("Unexpected parameters")
account = params["username"]
if not isinstance(account, str) or not account.startswith("gh_"):
raise ValueError("This helper uses the account's verified gh_ username")
if username is None:
username = account
if account != username:
raise ValueError("Mixed account or old state belongs to another account")
if index == 0 and params.get("offset") not in (None, ""):
raise ValueError("Each new scan starts at the first page")
if index > 0 and params.get("offset") != expected:
raise ValueError("Broken offset chain")
current_time = stamp(entry["collected_at"])
if last_time and current_time < last_time:
raise ValueError("Manifest times must be chronological")
last_time = current_time
http = entry["http_status"]
if type(http) is not int or not 100 <= http <= 599:
raise ValueError("Record the actual integer HTTP status")
path = manifest_path.parent / entry["response_file"]
raw = path.read_bytes(); raw_pages.append(raw)
result = json.loads(raw)
page = {"page": index + 1, "params": params, "collected_at": entry["collected_at"], "http_status": http,
"source": str(path), "sha256": hashlib.sha256(raw).hexdigest()}
pages.append(page)
if http != 200:
error = result.get("error", {})
if not isinstance(error, dict) or not isinstance(error.get("code"), str) or not error["code"]:
raise ValueError("Expected gateway error envelope")
page.update(error_code=error["code"], request_id=error.get("request_id"))
stop = "gateway_failed"
continue
if result.get("action") != ACTION or result.get("status") != "succeeded":
raise ValueError("Wrong action or unsuccessful response")
request = result.get("request_id")
if not isinstance(request, str) or not request.strip() or request in request_ids:
raise ValueError("Missing/repeated request ID")
request_ids.add(request); page["request_id"] = request
replayed = request in processed
signature = {"sha256": page["sha256"], "params": params, "collected_at": entry["collected_at"]}
if replayed and processed[request] != signature:
raise ValueError("Processed request was changed")
if not replayed and watermark and current_time < watermark:
raise ValueError("Do not apply older unseen observations")
page["replayed"] = replayed
data = result.get("data")
if not isinstance(data, dict) or data.get("biz_username") != username or not isinstance(data.get("articles"), list):
raise ValueError("Expected simplified article list for the same account")
for article in data["articles"]:
k = key(article)
if not isinstance(article.get("title"), str) or not isinstance(article.get("url"), str) or not article["url"]:
raise ValueError("Article requires title and URL")
for field in metadata:
if article.get(field) is not None and (isinstance(article[field], bool) or not isinstance(article[field], (str, int))):
raise ValueError("Unexpected article field type")
values = {field: article.get(field) for field in metadata}
observations.append({"article_key": k, "page": index + 1, "request_id": request, "article": article})
observed.add(k)
if replayed:
if k not in records:
raise ValueError("Processed request refers to a missing stored article")
continue
prior = records.get(k)
if prior is None:
new.add(k)
elif k in initial and any(prior.get(field) != values[field] for field in metadata):
changed.add(k)
record = {"app_msg_id": number(article["app_msg_id"]), "idx": number(article["idx"]), **values,
"first_seen_at": prior["first_seen_at"] if prior else entry["collected_at"],
"last_seen_at": entry["collected_at"], "last_request_id": request}
records[k] = record
if not replayed:
processed[request] = signature
state["last_collected_at"] = entry["collected_at"]
watermark = current_time
page["returned_objects"] = len(data["articles"])
end, cursor = data.get("is_end"), data.get("next_offset")
if type(end) not in (bool, int) or end not in (0, 1):
stop = "missing_end_flag"
elif end == 1:
stop = "provider_end"
elif not data["articles"]:
stop = "empty_page_before_end"
elif not isinstance(cursor, str) or not cursor:
stop = "missing_cursor"
elif cursor in cursors:
stop = "repeated_cursor"
else:
cursors.add(cursor); expected = cursor
next_request = {"action": ACTION, "params": {"username": username, "raw": False, "item_show_type": 0, "offset": expected}} if stop == "saved_pages_exhausted" and len(entries) < 3 else None
if stop == "saved_pages_exhausted" and len(entries) == 3:
stop = "scan_page_cap"
state.update(username=username, articles=[records[k] for k in sorted(records)], processed_requests=processed)
target.mkdir(exist_ok=False)
def safe(v):
if v is None:
return ""
value = str(v)
return "'" + value if value.startswith(("=", "+", "-", "@", "\t", "\r", "\n")) else value
columns = ["app_msg_id", "idx", *metadata, "first_seen_at", "last_seen_at", "last_request_id"]
for name, keys in [("new-articles.csv", new), ("changed-articles.csv", changed)]:
with (target / name).open("x", encoding="utf-8-sig", newline="") as f:
writer = csv.DictWriter(f, fieldnames=columns); writer.writeheader()
writer.writerows({field: safe(records[k].get(field)) for field in columns} for k in sorted(keys))
(target / "state.json").write_text(json.dumps(state, ensure_ascii=False, indent=2), encoding="utf-8")
with (target / "observations.jsonl").open("x", encoding="utf-8") as f:
for item in observations:
f.write(json.dumps(item, ensure_ascii=False) + "\n")
for index, raw in enumerate(raw_pages):
(target / ("raw-page-%02d.json" % (index + 1))).write_bytes(raw)
report = {"username": username, "new_articles": len(new), "changed_known_articles": len(changed),
"observed_keys": len(observed), "stored_articles": len(records), "stop_reason": stop,
"complete_history": False, "publication_dates_inferred": False, "pages": pages}
(target / "coverage.json").write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
if next_request:
(target / "next-request.json").write_text(json.dumps(next_request, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({"new": len(new), "changed": len(changed), "stop": stop}))

Deliver new-articles.csv for choosing body reads, and changed-articles.csv for known IDs whose list metadata changed. state.json is the input of the next scan. The raw publication and update values are kept separate from the first-seen and last-seen timestamps. IDs stay strings; import the ID columns as text in spreadsheets. Values that start with a formula character are escaped in the CSV files only; the original text stays in the observations and raw pages.

coverage.json records limits, errors and every source hash; observations.jsonl keeps repeated observations and the original objects. An old article followed by a new article is still processed. Two articles with the same app_msg_id but different idx stay separate. Replaying the same saved pages against the resulting output state produces no new IDs. Changed metadata does not show that the full body changed, and first discovery is not publication.

If the same scan needs another page, next-request.json contains the exact next_offset. Run the additional paid request only within budget, append the request’s actual log entry, and re-run the full expanded manifest against the same pre-scan state, not the interim output state, into another new directory. Re-running against the pre-scan state keeps the scan’s set of new items complete. Exact replay against an output state also works: a ledger of request ID, raw-byte hash, params and timestamp skips observations already applied and does not roll metadata back. wechat_sync.py rejects changed evidence for an already-processed request and older unseen requests. When the last page fails, the successful earlier pages are kept and marked gateway_failed; never report zero new items as “no updates anywhere.” A capped scan leaves older pages unchecked. Keep one writer per account: wechat_sync.py is not a concurrent database.

For ten accounts checked one page at a time, budget ten list calls. Add body reads for selected articles separately. Check current BeatAPI pricing before setting your schedule.

Identifier precision prerequisite

Unquoted JSON numbers decoded through float64 can lose precision above 9007199254740991 before wechat_sync.py reads the file. wechat_sync.py rejects any unquoted numeric ID above 9007199254740991, while IDs encoded as strings stay exact. Do not “fix” a rejected numeric ID by wrapping the rounded value in quotes: stop the export and keep request_id for investigation. Python’s large integers and spreadsheet text formatting cannot restore digits already rounded during JSON decoding.

Hand selected articles to a knowledge base

Use the incremental workflow on this page to produce a sourced list of selected links. For a knowledge base that accepts article URLs, import one link first and check the imported title, source and folder. To prepare body files, follow article Markdown export. Record pending, imported and failed links separately, and retry only the failed links from the saved list. Use an import method you have verified in ima or your destination. The incremental result records which articles were discovered, not which were imported.

Import links into ima