跳到导航

想研究一个公众号过去写了什么,怎么先建目录,再挑值得读的文章?

要批量导出一个公众号的历史文章,用已确认的账号 username 和 raw=false 调用 wechat_mp.v2.fetch_account_articles。每页文章列表返回标题、摘要、链接和发布时间;把每页的 next_offset 作为下一页的 offset 传回,再用下文的 wechat_history.py 把保存的响应导成 CSV。只有一篇文章链接时,文章详情能力 data:wechat_mp.v2.fetch_article_detail_h5 会在 data.content.user_name 返回账号标识。

例如要找某个公众号上季度的招聘公告:

  1. 把文章目录查到季度开始。
  2. 按实际发布日期和标题筛出候选文章。
  3. 打开入选原文,核对岗位、截止日期和适用条件。
  4. 交接时附上文章目录、实际查过的日期范围和待核的文章。

只给入选文章补正文读取、阅读统计或评论。

历史回填与每日查新从不同的位置开始。 历史回填中断后,用最后一个已保存页的 next_offset 接着向后翻。明天想看公众号又发了什么,从第一页重新开始,把第一页与已保存的文章键比对。 每日查新

文章目录表保留每页的原始请求号,并用每篇文章的原链接对照已保存的正文记录。每日查新沿用历史回填的同一账号和同一套文章身份,并且总是从第一页开始。

找到正确的公众号标识

文章列表接口需要 username,不是公众号的展示昵称。当前合同接受 gh_…、关联小程序的 gh_…@app,以及账号自定义的微信号。

如果只有一篇文章链接,先检查文章详情能力 data:wechat_mp.v2.fetch_article_detail_h5。文章详情的公开输出合同提供账号 data.content.user_name,可以直接用作 username;文章详情是一次额外的付费调用,不是免费解析步骤。鉴权和调用方式见社媒数据合同。

免费检查能力

curl -sS https://api.beatapi.io/v1/capabilities/inspect \
-H 'Content-Type: application/json' \
-d '{"reference":"data:wechat_mp.v2.fetch_account_articles"}'

先核对 wechat_mp.v2.fetch_account_articles 的 input_schema、readiness 和 pricing。查询第一页时不传 offset,或传空字符串;offset 是翻页游标,不是文章条数或起始序号。

获取一页文章列表

下面是付费执行示例。确认当前价格后,把 <username> 换成目标公众号的标识再运行:

curl -sS https://api.beatapi.io/v1/social-data/call \
-H "Authorization: Bearer $BEATAPI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"action":"wechat_mp.v2.fetch_account_articles","params":{"username":"<username>","raw":false,"item_show_type":0}}' \
-o wechat-articles-page-1.json

先确认 HTTP 状态和执行状态都成功。明确传 raw=false 时,响应返回 data.articles[]、data.biz_username、data.is_end(0/1)和 data.next_offset;每篇文章包含 app_msg_id、idx、title、digest、url、create_time、update_time。默认的 raw=true 返回嵌套的群发对象。已保存页面增量指南实现了 raw=false 简化结构和状态更新。

正确翻页

输入合同规定:下一次请求传入上一响应的 next_offset。next_offset 要原样保存、原样传递,不解码、不自增。每次下一页请求单独计费。

page_size 虽然出现在输入合同里,但当前说明明确指出平台会忽略 page_size,每页实际条数由账号侧结果决定。所以把 page_size 改成 100 不能保证一次取回 100 篇,也不能按“调用次数 × page_size”宣称采集量。

省略 item_show_type 时取文章;视频、音频、图文是另外的内容栏目。做文章同步时保持默认值,不把不同栏目的记录混进同一份历史文章表。

每日增量怎么做

历史采集按下面三步做:

  1. 每页保存原始响应、request_id、账号、内容栏目和页次。
  2. 用真实结果核对过的稳定文章键标识每篇文章。
  3. 遇到重复游标、错误或达到批准的页数上限时停止,并写明已采集的范围。

字段缺失要报错,解析不到字段不等于历史已经结束。

每日查新是另一项任务:重新请求第一页,并与已保存的文章键比较。完整的去重、恢复和调度见增量同步指南。文章列表没有排序保证,所以不采用“遇到一篇旧文章就停”。

给 AI 的任务

按我提供的公众号 username 整理文章目录,先取一页核对账号,再在本次预算和日期范围内继续。保存标题、链接、发布时间、原始响应和 request_id;正文等我选出文章后再取。结束时列出实际检查范围和请求数。

成本和限制

文章列表的每一页和每篇文章详情分别执行、分别计费。先用文章目录筛出目标,再只读取入选文章的正文,避免逐篇调用。文章目录不保证覆盖全部历史,也不提供实时更新。

社媒数据 API · 能力 API

保存可续接的历史批次,再把状态交给日扫描

历史回填使用与增量处理器相同的公开 raw=false 字段,但每个批次接着当前历史游标翻页,不回到第一页。下面的 wechat_history.py 只处理已保存的响应,不执行收费请求,也不重试错误。

  1. 准备 history-pages.json,放 1–20 个按采集时间排列的真实保存页。每项记录 action、params(规范 gh_ 账号、raw=false、item_show_type=0,仅续页时传 offset)、HTTP 状态、采集时间和响应文件名。清单格式与增量指南相同;每个下一页都必须原样传前一页的 data.next_offset。20 页是本地批次上限,不是平台总页数,也不是固定文章数。
  2. 把脚本保存为 wechat_history.py,运行第一批:python3 wechat_history.py - history-pages.json history-001。
  3. wechat_history.py 把下一批的首个请求写进 history-001/next-request.json。在批准的预算内采集下一批页面并记录真实日志,然后运行 python3 wechat_history.py history-001 next-pages.json history-002。
  4. 保留全部批次目录:state.json 是当前文章表,早期原始证据留在早期批次目录里。
import csv, hashlib, json, pathlib, sys
from datetime import datetime
previous_package, manifest_path, target = sys.argv[1:4]
previous = "-" if previous_package == "-" else str(pathlib.Path(previous_package) / "state.json")
restart = len(sys.argv) == 5 and sys.argv[4] == "--restart"
if len(sys.argv) not in (4, 5) or len(sys.argv) == 5 and not restart:
raise ValueError("Optional final argument is --restart")
entries = json.loads(pathlib.Path(manifest_path).read_text(encoding="utf-8"))
if not isinstance(entries, list) or not 1 <= len(entries) <= 20:
raise ValueError("Supply 1–20 saved pages for this history chunk")
tab = entries[0]["params"].get("item_show_type")
if type(tab) is not int or tab not in (0, 5, 7, 8):
raise ValueError("Select one explicit content tab: 0, 5, 7 or 8")
checkpoint = None if previous_package == "-" else json.loads((pathlib.Path(previous_package) / "checkpoint.json").read_text(encoding="utf-8"))
if checkpoint is not None and restart:
checkpoint = {"version": 1, "resume_params": {"username": json.loads(pathlib.Path(previous).read_text(encoding="utf-8"))["username"], "raw": False, "item_show_type": tab, "offset": ""}}
if checkpoint is not None and (checkpoint.get("version") != 1 or not isinstance(checkpoint.get("resume_params"), dict)):
raise ValueError("History checkpoint is terminal or requires investigation; do not issue another page")
manifest_path, target = pathlib.Path(manifest_path), pathlib.Path(target)
ACTION = "wechat_mp.v2.fetch_account_articles"
state = {"version": 1, "username": None, "item_show_type": tab, "articles": []} if previous == "-" else json.loads(pathlib.Path(previous).read_text(encoding="utf-8"))
if state.get("version") != 1 or (type(state.get("item_show_type")) is not int or state["item_show_type"] != tab) or not isinstance(state.get("articles"), list):
raise ValueError("Expected state for this same content tab")
def number(v):
if type(v) is int and v > 9007199254740991:
raise ValueError("Unsafe numeric ID; require an originally string-encoded ID")
if isinstance(v, bool) or not isinstance(v, (int, str)) or not str(v).isdigit() or int(v) < 1:
raise ValueError("Expected positive article ID/position")
return str(int(v))
def stamp(v):
t = datetime.fromisoformat(v.replace("Z", "+00:00"))
if t.tzinfo is None:
raise ValueError("Collection times must include timezone")
return t
def key(row):
return number(row["app_msg_id"]) + ":" + number(row["idx"])
records = {}
for row in state["articles"]:
k = key(row)
if k in records:
raise ValueError("Duplicate article identity in old state")
stamp(row["first_seen_at"]); stamp(row["last_seen_at"])
records[k] = row
initial = set(records)
new, changed, observed, request_ids, cursors = set(), set(), set(), set(), set()
if checkpoint is not None and not restart:
prior_cursors = checkpoint.get("seen_next_offsets", [])
if not isinstance(prior_cursors, list) or any(not isinstance(c, str) or not c for c in prior_cursors):
raise ValueError("Invalid history cursor ledger")
cursors.update(prior_cursors)
resumed_offset = checkpoint["resume_params"].get("offset")
if resumed_offset:
cursors.add(resumed_offset)
pages, raw_pages, observations = [], [], []
expected, username, last_time = None, state["username"], None
stop = "saved_pages_exhausted"
watermark = stamp(state["last_collected_at"]) if state.get("last_collected_at") else None
processed = state.get("processed_requests", {})
if not isinstance(processed, dict):
raise ValueError("Invalid processed-request ledger")
metadata = ("title", "digest", "url", "create_time", "update_time")
for index, entry in enumerate(entries):
if stop != "saved_pages_exhausted":
raise ValueError("Manifest contains pages after terminal response")
params = entry["params"]
if entry.get("action") != ACTION or params.get("raw") is not False or (type(params.get("item_show_type")) is not int or params["item_show_type"] != tab):
raise ValueError("Keep the same explicit content tab and raw=false")
if set(params) - {"username", "raw", "item_show_type", "offset", "page_size"}:
raise ValueError("Unexpected parameters")
account = params["username"]
if not isinstance(account, str) or not account.startswith("gh_"):
raise ValueError("This helper uses the account's verified gh_ username")
if username is None:
username = account
if account != username:
raise ValueError("Mixed account or old state belongs to another account")
if index == 0:
if checkpoint is None and params.get("offset") not in (None, ""):
raise ValueError("The first history chunk starts at the beginning")
if checkpoint is not None:
resume = checkpoint["resume_params"]
if any(params.get(field) != resume.get(field) for field in ("username", "raw", "item_show_type")) or (params.get("offset") or "") != (resume.get("offset") or ""):
raise ValueError("First page does not match the saved history checkpoint")
if index > 0 and params.get("offset") != expected:
raise ValueError("Broken offset chain")
current_time = stamp(entry["collected_at"])
if last_time and current_time < last_time:
raise ValueError("Manifest times must be chronological")
last_time = current_time
http = entry["http_status"]
if type(http) is not int or not 100 <= http <= 599:
raise ValueError("Record the actual integer HTTP status")
path = manifest_path.parent / entry["response_file"]
raw = path.read_bytes(); raw_pages.append(raw)
result = json.loads(raw)
page = {"page": index + 1, "params": params, "collected_at": entry["collected_at"], "http_status": http,
"source": str(path), "sha256": hashlib.sha256(raw).hexdigest()}
pages.append(page)
if http != 200:
error = result.get("error", {})
if not isinstance(error, dict) or not isinstance(error.get("code"), str) or not error["code"]:
raise ValueError("Expected gateway error envelope")
page.update(error_code=error["code"], request_id=error.get("request_id"))
stop = "gateway_failed"
continue
if result.get("action") != ACTION or result.get("status") != "succeeded":
raise ValueError("Wrong action or unsuccessful response")
request = result.get("request_id")
if not isinstance(request, str) or not request.strip() or request in request_ids:
raise ValueError("Missing/repeated request ID")
request_ids.add(request); page["request_id"] = request
replayed = request in processed
signature = {"sha256": page["sha256"], "params": params, "collected_at": entry["collected_at"]}
if replayed and processed[request] != signature:
raise ValueError("Processed request was changed")
if not replayed and watermark and current_time < watermark:
raise ValueError("Do not apply older unseen observations")
page["replayed"] = replayed
data = result.get("data")
if not isinstance(data, dict) or data.get("biz_username") != username or not isinstance(data.get("articles"), list):
raise ValueError("Expected simplified article list for the same account")
for article in data["articles"]:
k = key(article)
if not isinstance(article.get("title"), str) or not isinstance(article.get("url"), str) or not article["url"]:
raise ValueError("Article requires title and URL")
for field in metadata:
if article.get(field) is not None and (isinstance(article[field], bool) or not isinstance(article[field], (str, int))):
raise ValueError("Unexpected article field type")
values = {field: article.get(field) for field in metadata}
observations.append({"article_key": k, "page": index + 1, "request_id": request, "article": article})
observed.add(k)
if replayed:
if k not in records:
raise ValueError("Processed request refers to a missing stored article")
continue
prior = records.get(k)
if prior is None:
new.add(k)
elif k in initial and any(prior.get(field) != values[field] for field in metadata):
changed.add(k)
record = {"app_msg_id": number(article["app_msg_id"]), "idx": number(article["idx"]), **values,
"first_seen_at": prior["first_seen_at"] if prior else entry["collected_at"],
"last_seen_at": entry["collected_at"], "last_request_id": request}
records[k] = record
if not replayed:
processed[request] = signature
state["last_collected_at"] = entry["collected_at"]
watermark = current_time
page["returned_objects"] = len(data["articles"])
end, cursor = data.get("is_end"), data.get("next_offset")
if type(end) not in (bool, int) or end not in (0, 1):
stop = "missing_end_flag"
elif end == 1:
stop = "provider_end"
elif not data["articles"]:
stop = "empty_page_before_end"
elif not isinstance(cursor, str) or not cursor:
stop = "missing_cursor"
elif cursor in cursors:
stop = "repeated_cursor"
else:
cursors.add(cursor); expected = cursor
resume_params = None
if stop == "saved_pages_exhausted":
resume_params = {"username": username, "raw": False, "item_show_type": tab, "offset": expected}
stop = "history_chunk_saved"
elif stop == "gateway_failed":
resume_params = dict(entries[-1]["params"])
next_request = {"action": ACTION, "params": resume_params} if resume_params is not None else None
state.update(username=username, articles=[records[k] for k in sorted(records)], processed_requests=processed)
target.mkdir(exist_ok=False)
def safe(v):
if v is None:
return ""
value = str(v)
return "'" + value if value.startswith(("=", "+", "-", "@", "\t", "\r", "\n")) else value
columns = ["app_msg_id", "idx", *metadata, "first_seen_at", "last_seen_at", "last_request_id"]
for name, keys in [("new-articles.csv", new), ("changed-articles.csv", changed)]:
with (target / name).open("x", encoding="utf-8-sig", newline="") as f:
writer = csv.DictWriter(f, fieldnames=columns); writer.writeheader()
writer.writerows({field: safe(records[k].get(field)) for field in columns} for k in sorted(keys))
(target / "state.json").write_text(json.dumps(state, ensure_ascii=False, indent=2), encoding="utf-8")
with (target / "observations.jsonl").open("x", encoding="utf-8") as f:
for item in observations:
f.write(json.dumps(item, ensure_ascii=False) + "\n")
for index, raw in enumerate(raw_pages):
(target / ("raw-page-%02d.json" % (index + 1))).write_bytes(raw)
report = {"username": username, "requested_tab": tab, "new_articles": len(new), "changed_known_articles": len(changed),
"observed_keys": len(observed), "stored_articles": len(records), "stop_reason": stop,
"complete_history": False, "visible_history_end_observed": stop == "provider_end", "publication_dates_inferred": False, "pages": pages}
(target / "coverage.json").write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
if next_request:
(target / "next-request.json").write_text(json.dumps(next_request, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps({"new": len(new), "changed": len(changed), "stop": stop}))
(target / "checkpoint.json").write_text(json.dumps({"version": 1, "username": username, "requested_tab": tab,
"stop_reason": stop, "resume_params": resume_params, "seen_next_offsets": sorted(cursors), "state_file": "state.json",
"prior_package": previous_package, "warning": "A cursor is not a permanent checkpoint; preserve all chunks and restart from page one if invalid"}, ensure_ascii=False, indent=2), encoding="utf-8")

checkpoint.json 记录下一次请求参数、前一批次、已见游标账本和停止原因。游标账本能拦住游标返回自身和跨批次循环;只有显式传 --restart 才开始新的游标序列。停止原因按下表理解:

停止原因含义下一页请求
history_chunk_saved后面还有内容待查,回填尚未完成。写入 next-request.json
provider_end本轮可见序列到头。provider_end 不证明已访问被删除或私密的历史。不生成
missing_cursor、repeated_cursor、missing_end_flag游标缺失或重复,或 is_end 异常。先调查,不能静默跳页。不生成

导出的 coverage.json 始终不把归档标为“完整历史”。

网关失败时,wechat_history.py 保留前面已经解析成功的页,checkpoint.json 保存失败请求的原参数。保存的参数只是恢复提案,不是自动重试,也不保证没有扣费。再次执行前先查错误与用量,并批准新增支出。超时且没有保存错误体的情况在解析器之外处理,不能编造成功页或空页。

如果平台拒绝旧游标:

  1. 发起一条新的 raw=false 首页请求,保存真实响应与日志。
  2. 运行 python3 wechat_history.py history-002 restart-pages.json history-003 --restart。

重扫保留已存的文章身份和首次发现时间,只重置分页位置;重扫不清空历史,也不再使用被拒绝的游标。已处理请求的精确重放仍然核对哈希;已有文章键的标题变化记为变更文章,与新文章分开记录。

切到日检查时,使用增量指南的 python3 wechat_sync.py history-003/state.json today-pages.json daily-001。日扫描的首个请求省略 offset,不沿用 checkpoint.json。历史覆盖状态与日扫描报告一起保留,首次回填因预算上限停止时尤其要保留。需要正文的新增 URL 单独挑选、单独付费;原始时间值不猜单位,首次发现时间不等于发布时间。

列表预算按计划页数乘以价格页当前列表单价计算。wechat_history.py 每批最多处理 20 个已保存页,不发起取数请求;更长的历史接着跑下一批。每批记录实际范围与停止原因,供下次续批使用。

标识符精度前提

JSON 中不带引号的裸数字按 float64 解码时,超过 9007199254740991 的数字 ID 在 wechat_history.py 读取前就可能已被舍入。wechat_history.py 直接拒绝此类裸数字 ID。响应里本来就是字符串的 ID 保持精确。不要给已经舍入的值加引号假装修复:停止这次导出,并保存 request_id 查证。Python 大整数和表格文本格式都无法恢复解码阶段已经舍入的数字。

同一套续批规则用于一个其他内容栏目

wechat_history.py 接受显式的 item_show_type:0 文章、5 视频、7 音频、8 贴图。wechat_history.py 从第一条请求读取栏目,之后在清单、状态、续批和重扫中固定使用该栏目。每个账号的每个栏目单独保留一条批次目录链;旧的文章栏目状态继续兼容,但不能用来开始视频栏目。日扫脚本 wechat_sync.py 仍只接受文章栏目 0。

为兼容既有流程,输出文件仍叫 new-articles.csv、changed-articles.csv,状态数组仍叫 articles;非零栏目时,输出行保存该栏目返回的作品元信息,不是经典文章正文。原始对象中的 item_show_type、msg_type、cover 等字段留在 observations.jsonl 和原始页面文件里。作品 URL 不能当作视频或音频下载地址,封面数量也不代表正文的全部图片。非零内容栏目在 wechat_history.py 中沿用同样的栏目锁定与续批规则;实际使用媒体栏目时,先对照首份保存响应核对字段结构。

筛选编辑位置子集时,用群发内头条位置筛选,不要按页内顺序或只按 app_msg_id 判断。