Skip to navigation

How do I get the next search page?

To fetch the next X search page, call twitter.web.fetch_search_timeline again with the same keyword and search_type, passing data.next_cursor from the previous response unchanged as params.cursor. Omit cursor on page one, and start a fresh page-one search whenever you change keyword or switch between Top and Latest.

For example, when page one queries a brand name with search_type:"Latest", page two sends the exact same brand name, search_type:"Latest", and page one’s data.next_cursor. See Search and CSV export.

When do I stop, and what can I report?

Read the stop state recorded in coverage.json and act accordingly:

coverage.json stateNext stepWhat you can report
saved_batch_endedThe saved batch reached its last saved page. Send next_params only if budget permits another page.The saved pages were processed; the sample is not exhaustive.
cursor_absentStop and send no further requests.The response returned no continuation cursor. An absent cursor does not prove full historical coverage.
repeated_cursorStop and inspect the raw response files.The cursor repeated, so continuing could loop.
empty_with_cursorStop and diagnose the empty page.The page succeeded with zero posts while still returning a cursor; do not purchase another page automatically.

HTTP errors, business failures, wrong actions, missing arrays, and malformed JSON are failures rather than successful empty pages. Keep the attempt logs and raw response bytes, then follow the empty-result diagnosis. This workflow never retries automatically. If you receive approval to investigate after an empty_with_cursor stop, record that investigation as a new batch instead of appending the request to the stopped cursor chain.

Does a two-page budget mean forty posts?

No. A two-page budget limits the number of requests rather than guaranteeing a fixed post count per page, unique post count, or date range. Page two can overlap with page one, so fewer rows after deduplication does not mean posts were lost.

At the current Inspect rate of $0.003 per request, two successful pages cost an estimated $0.006, excluding retries or extra pages. Stop early on an empty page or an absent cursor instead of spending the remaining budget just to complete two pages.

How do I continue saved JSON without paying for it again?

Export saved responses locally using export_saved_x.py from the search guide; fetching responses and exporting saved responses are separate workflows.

  1. Record each page’s actual parameters, query time, and original response filename in pages.json.
  2. Run export_saved_x.py. export_saved_x.py verifies that every page shares the same keyword and search_type, that each cursor continues the previous page, that request IDs are distinct, and that collection timestamps are ordered, then writes posts.csv, coverage.json, sources.json, and byte-identical copies of the source files without network access.
  3. Inspect stop_reason before fetching another page. Only saved_batch_ended with valid next_params prepares a continuation, and sending that continuation still requires budget approval.
  4. Save the new response, append the page’s actual metadata to pages.json, and export into a new output directory.

Never rerun page-one collection solely to add CSV columns or adjust local parsing.

How should overlaps, timing and evidence appear in the report?

posts.csv retains the first observation of each original tweet_id. Later observations of the same post remain in the raw JSON files, and coverage.json records the duplicate count. Do not overwrite the first observation’s like count with a later value.

State the actual earliest and latest created_at timestamps and every page’s collection time in the deliverable. A two-page sample count is not total mention volume. File hashes identify the saved bytes without proving server authenticity, timestamp accuracy, or completeness.

Contract reference: data:twitter.web.fetch_search_timeline.