SEC EDGAR already publishes everything you need for free: XBRL "company facts" for the numbers, plus the filings themselves. The annoying part is the plumbing. Ticker-to-CIK lookup, choosing among competing US-GAAP tags for "revenue", de-duplicating restated periods, and cutting Item 1A out of a 10-K's HTML without grabbing the table of contents.

This post uses a small tool I built that does that plumbing and returns JSON: SEC EDGAR Financials, Filings & 10-K Sections on the Apify Store. If you'd rather self-host, EdgarTools is a solid open-source Python library. The patterns below carry over.

Disclosure: I built this tool. Data comes only from SEC EDGAR's public APIs, and it is not investment advice.

The five modes

Unknown companies, missing sections, failed SEC requests and searches with no hits are not charged. The query field takes a ticker (AAPL), a company name (Microsoft) or a CIK.

Setup

pip install apify-client
export APIFY_TOKEN=apify_api_xxx   # Apify Console → Settings → API & Integrations

One call: five years of standardised financials

curl -X POST \
  "https://api.apify.com/v2/acts/poised_beardlichen~sec-edgar-financials/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"query": "NVDA", "mode": "financials", "period": "annual", "periods": 5}'

The response is one item per company, with a periods array:

{
  "name": "WALMART INC.", "cik": "0000104169", "ticker": "WMT", "period": "annual", "currency": "USD",
  "periods": [{
    "periodEnd": "2026-01-31", "fiscalYear": 2026, "form": "10-K",
    "revenue": 713163000000, "netIncome": 21893000000, "epsDiluted": 2.73,
    "operatingCashFlow": 41565000000, "capitalExpenditure": 26642000000, "freeCashFlow": 14923000000,
    "grossMargin": 0.2493, "netMargin": 0.0307
  }],
  "source": "https://data.sec.gov/api/xbrl/companyfacts/CIK0000104169.json"
}

(That example is from the tool's README.) Each metric is mapped from the most common US-GAAP tags in priority order, and the latest filing for each period wins, so restatements are picked up. Margins and free cash flow are worked out for you. Every item carries a source URL, so the agent can cite where the numbers came from.

Two caveats worth passing on to your users. Quarterly figures come from 10-Qs, so Q4 isn't reported on its own; derive it from the annual figure. And funds, trusts and some foreign filers have no standardised XBRL financials. They come back with an error field rather than numbers, and aren't charged.

A small screen across several companies (Python)

queries batches several companies into one run in the same mode:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("poised_beardlichen/sec-edgar-financials").call(run_input={
    "queries": ["AAPL", "MSFT", "GOOGL", "AMZN", "META"],
    "mode": "financials",
    "period": "annual",
    "periods": 3,
})

for co in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "error" in co:
        print(co["query"], "->", co["error"]); continue
    latest = co["periods"][0]
    print(f'{co["ticker"]:6} FY{latest["fiscalYear"]}  '
          f'rev ${latest["revenue"]/1e9:,.1f}B  '
          f'net margin {latest["netMargin"]:.1%}  '
          f'FCF ${(latest.get("freeCashFlow") or 0)/1e9:,.1f}B')

Five companies in financials mode cost 5 × \(0.004 = \)0.02.

Just the risk factors, not the whole 10-K

run = client.actor("poised_beardlichen/sec-edgar-financials").call(run_input={
    "query": "TSLA",
    "mode": "section",
    "section": "risk_factors",     # also: mdna, business, market_risk, legal_proceedings, cybersecurity, ...
    "filingForm": "10-K",
})
item = next(client.dataset(run["defaultDatasetId"]).iterate_items())
print(item["form"], item["filingDate"], item["words"], "words")
print(item["documentUrl"])
print(item["text"][:1000])

You get text, plus chars, words, truncated, accessionNumber, filingDate and documentUrl. The table of contents is skipped automatically. maxSectionChars (default 200,000) caps very long sections, and truncated tells you when that happened. If a company puts the item in an exhibit or uses unusual headings, you get an explanatory error instead, at no charge.

Year-over-year risk factor diff

This is the use case I built section mode for: what did management add to the risk factors this year? Pull the last two 10-K accession numbers, fetch Item 1A from each, and diff by paragraph:

import difflib, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
ACTOR = "poised_beardlichen/sec-edgar-financials"

def run(inp):
    r = client.actor(ACTOR).call(run_input=inp)
    return list(client.dataset(r["defaultDatasetId"]).iterate_items())

ticker = "AAPL"
filings = run({"query": ticker, "mode": "filings", "formTypes": ["10-K"], "limit": 2})
new_acc, old_acc = filings[0]["accessionNumber"], filings[1]["accessionNumber"]

def risk_paragraphs(acc):
    item = run({"query": ticker, "mode": "section", "section": "risk_factors",
                "filingForm": "10-K", "accessionNumber": acc})[0]
    return [p.strip() for p in item["text"].split("\n\n") if len(p.strip()) > 80]

new, old = risk_paragraphs(new_acc), risk_paragraphs(old_acc)
old_set = set(old)

added = [p for p in new if p not in old_set
         and max((difflib.SequenceMatcher(None, p, o).ratio() for o in old), default=0) < 0.8]

print(f"{len(added)} new or substantially rewritten risk paragraphs in {filings[0]['filingDate']}:")
for p in added:
    print("-", p[:300], "...\n")

The cost is 2 filing rows plus 2 sections: 2 × \(0.0005 + 2 × \)0.008 = $0.017. The script compares every new paragraph against every old one, which is fine for one company. For a large batch, swap in embeddings or TF-IDF. Then hand only the added paragraphs to the LLM with a prompt like "Classify each new risk (regulatory, supply chain, AI, litigation, other) and quote the sentence that justifies the class." That's a few thousand tokens instead of two full 10-Ks, and every claim maps back to a paragraph you can show the user.

Full-text search across all filers

{ "query": "\"going concern\" lithium", "mode": "search", "formTypes": ["10-Q"], "dateFrom": "2026-01-01", "limit": 25 }

This returns matching filings across every company, including ones you didn't know to ask about. It's useful for thematic research ("who mentioned tariffs in an 8-K this quarter?").

Using it from an agent (MCP)

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=poised_beardlichen/sec-edgar-financials",
      "headers": { "Authorization": "Bearer <YOUR_APIFY_TOKEN>" }
    }
  }
}

Typical calls finish in 1–3 seconds, so results usually come back inline. Prompts that work:

• "Compare NVDA and AMD revenue and net margin over the last 5 fiscal years. Cite the source URL for each company."
• "Get Microsoft's latest 10-K risk factors and list anything about AI regulation, with quotes."
• "Find 8-Ks filed this month that mention 'material weakness'."

Agents can also pay per call through Apify's agentic payments, without an Apify account, so a hosted agent can use it without you handing out a token.

Rules worth putting in the system prompt:

1. Numbers come from financials, never from prose. Have the model compute ratios from the returned fields instead of quoting figures from MD&A text.
2. Always cite source or documentUrl.
3. Say "not reported" when a field is null. Don't fill gaps.

Being a good EDGAR citizen

The tool sends a declared User-Agent, throttles to half of the SEC's fair-access limit, and caches within each run. If you build your own client instead, do the same. The SEC asks for a descriptive User-Agent with contact details, and caps automated access at 10 requests per second.

Wrap-up

For most "ask questions about public companies" agents, three calls cover the bulk of what users ask: financials for numbers, section for the narrative, and filings or search for what changed. Keep the model's context to exactly those pieces and the answers get cheaper and easier to check.

Spotted a wrong number or a section it couldn't find? Open an issue on the tool's Issues tab with the ticker and filing. That's exactly the feedback I need.

A smaller pattern works better for a lot of tasks: give the agent a handful of single-purpose tools, each taking a URL and returning JSON, at a fixed price per call. This post wires up that toolkit through the Apify MCP server:

Disclosure: I built and publish these tools. Failed or blocked captures and failed audits are free. Everything below also works through the plain REST API if you don't use MCP.

1. Connect the tools

Add one MCP server entry that loads only the tools you want, comma-separated, in the tools parameter. Loading a small, explicit set keeps the agent's tool list short and its behaviour predictable:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=poised_beardlichen/website-screenshot-pdf-agent,poised_beardlichen/pagespeed-lighthouse-checker,poised_beardlichen/website-legitimacy-check,poised_beardlichen/tech-stack-detector,get-actor-run,get-dataset-items",
      "headers": { "Authorization": "Bearer <YOUR_APIFY_TOKEN>" }
    }
  }
}

That's the format for Claude Desktop, Cursor and VS Code. If your client supports OAuth, you can drop the headers block and sign in on first use instead. The Apify CLI can also write the config for you (apify mcp install cursor).

I added get-actor-run and get-dataset-items explicitly because of the next section.

2. The async gotcha (read this before you blame the tool)

The MCP server waits only about 30–45 seconds for a run to finish. A single default screenshot usually takes 10–20 seconds, so it comes back inline. A full-page PDF of a heavy site, or a Lighthouse audit, can take longer. In that case the tool call returns a run ID instead of results.

Agents often read that as a failure and retry, which costs you twice. Put a line like this in your system prompt or rules file:

If an Apify tool returns a run ID instead of results, call get-actor-run with that ID until the status is SUCCEEDED, then call get-dataset-items with its defaultDatasetId. Do not re-run the tool.

The same pattern in plain HTTP, for scripts:

# 1. start
RUN=$(curl -s -X POST "https://api.apify.com/v2/acts/poised_beardlichen~website-screenshot-pdf-agent/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "pdf"}')
RUN_ID=$(echo "$RUN" | jq -r .data.id); DS=$(echo "$RUN" | jq -r .data.defaultDatasetId)

# 2. wait (repeat until status is SUCCEEDED)
curl -s "https://api.apify.com/v2/actor-runs/$RUN_ID?waitForFinish=60" \
  -H "Authorization: Bearer $APIFY_TOKEN" | jq -r .data.status

# 3. fetch
curl -s "https://api.apify.com/v2/datasets/$DS/items?clean=true" -H "Authorization: Bearer $APIFY_TOKEN"

3. Screenshots and PDFs the agent can actually use

The capture tool returns one item per URL and device, with a public screenshotUrl that the agent can hand to a vision model, attach to a ticket or store:

{
  "url": "https://www.bbc.com",
  "device": "mobile",
  "format": "jpeg",
  "fullPage": true,
  "success": true,
  "finalUrl": "https://www.bbc.com/",
  "status": 200,
  "screenshotUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/screenshot-...jpg",
  "width": 1170,
  "height": 24180,
  "error": null
}

Prompts that work well:

• "Take a full-page mobile screenshot of https://example.com/pricing and tell me whether the plan table is readable without horizontal scrolling."
• "Save https://example.com/terms as an A4 PDF and give me the link. I need a dated copy."
• "Screenshot only the #checkout element on desktop and on mobile." (the tool takes a CSS selector)

What the tool does for you that a raw headless browser doesn't:

• Cookie banners and consent pop-ups hidden (OneTrust, Cookiebot, Didomi, Quantcast, Usercentrics and more), plus ads, trackers and chat bubbles. That matters because a consent wall covering 40% of the page wrecks vision-model answers.
• Lazy-loaded images handled by auto-scrolling before a full-page capture.
• Blocked pages flagged. A captcha or 403 comes back as success: false or blocked: true instead of a picture of the block page, and it isn't charged.

Limits, so your agent doesn't promise too much: it captures public pages only. It doesn't bypass logins or paywalls, and some heavily protected sites will still refuse (those are flagged, not billed). Very long pages are cut at 16,000 px by default.

There are two versions of the screenshot tool. website-screenshot-pdf-agent is the lightweight one for agent calls (url in, item out). website-screenshot-pdf is the bulk version: thousands of URLs per run, schedules, and a real-time Standby HTTP endpoint. Prices are the same.

4. Page speed and Core Web Vitals on request

{ "urls": ["https://example.com", "https://example.com/blog"], "strategies": ["mobile", "desktop"] }

Each result has performance, accessibility, bestPractices and seo scores, lab metrics (lcpMs, cls, tbtMs, fcpMs, ttfbMs), and opportunities ranked by estimated savings ("Reduce unused JavaScript, est. savings 1,359 KiB").

Two cost notes the agent should know:

• Calls that leave out strategies audit mobile only, which is one audit and the fastest option.
• With your own free Google PageSpeed Insights API key (apiKey), an audit costs $0.005 instead of $0.05, and you also get real-user CrUX field data (p75 LCP, INP, CLS) when Google has it for the page.

Good prompt: "Audit our five landing pages on mobile and list the top three fixes that appear on more than one page."

5. "Should I trust this site?" before the agent clicks, pays or emails

Agents follow links far more obediently than people do. A cheap pre-check before any irreversible action helps:

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("poised_beardlichen/website-legitimacy-check").call(run_input={"url": "example-shop.com"})
r = next(client.dataset(run["defaultDatasetId"]).iterate_items())
print(r["signalScore"], r["signalLevel"], r["redFlags"])

The output includes domain.ageDays, registrar, tls.httpsValid, Wayback history, parked-or-for-sale detection, lookalike-domain hints, and lists of positiveSignals and redFlags. signalLevel is one of many_positive_signals, mostly_positive, mixed or several_red_flags.

Be clear with your agent (and your users) about what this is: automated signals, not a verdict. A brand-new domain isn't proof of a scam, and an old domain can be compromised. I use it as a gate. Below a threshold, the agent stops and asks a human.

6. What's this site built on?

{ "urls": ["https://wordpress.org", "https://www.shopify.com"], "mode": "fast" }

The output includes technologyNames, byCategory (CMS, analytics, web servers and more) and per-technology confidence and detectedVia. Fast mode ($0.0005 per site) reads the HTML and headers. Browser mode ($0.005 per site) renders the page, which catches tools that only appear after JavaScript runs. It's handy for agents doing lead research ("which of these 200 prospects run Shopify?") or for audits before a migration.

Keeping an agent's spend predictable

• Fixed prices per item let the agent estimate cost before it calls: 40 screenshots cost $0.06.
• Set a maximum cost per run (Console or run options). The run stops cleanly at the limit.
• Load only the tools you need in the tools parameter. Fewer tools means fewer wrong calls.

Wrap-up

You don't always need a browser-driving agent. For "show me", "save this", "how fast is it" and "can I trust it", one call per URL with a fixed price is faster, cheaper and easier to audit afterwards. If a cookie banner gets through or a page renders wrong, open an issue on the tool's Issues tab with the URL and I'll look at it.

This tutorial builds a small ingestion step that avoids those problems and keeps page numbers so answers can cite them. It uses a hosted converter I built and publish on the Apify Store (PDF to Text & Markdown with OCR). The pipeline itself (route by page type, chunk per page, keep provenance) works with any parser, so swap in Docling, Marker or PyMuPDF if you'd rather self-host.

Disclosure: I built the converter used here. The prices below come from its public pricing page. I have not included benchmarks; test it on your own worst documents.

What we want from the extraction step

For RAG, a good extractor should:

1. Read in visual order. Multi-column layouts should not interleave.

2. OCR only the pages that need it. Pages with a real text layer should be read directly, which is fast and accurate. Scanned or garbled pages should go through OCR.

3. Drop the noise. Repeating headers, footers and page numbers should go, and hyphenated line breaks should be joined.

4. Keep the structure. Headings should become #/##, so you have natural chunk boundaries.

5. Keep provenance. Every chunk should keep its source URL and page number.

The converter does 1–5 and returns one JSON item per document or per page. One honest limitation up front: table text comes out in reading order, but tables are not rebuilt as Markdown tables. If your corpus is mostly financial tables, test that first.

What it costs

Pricing is pay-per-event, with no subscription and no start fee:

Worked examples from the pricing page:

• 10-page digital PDF: $0.006

• 10-page scanned PDF: $0.053

• 500-page digital PDF: $0.012 (text pages are capped at 30 per document)

So 1,000 digital pages cost \(0.30 in page fees plus the per-document fee. For 100 ten-page PDFs that comes to about \)0.60 in total. Scans cost more ($5 per 1,000 OCR pages), because OCR takes real compute. Broken links, HTML pages and failed documents are free, and blank pages are never billed as OCR pages.

Setup

You need an Apify account and an API token (Console → Settings → API & Integrations).

pip install apify-client
export APIFY_TOKEN=apify_api_xxx

Step 1: one call, one document (curl)

The synchronous endpoint runs the converter and returns the dataset items in the response:

curl -X POST \
  "https://api.apify.com/v2/acts/poised_beardlichen~pdf-to-text-markdown-ocr/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://arxiv.org/pdf/1706.03762"], "outputFormat": "markdown"}'

You get back one item per document, with markdown, pageCount, textPages, ocrPages, words and a metadata object (title, author, producer, PDF version and so on). The sync endpoint waits up to 300 seconds, which is fine for a handful of files. For batches, use the async pattern below.

Step 2: page mode, so every chunk can cite a page

For RAG, switch on "outputMode": "page". Each page becomes its own item with url, page, ocr (true/false), ocrConfidence and markdown:

import os, json
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run_input = {
    "urls": [
        "https://arxiv.org/pdf/1706.03762",
        # Google Drive and Dropbox share links work too
    ],
    "outputMode": "page",          # one item per page
    "outputFormat": "markdown",
    "ocrMode": "auto",             # OCR only pages without a usable text layer
    "ocrLanguages": ["eng"],       # 27 languages available, e.g. ["eng", "deu"]
    "removeHeadersFooters": True,
}

# .call() starts the run and waits for it to finish
run = client.actor("poised_beardlichen/pdf-to-text-markdown-ocr").call(run_input=run_input)

chunks = []
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if not item.get("success") or not item.get("markdown"):
        continue
    chunks.append({
        "id": f'{item["url"]}#page={item["page"]}',
        "text": item["markdown"],
        "metadata": {
            "source": item["url"],
            "page": item["page"],
            "ocr": item["ocr"],
            "ocr_confidence": item.get("ocrConfidence"),
        },
    })

with open("chunks.jsonl", "w") as f:
    for c in chunks:
        f.write(json.dumps(c) + "\n")

print(f"{len(chunks)} page chunks written")

chunks.jsonl now holds one record per page with a stable ID such as https://arxiv.org/pdf/1706.03762#page=3. Feed it to whatever vector store you use. A page is usually a sensible chunk size for reports and papers. If your pages are long, split each page on its ## headings and keep the page metadata on every sub-chunk.

Route low-confidence OCR to a human

Because every page reports ocr and ocrConfidence, you can quarantine bad scans instead of silently indexing them. Do the split before you embed anything, so suspect pages go to a review file and never reach chunks.jsonl:

def low_confidence(c, threshold=70):
    return c["metadata"]["ocr"] and (c["metadata"]["ocr_confidence"] or 0) < threshold

needs_review = [c for c in chunks if low_confidence(c)]
clean = [c for c in chunks if not low_confidence(c)]

# Overwrite chunks.jsonl with only the pages that are safe to embed
with open("chunks.jsonl", "w") as f:
    for c in clean:
        f.write(json.dumps(c) + "\n")

# Park the suspect pages for a human to check (or re-run them with a different language setting)
with open("needs_review.jsonl", "w") as f:
    for c in needs_review:
        f.write(json.dumps(c) + "\n")

print(f"{len(clean)} pages ready to embed, {len(needs_review)} sent to review")

The threshold of 70 is just a starting point. Look at a few of your own pages and pick a value.

Step 3: batches without blocking (async pattern)

Text PDFs usually finish quickly. Long scans take roughly 1–3 seconds per OCR page. For big batches, start the run, poll it, then fetch the results:

import time

run = client.actor("poised_beardlichen/pdf-to-text-markdown-ocr").start(run_input={
    "urls": open("pdf_urls.txt").read().split(),
    "outputMode": "page",
    "outputFormat": "markdown",
    "maxConcurrency": 4,
})

while True:
    info = client.run(run["id"]).wait_for_finish(wait_secs=60)
    if info["status"] in ("SUCCEEDED", "FAILED", "ABORTED", "TIMED-OUT"):
        break

items = client.dataset(run["defaultDatasetId"]).list_items(clean=True).items
print(info["status"], len(items), "page items")

Two settings worth knowing:

• Maximum cost per run. Set it in the run options (or the Console) and the run stops cleanly at your limit, so a 5,000-URL batch can't surprise you.

• maxPagesPerDocument. Use it to sample huge files cheaply before committing to a full run.

The same thing in JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('poised_beardlichen/pdf-to-text-markdown-ocr').call({
  urls: ['https://arxiv.org/pdf/1706.03762'],
  outputMode: 'page',
  outputFormat: 'markdown',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const chunks = items
  .filter((it) => it.success && it.markdown)
  .map((it) => ({ id: `${it.url}#page=${it.page}`, text: it.markdown, page: it.page }));

console.log(chunks.length, 'chunks');

Letting an agent read PDFs directly (MCP)

If you'd rather have Claude, Cursor or another MCP client fetch documents on demand, add the converter through the Apify MCP server:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=poised_beardlichen/pdf-to-text-markdown-ocr",
      "headers": { "Authorization": "Bearer <YOUR_APIFY_TOKEN>" }
    }
  }
}

Then ask something like: "Read https://arxiv.org/pdf/1706.03762 and summarise section 3 with page references."

One practical detail: the MCP server waits only about 30–45 seconds for a run. For a long scanned PDF the tool call can return a run ID instead of results. The agent then uses get-actor-run and get-dataset-items with that ID. Tell your agent this in its instructions, or it may assume the call failed.

When not to use this

• You need table structure. Tables come out as text in reading order and are not rebuilt as Markdown tables. A table-focused parser or a vision model is a better fit there.

• Your documents can't leave your network. This is a hosted service. Files are downloaded into the run's temporary container and deleted after conversion, and results live in your own Apify storage. If that still isn't acceptable, run Docling or Marker locally. The pipeline above stays the same.

• The PDFs are password-protected to open. Those are reported as errors and not charged. PDFs that only have owner restrictions work.

Recap

• Extract per page and keep url and page on every chunk, so answers can cite pages.

• OCR only the pages that need it, and quarantine pages with low OCR confidence.

• Strip repeating headers and footers before embedding.

• Cap your spend per run, and sample big files first.

If you try it on a document that converts badly, open an issue on the Actor's Issues tab with the URL. That's the fastest way to get it fixed.