mcpdata Request an invite

Answers / Get data into your AI

My CSV is too large for ChatGPT to analyse. What can I do?

Recipe verified on Ran on a private test instance
Short answer

Chat assistants analyse uploaded files in a sandbox with limited memory and time, so a large CSV gets cut short or fails. Load the file into a database your AI can query instead: it's loaded once, your AI runs exact queries over every row, and only the results come back into the conversation.

Steps

  1. Connect your assistant to your instance (this writes, so it needs your own instance with a ReadWrite key from the accounts portal; the public demo is read-only). In Claude Code: 
    claude mcp add --transport http mcpdata https://[you].mcpdata.org/mcp --header "Authorization: Bearer $CDB_API_KEY"

    Replace [you] with your instance's name. Other apps: Connect your AI.

  2. Install the client and sign it in. One pip install, then approve this computer in your browser (the key is kept in ~/.cdb/): 
    pip install https://[you].mcpdata.org/help/download/cdb-0.3.4-py3-none-any.whl
cdb-localdata login --to https://[you].mcpdata.org
  3. Load the folder that holds the CSV. Here the sample orders_sample.csv (100,000 rows) in a folder called csv; each file becomes a table under the namespace you choose: 
    cdb-localdata add ./samples/csv --to https://[you].mcpdata.org --namespace files
cdb-localdata sync
  4. Ask your question. Give your AI this prompt:
    Using mcpdata, total revenue and order count by region from files.orders_sample.
What your AI does: the tool calls it makes for this prompt, and what came back
describe_table(table_name="files.orders_sample") → 15 columns, 100,000 rows
query(table_name="files.orders_sample", group_by=["region"], agg=[{"func": "sum", "column": "revenue", "alias": "revenue"}, {"func": "count", "column": "order_id", "alias": "orders"}], sort=[{"column": "revenue", "desc": true}]) → 3 rows
What you should get · the recipe's real result from its last run
regionrevenueorders
Americas22,107,38541,803
EMEA20,392,04637,936
APAC10,875,25820,261
Ask Claude with this prompt Ask ChatGPT with this prompt

The assistant needs mcpdata connected (step 1) to run the query.

Why this works

A context window holds a limited amount of text, and in-chat file analysis runs with limited memory and time. A database keeps the data outside the conversation: the AI asks for exactly the totals, filters or rows it needs, and the answer is computed over all of the data. Big files are sent in parts and retried safely, so the size of the file isn't the problem any more.

Related questions