Skip to main content

AI and machine learning · Beginner · 18 min

How to Split Documents into Chunks for a RAG App

By Padding · Updated 15 Nov 2026
Python 3.15

You will write a Python chunker with no dependencies that splits Markdown help pages at their headings and sentences into overlapping pieces, saves them with their source and heading, and checks that no text was lost and no piece is too long.

On this page

Retrieval-augmented generation, or RAG, answers a question by first finding the passages of your own documents that are relevant, then giving those passages to a language model along with the question. This series builds one from scratch: split documents into chunks (this tutorial), store and search them in PostgreSQL, answer questions through an API, then measure how well it works.

Chunking comes first because everything later depends on it. An embedding model turns a piece of text into one vector, so a whole 20-page document becomes one blurry point, and a question about one paragraph cannot find it. Pieces that are too small lose their context. The goal is chunks that each hold one idea, carry enough surrounding text to make sense alone, and know where they came from.

The tutorial uses a small, made-up help centre for a hosting company called Fernwood: three Markdown files. The chunker uses only Python's standard library.

Step 1 · Create the sample documents

Create a project folder with a docs folder inside it, and add three short help pages:

Bash
mkdir -p ~/rag-demo/docs
cd ~/rag-demo
Bash
cat > docs/billing.md <<'MD'
# Billing and refunds

## When you are charged

Fernwood Hosting bills each plan monthly, on the day of the month you signed up. If you signed up on the 31st, you are billed on the last day of shorter months. The invoice lists each plan, add-on and domain separately, and it is emailed to the billing contact on the account.

Prices are shown without sales tax. Tax is added at checkout for customers in countries where we must collect it.

## Refunds

Every new plan has a 30-day money-back guarantee. Cancel within 30 days of the first payment and we refund the plan fee in full, without questions. Domain registrations and SSL certificates bought from partners are not covered, because we pay for them when you order.

After the first 30 days we do not refund the current month, but you keep the service until the end of the period you paid for. To ask for a refund, open a ticket from the Billing page and include the invoice number. Refunds go back to the original payment method and take five to ten business days to appear.

## Failed payments

If a payment fails, we try the card again after three days and again after seven days. We email you after each failed attempt. If the third attempt fails, the account is suspended and sites go offline, but your files are kept for 14 days. Update the card on the Billing page and the account is reactivated within minutes.
MD
Bash
cat > docs/domains.md <<'MD'
# Domains and DNS

## Point a domain at your site

A domain you bought elsewhere can point at your Fernwood site with two DNS records. At your registrar, create an A record for the bare domain, such as example.com, with the IP address shown on your site's Overview page. Then create a CNAME record for www that points to the bare domain.

Do not delete any MX records while you do this. Those records route your email, and removing them stops mail from arriving.

## DNS changes take time

A DNS change is not instant, because resolvers around the world keep the old answer until its time to live (TTL) runs out. Most records are visible within an hour, but allow up to 48 hours before you decide something is wrong. To make a future change faster, lower the TTL to 300 seconds a day before you edit the record.

## Transfer a domain in

You can transfer a domain to Fernwood if it is at least 60 days old and unlocked at the current registrar. Ask your registrar for the authorization code, then enter it on the Domains page. A transfer adds one year to the registration and takes up to seven days to finish.
MD
Bash
cat > docs/backups.md <<'MD'
# Backups and restores

## Automatic backups

Fernwood takes a full backup of every site each night and keeps the last seven. Plans with the Backup Plus add-on keep thirty. Backups are stored in a different data centre from your site, so a failure at one location does not lose both copies.

## Restore from a backup

Open the site, choose Backups and pick the date you want. A restore replaces the files and the database with the backup. Anything written after that backup is lost, so download a copy of the current site first if it matters. A typical restore finishes in under ten minutes, and larger sites take longer.

## Download a copy

You can download any backup as a zip file from the Backups page. The file is available for 24 hours and then deleted. Keep a copy somewhere you control as well, because the guarantee in our terms covers the service, not data you have not copied elsewhere.
MD

Check it worked

Bash
ls docs
wc -w docs/*.md

The listing shows backups.md, billing.md and domains.md. The word counts are a few hundred words per file, which is why this tutorial uses small chunks. Real documents are longer, and the same code works on them.

Step 2 · See why splitting every N characters goes wrong

The simplest chunker cuts the text every 300 characters. Try it on the billing page:

Bash
cat > naive.py <<'PY'
from pathlib import Path

text = Path("docs/billing.md").read_text()
chunks = [text[i : i + 300] for i in range(0, len(text), 300)]

print("chunk 0 ends with:", repr(chunks[0][-30:]))
print("chunk 1 starts with:", repr(chunks[1][:30]))
PY
python3 naive.py

Check it worked

The output shows chunk 0 ending in the middle of a sentence, at 'd it is emailed to the billing', and chunk 1 starting with the rest of it, ' contact on the account.\n\nPric'. It would just as happily cut a word in half. A chunk that starts mid-sentence embeds badly and reads badly when you later show it to a model. It also ignores headings, so text about refunds and text about failed payments can land in the same chunk.

The fix is to cut at boundaries the author already chose: headings first, then paragraphs and sentences, with a size limit as the last resort.

Step 3 · Write the chunker

Create chunker.py. It has four jobs: split the Markdown at headings, remember the heading path, split each section into sentences, and pack sentences into chunks up to a size limit, carrying a little text from the end of one chunk into the next:

Bash
cat > chunker.py <<'PY'
import json
import re
import sys
from pathlib import Path

MAX_CHARS = 300
OVERLAP_CHARS = 120


def load_documents(folder):
    for path in sorted(Path(folder).glob("*.md")):
        yield path.name, path.read_text(encoding="utf-8")


def split_sections(text):
    """Split Markdown into (heading path, body) pieces at every heading."""
    sections, path, body = [], [], []

    def flush():
        content = "\n".join(body).strip()
        if content:
            sections.append((" > ".join(path), content))
        body.clear()

    for line in text.splitlines():
        match = re.match(r"^(#{1,6})\s+(.*)$", line)
        if match:
            flush()
            level = len(match.group(1))
            path[:] = path[: level - 1] + [match.group(2).strip()]
        else:
            body.append(line)
    flush()
    return sections


def split_sentences(body):
    sentences = []
    for paragraph in re.split(r"\n\s*\n", body):
        paragraph = " ".join(paragraph.split())
        if paragraph:
            sentences.extend(re.split(r"(?<=[.!?])\s+", paragraph))
    return sentences


def hard_split(sentence, max_chars):
    return [sentence[i : i + max_chars] for i in range(0, len(sentence), max_chars)]


def pack(sentences, max_chars=MAX_CHARS, overlap=OVERLAP_CHARS):
    units = []
    for sentence in sentences:
        units.extend(hard_split(sentence, max_chars) if len(sentence) > max_chars else [sentence])

    chunks, current = [], []
    for unit in units:
        if current and len(" ".join(current + [unit])) > max_chars:
            chunks.append(" ".join(current))
            # carry the last few sentences into the next chunk as overlap
            carried, size = [], 0
            for old in reversed(current):
                if size + len(old) + 1 > overlap:
                    break
                carried.insert(0, old)
                size += len(old) + 1
            current = carried
            if len(" ".join(current + [unit])) > max_chars:
                current = []
        current.append(unit)
    if current:
        chunks.append(" ".join(current))
    return chunks


def chunk_documents(folder, max_chars=MAX_CHARS, overlap=OVERLAP_CHARS):
    for source, text in load_documents(folder):
        number = 0
        for heading, body in split_sections(text):
            for piece in pack(split_sentences(body), max_chars, overlap):
                yield {
                    "id": f"{source}#{number}",
                    "source": source,
                    "heading": heading,
                    "text": piece,
                }
                number += 1


if __name__ == "__main__":
    folder, out = sys.argv[1], sys.argv[2]
    with open(out, "w", encoding="utf-8") as handle:
        count = 0
        for chunk in chunk_documents(folder):
            handle.write(json.dumps(chunk) + "\n")
            count += 1
    print(f"wrote {count} chunks to {out}")
PY

How it works:

  • split_sections walks the file line by line. A line starting with one to six # characters is a heading. The function keeps a path such as Billing and refunds > Refunds, so every section knows where it sits.
  • split_sentences splits a section into paragraphs, then into sentences at a full stop, question mark or exclamation mark followed by a space. This is simple and fine for clean prose. Text with abbreviations like "e.g." needs a better splitter.
  • pack adds sentences to the current chunk until the next one would push it over MAX_CHARS, which is 300 here. A sentence longer than the limit is cut by characters as a last resort.
  • When a chunk is full, the last sentences that fit in OVERLAP_CHARS (120) are carried into the next chunk. The overlap means a fact that sits on a boundary appears whole in at least one chunk, and a chunk that starts with a pronoun still has the sentence before it.
  • Each chunk is a dictionary with an id made from the file name and a counter, the source file, the heading path and the text. You will want the source and heading later, to show the reader where an answer came from.

300 characters is far smaller than you would use for real documents, where 800 to 2,000 characters is a common starting point. It is small here so that three short pages produce enough chunks to see the behaviour.

Step 4 · Run it and read the output

Chunk the folder and write the result as JSON Lines, one chunk per line:

Bash
python3 chunker.py docs chunks.jsonl

Check it worked

The output is wrote 16 chunks to chunks.jsonl. Look at how the chunks came out, with their lengths and headings:

Bash
python3 - <<'PY'
import json

for line in open("chunks.jsonl"):
    chunk = json.loads(line)
    print(chunk["id"], len(chunk["text"]), chunk["heading"])
PY

The list has 16 rows, from backups.md#0 to domains.md#4, and every length is at most 300. A short section such as "Automatic backups" is one chunk, and longer ones such as "Refunds" are three. Print one chunk in full to see the overlap:

Bash
python3 - <<'PY'
import json

chunks = [json.loads(line) for line in open("chunks.jsonl")]
for chunk in chunks:
    if chunk["id"] in ("billing.md#2", "billing.md#3"):
        print(chunk["id"], "|", chunk["text"], "\n")
PY

The sentence beginning Domain registrations and SSL certificates ends billing.md#2 and starts billing.md#3. That is the overlap at work. A question about whether a domain is refundable can match either chunk.

Step 5 · Check the chunks for lost text and wrong sizes

Chunking bugs are quiet: text goes missing and nothing fails. Write a check that proves two things. No chunk exceeds the size limit, and every sentence of every source file appears in at least one chunk:

Bash
cat > check_chunks.py <<'PY'
import json

from chunker import MAX_CHARS, load_documents, split_sections, split_sentences

chunks = [json.loads(line) for line in open("chunks.jsonl")]

too_long = [c["id"] for c in chunks if len(c["text"]) > MAX_CHARS]
assert not too_long, f"chunks over {MAX_CHARS} characters: {too_long}"

missing = []
for source, text in load_documents("docs"):
    mine = " ".join(c["text"] for c in chunks if c["source"] == source)
    for heading, body in split_sections(text):
        for sentence in split_sentences(body):
            if sentence not in mine:
                missing.append((source, sentence[:40]))
assert not missing, f"sentences not found in any chunk: {missing}"

assert len({c["id"] for c in chunks}) == len(chunks), "duplicate chunk ids"
print(f"ok: {len(chunks)} chunks, none too long, no text lost")
PY
python3 check_chunks.py

Check it worked

The output is ok: 16 chunks, none too long, no text lost. Run this check whenever you change the chunker or the documents. If it fails, the message names the chunks that are too long or the sentences that vanished.

Troubleshooting

If the check reports missing sentences after you edited a document, you probably re-ran check_chunks.py without running chunker.py first, so chunks.jsonl is out of date. Run both in order. If a chunk comes out longer than the limit, look for a long line without a full stop, such as a URL or a table row: it is cut by characters, and the next chunk's overlap may pull it back over.

Step 6 · Choose a chunk size by looking, not guessing

Size is the main setting you will tune. See how it changes the number of chunks for the same documents:

Bash
python3 - <<'PY'
from chunker import chunk_documents

for size, overlap in [(150, 40), (300, 120), (600, 120), (1200, 200)]:
    chunks = list(chunk_documents("docs", size, overlap))
    print(f"size {size:>4}, overlap {overlap:>3}: {len(chunks):>2} chunks, longest {max(len(c['text']) for c in chunks)}")
PY

Check it worked

The output has four rows: 29 chunks at size 150, 16 at 300, and 9 at both 600 and 1,200. At 600 and above, each of the nine sections already fits in a single chunk, so a bigger limit changes nothing: the headings are the real boundaries.

What to weigh when you choose:

  • Small chunks match a specific question precisely, but they lose context and make the later steps work harder, because the model sees many fragments.
  • Large chunks keep context, but one vector now has to represent several ideas, so matches get vaguer, and each question sends more text to the model, which costs more.
  • Embedding models have an input limit. Voyage AI's voyage-4, which the later tutorials use, accepts up to 32,000 tokens, so the limit is rarely your constraint. Retrieval quality is.
  • Characters are a rough proxy for tokens. For English prose a token is about four characters, so 1,200 characters is about 300 tokens. Use a tokenizer if you need exact counts.

The honest way to pick is to measure: build a set of questions with known answers and compare settings. The last tutorial in this series does that.

Step 7 · Keep the chunks for the next tutorial

The file chunks.jsonl is the input for the next tutorial, where you store the chunks in PostgreSQL. Keep the rag-demo folder. Nothing in this tutorial needs cleaning up apart from that folder, which you remove only when you finish the whole series.

Next steps

Need this done for you?

Padding does this work for growing companies, from the first diagnosis to a system that keeps working.

AI-Integrated Solutions

Get new tutorials

New tutorials, about twice a month.