How to prepare company documents for RAG with a Markdown pipeline

Most RAG failures start before the model ever answers. Company knowledge sits in PDFs, Word files, decks, and spreadsheets. Those formats hide headings, break tables, and dump page noise into your index. A Markdown pipeline fixes the order of work: convert each source to clean Markdown, fix structure by eye, then chunk on headings so retrieval returns coherent passages. You keep one text format across tools, APIs, and local assistants. This guide shows how to prepare documents for RAG that way, when PDF to Markdown is enough, and what to check before you embed. For mixed file types, start at Convert to Markdown. For PDF-only batches, use PDF to Markdown.

Why raw company files break RAG

RAG systems retrieve text chunks, then pass them to a model. The model can only use what the retriever found. If your store holds garbled PDF extraction, missing section titles, or whole decks as one blob, answers drift even when the right file lives in the corpus.

PDFs mix layout with content. Footers, page numbers, and multi-column text often land in the wrong order. Word exports keep styles that converters misread as body copy. Slide decks store titles and bullets in separate shapes. Spreadsheets need table markup or the model loses row context. That failure lives in ingestion, not in the chat model.

Teams often paste whole PDFs into a chatbot or dump binary files into a vector store. Both paths skip structure. Without clear # / ## headings, chunkers cut mid-sentence or glue two topics together. Retrieval then scores a fragment that looks similar but answers the wrong question.

Markdown does not fix every scan on its own. It gives you a readable intermediate you can edit. You see broken headings. You delete boilerplate. You split a 80-page policy into sections before chunking. That human pass is cheaper than debugging “hallucinations” that were bad chunks.

Treat preparation as part of the product. The same Markdown you index can feed Obsidian, docs sites, and eval sets. One pipeline beats a different export path for every tool.

What a Markdown pipeline looks like

A practical pipeline has four stages: convert, clean, chunk, index. Skip cleaning and you encode layout bugs into embeddings. Skip heading-aware chunking and you encode random cuts. Conversion alone is not enough.

The convert stage turns binaries into .md files you can open in any editor. The clean stage is a human or scripted lint of headings, tables, and junk. The chunk stage cuts those files into retrieval units. The index stage writes embeddings plus metadata your app can cite.

Teams that jump from PDF drop to “vectorize everything” miss the middle two stages. They then blame the model when retrieval returns footer noise or half a table. Put the Markdown tree in version control or object storage so you can re-chunk without re-uploading originals every time.

Keep naming consistent across stages: one folder for sources, one for Markdown, one for chunk JSON if you export it. That layout makes audits and re-runs boring, which is what you want.

Convert sources to Markdown

Pull text out of PDF, DOCX, PPTX, XLSX, HTML, or EPUB into .md files. Prefer a converter that keeps heading levels and lists when the source has them. Online hubs such as Convert to Markdown cover mixed batches. Dedicated PDF to Markdown or Word to Markdown pages help when every file shares one type.

Expect imperfect tables and image-heavy pages. Conversion gives you a draft. You still own the QA step. For API-driven ingestion, see API docs once the manual path is stable.

Clean, chunk, then index

Open each .md and fix the structure you care about: title, section headings, list nesting, and table rows that matter for questions users ask. Delete repeated headers, legal disclaimers on every page, and empty sections.

Chunk on headings first, then fall back to size limits with overlap. Store metadata: source filename, section title, and last updated date. Index those chunks in your vector store or hybrid search. Re-run conversion when source files change; do not edit only the vectors.

How to prepare source files before conversion

Garbage in stays garbage. Spend ten minutes on the source before you upload.

Prefer text PDFs over photo scans. If the PDF is a scan, run OCR first or use an image path and expect more cleanup. Export Word as .docx, not old .doc. Save decks as .pptx. Flatten password protection so the converter can read the file.

Remove pages you will never retrieve: cover art, blank dividers, full-bleed marketing slides with no prose. Split huge binders into topic PDFs when possible. A 300-page employee handbook converts, but heading QA is painful as one file.

Name files with stable IDs you can put in metadata (hr-pto-policy-2026.pdf). Avoid spaces and version spaghetti in the name if your pipeline keys on filenames. Keep originals in a read-only folder and write Markdown to a separate tree so you can re-convert without losing edits.

If the corpus includes secrets, decide the trust boundary before any upload. Public converters are for non-sensitive drafts and published manuals. Confidential policies belong in a self-hosted MarkItDown CLI, a private worker, or an on-prem path. Privacy details for Toolsy uploads are below; the sibling post Is it safe to upload documents online? covers the checklist in more depth.

Convert PDFs and Office files to Markdown

You do not need a custom parser for the first corpus. Convert a sample of each format, inspect the Markdown, then scale.

Pick three representative files: a clean text PDF, a messy scan or brochure, and a Word handbook. Convert each, time the cleanup, and decide which class of file you will allow into the index. That sample set becomes your quality bar for automation later.

Budget time for QA. A five-minute conversion can need twenty minutes of heading fixes on a bad export. Write those fixes back into the .md (or into a small cleanup script) so the next re-convert does not lose them.

If your team ships an internal assistant, document the allowed formats and the owner of the Markdown tree. Otherwise every department invents a different export path and retrieval quality drifts.

Use an online converter for a first pass

For a mixed folder, open Convert to Markdown, upload a file under the size limit, and download the .md. Free quotas allow a few files per day; Pro raises caps if you batch. For PDF-only work, PDF to Markdown is the shorter path.

Check the first screen of output: does the document title appear as #? Do section titles use ##? Are lists real - or 1. lines, or mashed paragraphs? If headings are flat, fix them in the editor before chunking. That fix matters more than perfect typography.

Tables often need a second look. Wide Excel sheets may wrap into broken columns. Merged cells in Word can collapse. Fix the rows you will query; leave decorative formatting alone.

When CLI or library conversion fits better

Large private corpora, CI jobs, and air-gapped boxes belong on a local stack. Microsoft’s MarkItDown library is the common open-source choice for scripted conversion. Use it when you need batch folders, no browser upload, or reproducible builds. The wave post What is MarkItDown? compares library vs online.

Online conversion still wins for one-off QA, demos, and teaching the pipeline to non-engineers. Many teams convert samples online, lock the cleaning rules, then automate MarkItDown in CI for production.

How to chunk Markdown for RAG

Chunking decides what the retriever can return. Heading-based splits keep a section together. Fixed token windows ignore structure and cut mid-policy.

Start with Markdown headers as boundaries. Each ## section becomes a candidate chunk. If a section is still huge, split on ### or on paragraphs with a max token budget and a small overlap. Keep the parent heading in the chunk text or metadata so the model sees topic context.

Avoid one chunk per page from the original PDF. Page breaks are layout, not meaning. Avoid dumping an entire handbook into a single chunk. Retrieval then always returns the same giant blob.

Tune chunk size to your questions. FAQ-style queries need short sections. Procedural “how do I…” questions need full step lists in one chunk. Run a small eval set: ten real questions, retrieve top-k, read the chunks. If answers miss steps, enlarge chunks or raise overlap. If answers mix topics, tighten heading splits.

Store source path and section title with each embedding. When the model cites a wrong clause, you open the Markdown, not a binary PDF, and fix the section once.

Check the Markdown before you index

Do not embed the first download. Spot-check every format in the corpus.

Read the outline. Jump headings in an editor with a Markdown preview. Confirm the hierarchy matches how people talk about the doc (“PTO accrual” should be a heading, not buried in a paragraph). Search for OCR junk: l vs 1, broken hyphenation, repeated footer lines.

Skim tables that support numeric answers. If a benefits table lost a column, fix it or exclude that section until you have a better export. Drop pure image pages unless you OCR them into text; empty alt blobs waste index space.

Compare two converters on a hard PDF if quality looks poor. Some files need a different tool or a manual paste from the source Word. Record which sources fail so empty chunks do not reach the index unnoticed.

Only then run your chunker and load the store. Re-check after the first week of user questions: failed queries often point to a missing heading or a section you deleted as “boilerplate” that people still ask about.

Related jobs and formats

Company corpora mix formats. Word policies, Excel matrices, and training decks show up in the same knowledge base. Route each type through the matching converter, then merge Markdown into one tree with consistent heading style.

Use Word to Markdown for handbooks written in Word. Use the hub at Convert to Markdown when the batch mixes formats. Keep PDF batches on PDF to Markdown. If you need programmatic conversion for an ingestion service, wire API docs after the manual quality bar is clear.

Markdown also explains why models handle clean text better than raw PDFs. For the model-side argument, read Why LLMs prefer Markdown over PDF. For MarkItDown library details, see What is MarkItDown?.

Obsidian and other local vaults fit the same pipeline: convert, clean, then link notes. That reuse is a side benefit of treating Markdown as the system of record for RAG text.

Limits, privacy, and when not to upload

Converters struggle with heavy scans, handwritten notes, complex multi-column layouts, and documents that are screenshot galleries. You will spend more time cleaning than converting. In those cases, fix the source (text export, better scan) or hire a human pass for the critical sections.

Do not upload credentials, customer PII, health records, or unreleased deal terms to a public site unless your counsel accepts the risk. Toolsy processes uploads to produce the result, designs files to be discarded afterward (typically within about one hour), and does not use uploads to train its own models. Some AI-assisted steps may send content to external model providers. Product pages note when a provider is involved. For a fuller trust walkthrough, see Is it safe to upload documents online?.

Self-host MarkItDown or run conversion inside your VPC when the corpus is confidential. Use the online tools for public manuals, marketing PDFs you already publish, and non-sensitive drafts.

Size and quota limits apply on free tiers (for example a few files per day and a per-file MB cap). Split large books or upgrade when you batch. Always keep the original binary; Markdown is a derived artifact you may regenerate.

Frequently asked questions

How do I prepare documents for RAG?

Start by converting each source file to Markdown so headings and lists are visible as text. Clean the outline, remove repeated footers, and fix tables you will query. Chunk on Markdown headings, attach source metadata, then embed and index. Re-convert when the source document changes instead of editing vectors by hand.

Why convert PDF to Markdown for RAG instead of indexing the PDF?

Most PDF extractors lose reading order and heading levels, so chunkers cut in the wrong places. Markdown makes structure editable: you fix sections before embedding. You also get one format that works across RAG stacks, notes apps, and eval scripts. Keep the PDF as the legal original; use Markdown as the retrieval text.

How should I chunk Markdown for RAG?

Prefer splits on # / ## / ### so each chunk is a coherent section. If a section is still too large for your model context, split on paragraphs with a max size and a small overlap, and keep the parent heading in the chunk or metadata. Test with real questions and adjust size when retrieval misses steps or mixes topics.

Can I use MarkItDown for a company RAG pipeline?

Yes. MarkItDown is a common library for scripted PDF and Office conversion to Markdown. Use the CLI or Python package in CI for private batches. Use an online converter when you need a quick sample or a non-engineer QA pass. See What is MarkItDown? for library vs online tradeoffs.

Is PDF to Markdown enough for an LLM knowledge base?

It is enough for text-heavy reports and policies with a clear outline. It is not enough alone for scans without OCR, complex forms, or image-only slides. After conversion, you still clean headings and choose a chunking strategy. Mixed corpora also need Word, Excel, or deck converters into the same Markdown tree.

What file types should go through a Markdown pipeline?

PDFs, Word (.docx), PowerPoint, Excel tables you want as text, HTML exports, and EPUB notes all fit. Images need OCR first. Audio and video need transcripts before Markdown. Skip binaries you cannot turn into meaningful text; they only add noise to retrieval.

Is it safe to upload company documents to an online converter?

Only upload content you are allowed to process in a third-party cloud. Prefer public or low-sensitivity files online. For confidential corpora, convert on your own machines. On Toolsy, uploads are processed for the request, designed to be discarded within about an hour, and not used to train Toolsy's own models. Some steps may use external AI providers. Read Is it safe to upload documents online? before you send regulated data.

How do I handle tables and images after PDF to Markdown?

Open the .md and verify table pipes and column counts for any figure users ask about. Rebuild broken tables by hand or re-export from Excel via a spreadsheet-to-Markdown path. For images that contain text, run OCR into Markdown; for diagrams that matter, add a short caption paragraph so retrieval has words to match.

Should I build a document ingestion pipeline or convert files by hand?

Convert by hand for the first ten files so you learn failure modes. Then automate: watch a folder, convert with MarkItDown or an API, run lint checks on headings, chunk, and upsert to the index. Manual QA stays in the loop for new templates. Full automation without spot checks ships silent bad chunks.

Where do I convert files to Markdown on Toolsy?

Use Convert to Markdown for mixed formats in one place. Use PDF to Markdown when every file is a PDF, or Word to Markdown for .docx. For scripted ingestion, check API docs. Pair conversion with the cleaning and chunking steps in this guide before you embed.

For why models behave better on clean Markdown than on raw PDFs, continue with Why LLMs prefer Markdown over PDF. For MarkItDown specifics, see What is MarkItDown?. When you already have Pandoc installed for scripted jobs, see Pandoc PDF to Markdown: when to skip the CLI. When you are shopping converters, see ChatGPT PDF upload vs Markdown-first and pdf2md.net vs a full document converter suite. For the full index of Markdown + AI workflows, start at Markdown for AI: complete guide.