Building a knowledge base for a local AI assistant from PDFs and Word files

A local AI assistant only answers well from text it can retrieve. Your manuals and policies still live in PDFs and Word files. Those binaries fight chunkers and small context windows. Build an LLM knowledge base by converting each source to Markdown, fixing headings by eye, then pointing your local stack (Ollama-based apps, AnythingLLM-style workspaces, PrivateGPT-class tools) at that folder. Toolsy converts documents; you run the model and index on your machine. This guide covers the convert → clean → load path for a document knowledge base, when Word to Markdown or PDF to Markdown is enough, and when to keep secrets off the public internet.

Why a local AI needs a document knowledge base

Chat memory is not a corpus. A local model forgets previous sessions unless you wire retrieval or paste context by hand. Pasting whole PDFs into a prompt burns tokens and still loses section structure. You need a knowledge base: a set of text files (or chunks derived from them) that your app can search before it answers.

PDFs and Word dumps are a poor first store. Extractors scramble columns, drop styles as body noise, and leave page footers in every chunk. Local RAG stacks that ingest binaries often hide that mess inside a one-click “upload PDF” button. You get fluent wrong answers with no clear place to edit the source.

Markdown gives you an intermediate you own. You open a .md file, see broken headings, delete repeated legal banners, and split a 200-page handbook into topic notes. The same tree can feed a local assistant, a notes app, and later a company RAG service. One conversion step beats maintaining a different export for every chat UI.

Treat the knowledge base as product data. Name files, keep originals read-only, and version the Markdown when policies change. Re-index after edits. Do not fine-tune a small model on every memo when retrieval over clean Markdown already solves most “answer from our docs” jobs.

What an LLM knowledge base means on your machine

In practice, an LLM knowledge base is a folder of text plus an index your assistant queries. Some apps embed each file into a local vector store. Others use keyword search or hybrid retrieval. The model still generates the reply; the knowledge base only supplies passages.

You do not need a cloud SaaS account for that loop. You need: (1) readable text with stable headings, (2) an ingestion path in your chosen app, (3) a local or private model endpoint. Conversion sits before step 2. Skip it and you ask the indexer to invent structure from layout.

Keep the job titles clear. Toolsy and similar converters turn DOCX/PDF into Markdown. They are not your local LLM host. Ollama, LM Studio, or a self-hosted API runs the model. AnythingLLM, Open WebUI plugins, PrivateGPT-style projects, or a small custom RAG script own retrieval. Confusing those roles leads people to expect an online converter to “chat with the document” forever after one upload. That is a different product.

Start small. Ten high-value manuals beat two thousand unread PDFs. Measure success with questions you already ask: “What is our PTO carryover?” “Where is the API rate limit?” If retrieval misses, fix the Markdown or chunk boundaries before you switch models.

Retrieval first, fine-tune later

Fine-tuning teaches style and narrow skills. It does not replace a living document store. Policies change weekly; re-training is slow and expensive. Prefer RAG or “chat with folder” retrieval for an LLM knowledge base built from PDFs and Word. Fine-tune only when you have a stable domain language and evals that retrieval cannot fix.

If a guide tells you to “train an LLM on your knowledge base,” read it as marketing shorthand for retrieval unless they show a training dataset and loss. For local assistants, index Markdown and cite sources. Keep fine-tuning as a later experiment.

Local stack pieces you still choose

Pick a model host you can run offline or on your LAN. Pick a UI or script that watches a directory of .md files. Pick embedding settings your hardware can hold. Conversion quality matters more than brand names on the UI. A clean Markdown tree improves every stack; a dirty PDF dump hurts all of them.

When you need Obsidian as the editing surface, convert first, then import notes. See Import PDFs into Obsidian as Markdown for that notes-first path. This article stays on the assistant knowledge-base job.

Convert PDFs and Word files to Markdown first

Do conversion before you open the local AI UI. Batch the files you trust, write .md next to a sources/ folder of originals, then ingest only Markdown. That order lets you QA text without re-uploading binaries every time the indexer fails.

Sample three files before you convert the whole drive: one clean Word handbook, one text-heavy PDF, one messy scan or brochure. Time the cleanup. That sample sets your quality bar. Reject formats that need hours of repair until you have OCR or a redesign of the source.

Prefer .docx over old .doc. Prefer text PDFs over photo scans. Flatten passwords so converters can read the file. Strip cover art pages that hold no prose. Name files with stable IDs (hr-pto-2026.docx) so metadata and citations stay boring.

For mixed batches, use Convert to Markdown. For Word-heavy corpora, use Word to Markdown. For PDF-only sets, use PDF to Markdown. For private corpora, prefer MarkItDown CLI on your own machine; details sit in What is MarkItDown? and in the privacy section below.

Word (.docx) to Markdown for handbooks and SOPs

Word is where many teams still write SOPs, HR policies, and client playbooks. Heading styles in Word map well to # / ## when the author used real styles instead of bold body text. Convert those files first; they often need less cleanup than marketing PDFs.

Upload a .docx, download the .md, and open it in a plain editor. Check that Heading 1 became # and that lists stayed lists. Fix fake headings (bold paragraphs) by adding # yourself. Tables need a quick column count check. That pass is the difference between a useful local knowledge base and a bag of broken fragments.

Keep the Word original for legal sign-off if you must. Edit the Markdown for retrieval. When the Word file changes, re-convert and merge your heading fixes, or keep a short cleanup script. Do not maintain two conflicting “sources of truth” without a named owner.

PDF to Markdown for manuals and exports

PDFs arrive as vendor manuals, scanned contracts, and “print to PDF” exports from tools that never offered DOCX. Text PDFs convert cleaner. Scans need OCR first; expect more cleanup. Multi-column layouts and footers are the usual failure modes. Why models prefer Markdown over raw PDF extract is covered in Why LLMs prefer Markdown over PDF.

Convert, then delete repeating headers and page numbers from the .md. Split huge binders into topic files when one PDF mixes five products. Local RAG with small context windows pays for that split. Leave the full PDF in sources/ for humans who need the layout.

If Marker vs MarkItDown vs an online converter is your next decision, read Marker vs MarkItDown vs online converters. For this guide, any path that yields editable Markdown with real headings is enough to proceed.

Clean and organize Markdown for a local assistant

Conversion is a draft. Cleaning is the work that makes retrieval trustworthy. Open each file and ask: would I paste this section into a prompt? If not, fix it before indexing.

Aim for one topic per file when files are large. Use a short title as the first # heading. Keep section headings consistent so chunkers that split on ## produce coherent units. Remove empty sections, “click here” leftovers from HTML-to-PDF exports, and decorative horizontal rules that add nothing.

Store metadata you care about in a YAML frontmatter block or a top paragraph: source filename, owner, last reviewed date. Local apps differ in how they read frontmatter; a plain first paragraph still helps humans and some retrievers. Consistency beats a perfect schema you never fill in.

Put the tree under version control if the corpus is shared. Local AI apps can re-scan the folder after a git pull. That workflow beats everyone dropping random PDFs into a desktop chat window.

Folder layout and naming

Use a boring layout:

• sources/: original PDF and DOCX (read-only)
• markdown/: cleaned .md ready to index
• rejected/: files that failed QA until OCR or rewrite

Name Markdown files after the source ID plus a short slug (hr-pto-2026.md). Avoid spaces if your scripts split on whitespace. Group by domain (hr/, eng/, legal/) so you can point the assistant at a subset workspace when the full corpus is too noisy.

Do not index sources/ and markdown/ together. Binary leftovers confuse some ingest pipelines and double your chunk count. Point the app at markdown/ only.

Headings, tables, and junk to strip

Fix heading hierarchy before you tune chunk size. A file with one giant # and no ## forces size-based splits that cut mid-procedure. Add ## at real section breaks. Demote decorative titles that are not sections.

For tables users ask about, verify pipe tables row by row. Rebuild broken ones from the Word or Excel source if needed. For figures that matter, add a one-sentence caption under the image placeholder so retrieval has words to match. Strip "Confidential - Page 12 of 48" lines that appear every page; they dominate similarity search for the wrong reasons.

Run a quick search across the folder for your company name in footers and for repeated disclaimer blocks. One cleanup pass here saves weeks of “the bot cites the footer” tickets.

Load Markdown into your local AI stack

With a clean markdown/ tree, open your assistant and add that folder as a workspace, collection, or docs path. Use the app’s “rescan” or “embed” action. Ask three known questions and read the cited passages. If the citation is wrong, fix the Markdown or exclude the file; do not only raise temperature.

Chunk settings matter once the text is clean. Prefer heading-aware splits when the UI offers them. If you only get a character limit, keep sections short in the files themselves. Overlap helps when answers span two subsections; too much overlap duplicates noise. Details on chunking for RAG live in Prepare documents for RAG with Markdown; reuse those habits locally.

Re-run embedding after you edit Markdown. Stale vectors are a common local-RAG bug: you fixed the SOP, the index still serves last week’s text. Make “rescan after edit” part of the checklist next to “save file.”

Keep the model and the docs on the same trust boundary. If the model calls out to a cloud API, your retrieved chunks may leave the machine even when files started local. Read your stack’s networking docs. Offline mode is a setting, not a vibe.

Privacy: local conversion vs online converters

Decide the trust boundary before the first upload. Customer contracts, credentials, health data, and unpublished financials belong in a local MarkItDown CLI, a private worker, or an air-gapped path. Public converters suit published manuals, marketing PDFs, and non-sensitive samples you use to test the pipeline.

Toolsy processes uploads to return Markdown, designs discard of uploads within about one hour, and does not use uploads to train Toolsy’s own models. Some conversion or OCR steps may send content through an AI gateway and model providers. That is still cloud processing. Prefer local conversion when residual cloud risk is unacceptable. The checklist in Is it safe to upload documents online? walks the decision in more depth.

Never paste API keys, passwords, or private keys into a file you plan to upload for “just a quick convert.” Redact first, or convert offline. Operational logs on web tools may keep non-content metadata; assume filenames can leak intent even when bodies are short-lived.

If your local assistant embeds on-device and never leaves LAN, keep conversion on the same side of the firewall. Mixing “secret PDF → public converter → private RAG” creates a one-time leak that your later offline setup cannot undo.

Limits, mistakes, and when cloud still wins

Scans without OCR produce empty or garbage Markdown. Complex forms and handwritten notes need human transcription or specialized OCR. Image-only slides need either OCR or a written summary page. Conversion will not invent missing text.

Huge corpora without owners rot. Someone must approve new files into markdown/ and retire obsolete ones. Otherwise the local AI cites two conflicting policies with equal confidence. Assign an owner per folder.

Cloud assistants still win for teams that need managed uptime, SSO, and shared evals across regions. You can keep the same Markdown pipeline and point a hosted RAG product at it later. The local path teaches you which files matter; it does not ban cloud forever.

Toolsy will not run your local model, host your vector index, or replace AnythingLLM. Use it (or CLI MarkItDown) to produce Markdown, then load that Markdown where you already run inference. Expecting an online converter to be a private LLM is the most common category error in this topic.

Frequently asked questions

How do I build a knowledge base for an LLM from PDFs and Word files?

Convert each PDF and Word file to Markdown, clean headings and tables, store the .md files in a folder, then point your local assistant or RAG app at that folder and embed or index it. Ask known questions and fix files that retrieve poorly. Prefer retrieval over fine-tuning for living documents. Keep originals in a separate sources/ directory so you can re-convert without losing edits.

What is an LLM knowledge base on a local AI assistant?

It is the set of documents your app searches before the model answers, usually as Markdown or chunks derived from Markdown on disk. The local model generates text; the knowledge base supplies evidence. Without that store, the assistant only uses its training data and whatever you paste into the prompt. A clean folder of Markdown is the usual practical form for solo and small-team local setups.

Do I need RAG for a local document knowledge base?

Most “chat with my docs” local apps already implement RAG or a close cousin: retrieve passages, then prompt the model. You still benefit from understanding chunk boundaries and metadata even if the UI hides the word RAG. If your tool only pastes whole files into context, keep files short or it will truncate. For a deeper company pipeline view, see Prepare documents for RAG with Markdown.

Should I convert Word to Markdown before loading into AnythingLLM or similar apps?

Yes when you want editable structure and consistent headings across PDF and DOCX sources. Many apps accept Word or PDF directly, but you lose a clear QA step and a format you can fix in git. Convert with Word to Markdown or a local CLI, clean the .md, then ingest. Re-convert when the Word source changes and refresh the index.

Is PDF to Markdown enough for local RAG?

It is enough for text-heavy manuals with a clear outline after you clean headings and strip footers. It is not enough alone for scans without OCR, dense multi-column layouts, or image-only pages. Mixed corpora also need Word (and sometimes spreadsheet) conversion into the same Markdown tree. Check citations on real questions before you trust the index.

Can Toolsy run my local LLM or host the knowledge base?

No. Toolsy converts documents to Markdown (and offers related file tools). You run the local model host and the retrieval UI or script yourself. After conversion, download the .md and load it into your stack. Treating an online converter as a private AI assistant overclaims the product and blurs the privacy boundary.

Is it safe to upload company Word files to convert them online?

Only when you are allowed to process that content in a third-party cloud and you accept residual risk. Prefer local MarkItDown or another offline converter for secrets and regulated data. On Toolsy, uploads are processed for the request, designed to be discarded within about an hour, and not used to train Toolsy’s own models; some steps may use external AI providers. Read Is it safe to upload documents online? before you send sensitive files.

How is this different from preparing documents for company RAG?

Company RAG guides focus on shared pipelines, chunking policy, and production indexes. This article focuses on a local assistant: private folders, Word/PDF conversion for a personal or team machine, and honest limits of online converters. The Markdown hygiene overlaps. The trust boundary and stack (local model + local index) are the difference. Link both posts when you move from laptop prototype to company service.

Should I fine-tune a local model on my documents instead of building a knowledge base?

Start with a knowledge base and retrieval. Fine-tuning is slower to update, harder to cite, and easy to overfit on a small SOP set. Use fine-tuning later for tone or domain slang once retrieval quality plateaus and you have evals. Most local “train on my PDFs” requests are solved by better Markdown and indexing.

Where do I convert files for a local AI knowledge base on Toolsy?

Use Word to Markdown for .docx handbooks, PDF to Markdown for PDF manuals, and Convert to Markdown when formats are mixed. Download the Markdown, clean it locally, then ingest it in your assistant. For scripted private batches, prefer MarkItDown CLI on your own hosts instead of uploading secrets.

For Markdown structure in a full RAG pipeline, continue with Prepare documents for RAG with Markdown. For why clean Markdown beats raw PDF extract, see Why LLMs prefer Markdown over PDF.