---
title: "How to make your Drupal content readable by AI Crawlers and LLMs with Markdown Mirror"
description: "Serve AI crawlers a clean Markdown version of your Drupal pages. See how the Markdown Mirror module works and how to set it up in 5 steps."
url: https://www.specbee.com/blogs/drupal-markdown-ai-crawlers
type: Article
author: "Preethy Ray"
categories: "Drupal AI"
published: 2026-11-17
updated: 2026-11-17
---

# How to make your Drupal content readable by AI Crawlers and LLMs with Markdown Mirror

AI tools, chatbots, and LLM-based crawlers read your website every day. They read it differently from a human visitor. A crawler pulls the full HTML page apart and goes through menus, scripts, and markup to find the content inside.

That extra work is slow and expensive. It also means your content can get misread or skipped.

That’s where the Markdown Mirror module I recently built comes in. In this article, I’ll cover what the module does, how it compares with Cloudflare's Markdown for Agents, and how to set it up in five simple steps.

**Key Takeaways**

- Markdown Mirror is a Drupal module that serves a clean Markdown copy of any enabled page at the same URL, with .md appended.
- In Cloudflare's test of one of its own blog posts, the Markdown version used 80% fewer tokens than the HTML version (Cloudflare, 2026).
- Its main value is a clean, cheap-to-read source for AI tools that fetch your pages directly and for your own chatbots or RAG pipelines.
- It can also publish /llms.txt as a low-cost extra, but no major AI provider has confirmed reading these files.
- Nothing is exposed until you opt in. Restricted content stays restricted, and it needs zero field mapping on Drupal 10.3+ and Drupal 11.

## What is Drupal Markdown Mirror and how does it work?

[Markdown Mirror](https://www.drupal.org/project/markdown_mirror) is a Drupal module that serves a Markdown-only version of your content at the same URL, with .md appended.

`https://example.com/blog/my-post      ->  the normal HTML page`
`https://example.com/blog/my-post.md   ->  the same content as Markdown`

The .md version keeps the page's own text, headings, links, and images. Navigation, scripts, and styling are stripped out.

It can also publish /llms.txt, an index of your enabled pages. Jeremy Howard of Answer.AI proposed the format in September 2024 ([llmstxt.org](https://llmstxt.org/)). Treat it as an optional extra rather than a ranking or citation lever.

In Ahrefs' server-log study of 137,000 domains, about 97% of published llms.txt files were never requested during May 2026. Google's AI optimization guidance, also from May 2026, lists llms.txt among the tactics you don't need for AI Overviews or AI Mode. We cover this in more depth in our [AEO/GEO playbook](https://www.specbee.com/blogs/how-to-get-content-cited-by-ai).

## Why should a Drupal site serve Markdown to AI tools?

An LLM working from raw HTML has to separate the real content from everything around it. Markdown removes that step.

The savings can be large. Cloudflare measured one of its own blog posts at 16,180 tokens in HTML and 3,150 tokens in Markdown, an 80% reduction ([Cloudflare, February 2026](https://blog.cloudflare.com/markdown-for-agents/)). Fewer tokens mean more of your page fits in a model's context window, at a lower cost per read.

The clearest benefit is for content you feed into AI yourself. If you're building a chatbot or retrieval pipeline on your own site, the .md pages give you clean source material without a custom export. You can also add [Drupal AI modules](https://www.specbee.com/blogs/drupal-ai-modules-for-smarter-website) to build those features into the site itself.

### Does it work with any Drupal content model?

Yes. Many content-to-Markdown tools need you to map specific fields for each content type.

Markdown Mirror never assumes that a particular field name, content type machine name, or View exists. The same install works on a site built with Paragraphs, plain body fields, or [Drupal Canvas](https://www.drupal.org/project/canvas) pages with no fixed field structure.

### Is it safe to expose publicly?

Dangerous links and markup are sanitized out of every .md page. Unpublished or access-restricted content is never exposed, because the module runs the same access checks as the HTML page.

Each .md copy also points search engines back to the HTML page with a “noindex” directive and a canonical link. Your real page stays the one that ranks.

## How is Drupal Markdown Mirror different from Cloudflare's Markdown for Agents?

Both serve Markdown to AI tools, but they work at different layers. [Cloudflare's Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) converts pages at the edge when an AI agent sends an Accept header asking for Markdown. Normal browser visits keep getting HTML, and coding agents like Claude Code and OpenCode already ask for Markdown this way.


 Drupal Markdown MirrorCloudflare Markdown for Agents**Where it runs**Inside DrupalAt Cloudflare's edge**How Markdown is requested**A .md URLAn Accept: text/markdown header**What gets converted**Only pages you opt in, by type, vocabulary, Canvas, or pathPages on the enabled Cloudflare zone**Access control**Runs Drupal's own access checksConverts whatever the edge serves**llms.txt**Generated from your enabled pagesNot included**What it needs**Drupal 10.3+ or 11, ComposerYour site behind Cloudflare, feature enabledMarkdown Mirror's documentation only covers .md URLs. It does not mention responding to the Accept header. If you want both request styles covered, the two can run side by side.

## What can you enable a Markdown version for?

- **Content types**: every page of the types you tick.
- **Taxonomy vocabularies**: every term, including a list of content tagged with it.
- **Drupal Canvas pages**: enabled site-wide with one toggle.
- **Specific pages**: opt in individual pages, one path per line, regardless of the settings above.
- **Excluded pages**: opt individual pages out, even if their content type, vocabulary, or Canvas pages are enabled. Useful for a thank-you page or one with private details.

![Markdown mirror module settings](https://www.specbee.com/sites/default/files/inline-images/markdown-mirror-settings.png)*Screenshot: the Markdown Mirror settings page — content types, vocabularies, and Canvas pages, each opted in individually.*

## How do you set up Markdown Mirror in Drupal?

Markdown Mirror supports Drupal 10.3 and later, plus Drupal 11. It needs the core Node module and Composer. Taxonomy, Metatag, Paragraphs, and Canvas are optional.

### Step 1: Install the module

Composer is required because it pulls in the [league/html-to-markdown](https://github.com/thephpleague/html-to-markdown) library automatically.

```plaintext
composer require drupal/markdown_mirror
drush en markdown_mirror
```

### Step 2: Grant the permission

Give the "Administer Markdown Mirror settings" permission to the right roles at `/admin/people/permissions`.

### Step 3: Choose what gets mirrored

Go to Configuration > Content authoring > Markdown Mirror (`/admin/config/content/markdown-mirror`). Tick content types and vocabularies, enable Canvas pages, and optionally list specific or excluded pages by path.

![Individually opted-in and excluded pages](https://www.specbee.com/sites/default/files/inline-images/configurable-frontmatter-fields.png)*Screenshot: individually opted-in and excluded pages, plus the configurable frontmatter fields shown at the top of each .md page.*

### Step 4: Configure the output

Under `llms.txt`, write an introduction and set how many pages each section lists. Turn on `/llms-full.txt` if you want the full content of every listed page in one file.

![llms.txt settings](https://www.specbee.com/sites/default/files/inline-images/llms-txt-settings.png)*Screenshot: llms.txt settings — introduction text, pages per section, and the optional llms-full.txt.*

### Step 5: Save and check the result

Visit any enabled page with .md appended to its URL.

To confirm the SEO safeguard, run the command below. Check the response for the noindex directive and the canonical link pointing to the HTML page.

`curl -I https://example.com/blog/my-post.md`

![A real page served as Markdown](https://www.specbee.com/sites/default/files/inline-images/real-page-served-as-markdown.png)*Screenshot: a real page served as Markdown — frontmatter at the top, followed by clean Markdown content and links.*

## Who should use Drupal Markdown Mirror?

- **Teams building a chatbot on their own content**. The .md pages give a retrieval pipeline clean, low-token source material without a custom export.
- **Sites that want AI tools to read them cleanly**. When an agent or AI tool fetches a page directly, a .md copy is quicker to parse and less likely to be misread.
- **Anyone who needs plain-text exports**. Documentation, content audits, and migrations all get a clean copy without hand-building one.

**Who doesn't need it**: If your only goal is appearing in Google's AI Overviews or AI Mode, this won't move the needle. Google has said Search doesn't use machine-readable AI files or Markdown versions of pages. Strong content and solid technical SEO matter more there.

## Final thoughts

Drupal Markdown Mirror leaves your site exactly as human visitors see it today. It adds a second, cleaner lane for AI tools, and you control it page by page.

It isn't a shortcut to AI citations. It is a cheap, low-risk way to give your own AI projects, and any tool that fetches your pages, a cleaner copy of your content.

Want your Drupal site ready for AI? [Talk to Specbee's Drupal AI team](https://www.specbee.com/services/drupal-ai-solutions) about making your content easier for LLMs to find, read, and use.

## Frequently Asked Questions

### Does Markdown Mirror change my existing Drupal pages?

No. Markdown Mirror adds a separate .md version at the same URL with .md appended, and your HTML pages stay untouched. Visitors, themes, and page layouts behave exactly as before. The module only responds when a request ends in .md for a page you have enabled, or when a tool requests /llms.txt or /llms-full.txt.

### Do I need to configure field mappings for each content type?

No field mappings are needed for any content type, including Drupal Canvas pages. Markdown Mirror reads stored content generically and never assumes that a specific field name, content type machine name, or View exists. Paragraphs, plain body fields, and Canvas pages with no fixed structure all convert cleanly. You enable Canvas site-wide with one toggle and can still exclude individual pages by path.

### Will the .md pages hurt my SEO?

No. Each .md copy carries a noindex directive and a canonical link back to the HTML page, so search engines treat the HTML version as authoritative. The Markdown copy exists for AI tools and crawlers and stays out of search rankings. Your existing pages keep their URLs, metadata, and rankings unchanged.

### Can unpublished or restricted content be exposed through the .md version?

No. Markdown Mirror runs the same access checks as the HTML page, so unpublished or access-restricted content never appears in the .md version. Dangerous links and markup are also sanitized out before output. If an anonymous visitor cannot see the HTML page, a crawler cannot see its Markdown copy either.

### Does llms.txt or Markdown help my content get cited by AI?

There's no evidence it does. In Ahrefs' study of 137,000 domains, about 97% of published llms.txt files were never requested in May 2026. Google has said Search doesn't use Markdown or machine-readable AI files. The proven value is cleaner, cheaper input for AI tools that fetch your pages and for your own chatbots. llms.txt is a low-cost optional extra.

### Does Markdown Mirror replace Cloudflare's Markdown for Agents?

Not exactly. They overlap, but they work differently. Cloudflare converts pages at the edge when an agent sends an Accept header asking for Markdown. Markdown Mirror runs inside Drupal, serves Markdown at .md URLs, and applies per-page opt-in and Drupal's access checks. Sites on Cloudflare can run both to cover both request styles.

### Does Markdown Mirror require a paid service or external API?

No paid service or external API is involved. Markdown Mirror runs entirely inside Drupal and converts content locally using the open-source league/html-to-markdown library, installed through Composer. Your content stays on your server during conversion, and you pay nothing per page or per request as your site grows.

### Which Drupal versions does Markdown Mirror support?

Markdown Mirror supports Drupal 10.3 and later, plus Drupal 11. It requires the core Node module and Composer. Taxonomy, Metatag, Paragraphs, and Canvas are optional, and Metatag adds a description field to each .md page's frontmatter. Check the project's releases page for the current version before you install.