When you build an AI agent or a RAG pipeline to read web pages, the easiest starting point is to fetch the HTML, strip the script and style tags, and pass the rest to the LLM. It feels like the safest way to ensure the model ‘sees’ everything on the page.
But in production, this approach fails quietly and consistently.
We analyzed the DOM structure of real production websites, specifically looking at a major footwear brand and a bedding retailer. Our findings show that sending markup to an LLM is not just inefficient-it actively destroys the model’s ability to answer the exact questions visitors ask.
The Markup Tax: Up to 97%
To understand the scale of the problem, we measured the character counts of real e-commerce pages.
On the footwear store, the simplified HTML (with basic scripts and styles removed) was 599,509 characters. On the bedding store, it reached 957,048 characters.
Yet, when we extracted the actual rendered, human-visible text from those same pages, it amounted to roughly 26,000 characters.
This means that 95.7% of the payload on the footwear store and 97.3% on the bedding store is markup noise: nested <div> wrappers, repetitive class names, tracking pixels, SVGs, and structural boilerplate.
To keep costs down and prevent LLM latency from spiking, most developers implement a character limit. In our early system design, we capped incoming HTML at 150,000 characters. When the DOM exceeded that limit, we cut it off.
Here is what happens when you apply a blind 150,000-character truncation on the bedding retailer’s page:
- Only 15.7% of the page survived the cut.
- 95% of the disclosure controls (such as tabs and accordions containing shipping info, materials, and return policies) were lost.
- The content inside the “Returns” accordion started at character offset 338,650-well past the truncation threshold.
When a user asks, “What is your return policy?” the agent fails because the relevant data was silently discarded before the LLM ever saw it.
Why “Just Filter the HTML” Fails
An obvious countermeasure is to target specific elements, like the <main> tag, or to strip attributes like class, data-*, and id.
Our tests showed that these heuristics do not solve the budget problem:
- The
<main>tag is still too large: On the bedding site, the<main>element made up 86.8% of the total body HTML. - Attribute stripping is insufficient: The most aggressive variant we tested-extracting only the
<main>tag and stripping all class, data, and ID attributes-was still 1.8 times over our 150,000-character budget.
If you rely on HTML, even cleaned HTML, you will eventually truncate critical data.
The Solution: Rendered Text + Hidden Sections
The fix is to stop sending markup entirely. Instead, compile the page into two distinct buckets:
- Rendered Visible Text: What the user actually sees on the screen.
- Hidden Content with Provenance: The content inside collapsed tabs, accordions, and
<details>elements that require a click to reveal.
This hidden content (such as care instructions, warranty info, or sizing charts) is precisely where the answers to user questions live. However, because it is hidden, naive scrapers often miss it, or fail to associate it with the correct button.
Rules of Provenance
When extracting hidden content, you must preserve the relationship between the control (the button or tab) and the panel it opens. Without this, the LLM cannot associate the text with its context.
Through testing, we found that you can only trust explicit accessibility relationships, such as aria-controls attributes or native <details>/<summary> elements.
We tested a popular alternative heuristic: assuming the panel is the immediate next sibling of the button in the DOM. In production, this assumption regularly failed, producing off-by-one errors and attributing the wrong text to a button. This led the LLM to give confidently incorrect answers. If a hidden panel does not have an explicit, programmatic link to a control, it is safer to mark it as “unattributed hidden content” rather than guessing.
Based on our production measurements, we settled on these practical allocation budgets:
- Visible Text: ~40,000 characters.
- Attributed Hidden Sections: ~20,000 characters (capped at ~3,000 characters per panel).
- Unattributed Hidden Sections: A much smaller, secondary budget.
On real sites, hidden sections with an identified control almost always contain high-value product details. Hidden sections without a control are usually site chrome, such as cart drawers, cookie banners, or newsletter sign-up popups.
Two Production Traps to Avoid
If you implement this extraction strategy, watch out for two common traps we encountered:
1. The textContent Script Leak
In many browser environments, reading element.textContent on a container still extracts the raw contents of child <script> and <style> tags if they haven’t been explicitly purged from the DOM tree first. If you omit this step, you will unintentionally send thousands of characters of minified JavaScript configuration objects or CSS rules directly into your LLM prompt, labeled as page text.
2. Cookie Consent Chrome
Third-party cookie-consent banners and privacy preference centers are highly structured. They often feature clean, valid, and accessible DOM structures with impeccable aria-controls markup.
Because of this setup, the clean aria-controls markup made the banner look like legitimate hidden content, so the model kept extracting topics from it during page analysis. On a WordPress test site, the LLM consistently dedicated 1 to 2 “topics” per analysis entirely to cookie preferences. These elements must be identified and filtered out deterministically at the scraping level, using known class patterns or selector lists, rather than relying on prompt instructions to ignore them.
Conclusion
Improving your RAG pipeline or web agent isn’t about finding a better way to compress raw HTML into a prompt. It is about discarding the structural noise entirely and intelligently rebuilding the page context from rendered visible text and mapped hidden controls.
This robust, context-aware extraction architecture is exactly how AskMyWebsite AI reads and understands its customers’ sites.