New Research: Why HTML-to-LLM Scraping Fails

The Core Findings of Our HTML-to-LLM Study

AskMyWebsite AI has published its first research analysis, a production study examining why feeding raw HTML directly to large language models (LLMs) fails in practice. The analysis highlights key vulnerabilities in standard scraping techniques that impact the quality of AI concierge services.

Our research covers three primary technical failure modes:

• Markup Noise: On real production pages, 95.7% to 97.3% of the simplified HTML payload is markup noise rather than visible text. For example, one retailer’s page contained 599,509 characters of simplified HTML compared to only about 26,000 characters of visible text.
• Payload Truncation: With a standard 150,000-character truncation limit, only 15.7% of another retailer’s page survived the ingestion process.
• Interactive Content Loss: Approximately 95% of disclosure controls-such as accordions and interactive tabs-were lost once the page hit the truncation limit, hiding vital information from the LLM.

Access the Analysis

The methodology used in this study is fully reproducible. You can read the complete, detailed breakdown here: Why HTML-to-LLM Scraping Fails: A Production Analysis.

A PDF preprint of the study is archived on Zenodo: DOI 10.5281/zenodo.21797856 (CC BY 4.0).