Most formatting guidance describes what parsers expect. This misses why parsers expect it and where the actual parsing boundaries create content visibility failures.
The tokenization boundary problem affects meaning extraction in ways formatting can prevent. Language models tokenize text into subword units before processing. Unusual formatting can split tokens unexpectedly. “AI-powered” might tokenize as [“AI”, “-“, “power”, “ed”] or [“AI”, “-powered”] depending on model. Hyphenation, special characters, and unusual spacing create tokenization variance that affects embedding consistency. Use conventional punctuation and spacing. Avoid em dashes, special Unicode characters, and unusual formatting that might tokenize inconsistently across models.
The heading hierarchy establishes document graph structure that parsing systems rely on. H1 creates root node; H2 creates primary branches; H3 creates secondary branches. Skip levels (H1 to H3 without H2) create ambiguous graph structure that parsers must resolve through heuristics. Parsers may mis-nest sections, assigning content to wrong parent sections. Maintain strict heading hierarchy not for SEO but for accurate structural parsing.
The paragraph boundary problem involves more than visual formatting. HTML paragraph tags create explicit boundaries. But within paragraphs, sentence boundaries matter for chunk segmentation. Many RAG systems chunk at paragraph boundaries, but some chunk at sentence boundaries within paragraphs. Ensure each sentence is self-contained enough to provide value if extracted independently. The sentence is the atomic semantic unit, not the paragraph.
The list semantics divergence creates parsing inconsistencies. Ordered lists imply sequence dependency; unordered lists imply categorical grouping. Parsers may apply different extraction logic to each. Ordered list items might be extracted as procedural steps; unordered items as independent options. Use list types that match your semantic intent. If order matters, use ordered lists. If items are alternatives, use unordered lists. Mismatched semantics confuse extraction.
Table structure enables structured data extraction that prose cannot provide. Parsers identify tables and attempt cell-by-cell extraction with header association. But tables only work if properly marked up. CSS-styled divs that look like tables to humans parse as prose to machines. Use semantic table elements (table, thead, tbody, th, td) for any tabular data you want extracted structurally.
The definition pattern provides a parsing hook for concept extraction. “Term: definition” structures, whether in definition lists (dl, dt, dd) or formatted as “X is Y” sentences, signal extractable concepts. AI systems looking for definitions of terms can identify and extract these patterns. If you want your definition of a term to be the definition AI systems cite, use explicit definition patterns.
The quote attribution pattern affects source extraction. Blockquotes with clear attribution provide extractable quotes associated with sources. Quote patterns in prose without clear attribution may be extracted without proper sourcing. If quote attribution matters, use blockquote elements with explicit attribution. If you don’t want content treated as quotes, avoid quotation mark patterns that might be misidentified.
The date format problem affects temporal parsing critical for freshness signals. “12/01/2024” is ambiguous: December 1 or January 12? Parsers may interpret incorrectly based on locale assumptions. ISO 8601 (2024-12-01) is unambiguous. For any date that affects freshness perception, use unambiguous format. Publication dates, update dates, and event dates all merit ISO format.
The semantic HTML principle extends beyond accessibility. Semantic elements (article, section, aside, nav, header, footer) signal document structure. Parsers can distinguish main content from navigation from supplementary material. Generic divs provide no structural signal. Content in main or article elements receives content treatment; content in nav or aside may be filtered as non-content. Use semantic elements to signal what’s content versus what’s interface.
The encoding and character consistency prevents parsing corruption. Mixed encodings, unusual Unicode characters, and encoding declaration mismatches cause parsing errors that may silently corrupt content or cause parsing failures. Serve UTF-8 with explicit charset declaration. Avoid unusual Unicode characters. Test that served content matches intended content by examining raw bytes.