AI-Friendly Text Cleaner

Converts text to Unicode Normalization Form C, ensuring consistent representation of accented characters, combining marks, and decomposed sequences. Prevents AI models from treating visually identical characters as different tokens (é vs e + ´ combining accent). Critical for multilingual text processing.

Strips all HTML markup including <div>, <span>, <script>, <style>, and 100+ other tags. Decodes HTML entities (&nbsp; → space, &quot; → ") to clean text. Essential for web scraping workflows—scraped content contains 40-60% HTML overhead that wastes tokens. Extracts only readable text for AI consumption.

Removes Markdown formatting: **bold**, *italic*, # headers, [links](url), , code blocks, lists, and blockquotes. Keeps only plain text content. Note: Claude preset preserves Markdown for structure since Claude handles it well—use ChatGPT or General presets for complete Markdown removal for token optimization.

Converts curly quotes (""), guillemets (« »), and 12+ other quote styles to straight ASCII quotes (" and '). Microsoft Word, Google Docs, and Apple Pages insert smart quotes automatically—these consume extra tokens and can confuse AI parsing. Normalization saves 2-3 tokens per 100 quotes while ensuring consistent interpretation.

Collapses multiple spaces, tabs, and mixed whitespace into single spaces. Removes trailing whitespace from line endings. Excessive whitespace is common in copy-pasted content from PDFs, emails, and formatted documents—can add 15-25% token overhead. Our normalization reduces this waste while maintaining text readability and structure.

Normalizes line endings to Unix LF (from Windows CRLF or Mac CR), removes excessive blank lines (3+ consecutive → 2), and ensures consistent paragraph spacing. PDFs often export with 5-10 blank lines between paragraphs—this wastes tokens on empty space. Configurable maximum line breaks (1-10) for different formatting needs.

Eliminates 16 types of invisible Unicode characters: zero-width spaces (U+200B), soft hyphens (U+00AD), BOM markers (U+FEFF), directional formatting marks, and other hidden characters. These are invisible to humans but count as tokens—found in 30% of web-scraped content and copy-pasted text from certain websites. Our tool removes them completely.

Strips all emojis across 15 Unicode ranges: emoticons (😀), symbols (🚀), flags (🇺🇸), and more. Emojis consume 2-4 tokens each despite adding little semantic value for text analysis. Disable for social media sentiment analysis where emojis carry meaning; enable for formal documents and API optimization where they're noise.

Removes verbal fillers: "um", "uh", "like", "you know", "basically", "actually", "literally" (17 patterns total). Perfect for cleaning transcripts from Zoom, Google Meet, or podcast audio-to-text conversions. Transcripts contain 30-50% filler words that waste tokens and obscure meaning—removal improves AI summarization quality significantly.

Converts 20+ bullet styles (•, ●, ▪, →, ➤, etc.) to standard ASCII dash (-). Maintains list structure while using minimal tokens. Different applications use different bullet Unicode characters—normalization ensures consistency and reduces tokenization overhead for list items in prompts and documents.

Converts spelled-out numbers to digits: "one hundred" → "100", "twenty" → "20". Removes currency symbols ($, €, £). Useful for financial documents, reports, and data analysis where numeric consistency matters. Digits tokenize more efficiently than spelled numbers—"100" = 1 token vs "one hundred" = 2 tokens.

Intelligently detects and preserves URLs and email addresses during cleaning (configurable). Extracts and counts found URLs/emails for reference. Critical for web content and business documents where links must remain intact. Alternative: enable removal for summarization tasks where URLs add no value but consume tokens.

Detects and removes consecutive duplicate lines automatically. Common in auto-generated content, logs, and poorly formatted exports where same text repeats 2-10 times. Saves significant tokens in data dumps and extracted content while preserving unique information and intentional repetition for emphasis (non-consecutive duplicates kept).

Standardizes various dash types (em dash —, en dash –) to hyphens, converts ellipsis (…) to three periods, removes excessive punctuation (!!!! → !, ??? → ?). Reduces exotic Unicode punctuation to ASCII equivalents for consistent tokenization across AI models. Maintains readability while optimizing for token efficiency.

Removes ASCII control characters (except tab/newline): NULL bytes, backspace, form feed, vertical tab, and other non-printable characters that cause parsing errors. Often present in binary file conversions, corrupted text exports, and certain database dumps. Ensures clean, AI-compatible output that processes without errors.