Data Collection & Processing Pipeline
Twitter Data: 100 recent, original tweets were scraped headlessly
via Playwright.
YouTube Data: 5 recent speeches were downloaded via yt-dlp. The
audio was processed on local M-series silicon via the mlx-whisper speech-to-text
algorithm. The raw transcripts were then fed through a local Llama 3 (8B) model to
extract only the target politician's speech, stripping preamble and extraneous
dialogue.
Lexical Complexity (Vocabulary)
Lexical complexity evaluates the vocabulary richness and rarity. We compute this using NLTK, SpaCy, and textstat.
- Type-Token Ratio (TTR): Measures the proportion of unique words. Higher ratios signify a diverse vocabulary.
- Corrected TTR (CTTR): A variation of TTR that mathematically standardizes the score against the text's length to prevent penalization of longer speeches.
- MATTR: Moving-Average Type-Token Ratio mitigates the effect of text length by averaging TTR over rolling 50-word windows.
- MTLD: Measure of Textual Lexical Diversity evaluates the number of words generated before the TTR drops below a specific threshold (0.72).
- Lexical Density (LD): The ratio of content words (nouns, verbs, adjectives, adverbs) to total words.
- Rarest Word Frequency: Approximated using the SUBTLEX-US frequency list to find the least common word utilized, highlighting obscure vocabulary.
- Academic List %: The proportion of the text that overlaps with a standardized Academic Word List.
- Mean Word Length: Average number of characters per word.
- Syllables Per Word: Average phonetic density.
- Flesch Reading Ease: A formula outputting how easy the text is to read on a 0-100 scale (higher is easier).
- Flesch-Kincaid Grade: Maps the reading ease score to a U.S. school grade level.
Structural Complexity (Syntax)
Structural complexity measures the arrangement of words and syntactical depth. We leverage the SpaCy (en_core_web_sm) dependency parsing algorithm to extract sentence-level telemetry.
- Mean Length of Sentence (MLS): Longer sentences indicate coordinate/subordinate clause chaining.
- Mean Dependency Distance (MDD): The average linear distance between a word and its syntactic head modifier. Higher MDD implies greater cognitive load to parse.
- Mean Length of T-unit (MLT): Average word count of a T-unit (an independent clause and all its dependent modifiers).
- Clauses per T-Unit (C/T): The ratio of total clauses to T-units.
- Dependent Clauses per T-unit (DC/T): Measures how often dependent clauses are nested within T-units.
- Mean Length of Clause (MLC): Average word count per grammatical clause.
- Complex Nominals / Clause (CN/C): The density of complex nominals (nouns with modifiers, adjectives, or phrases attached) per clause, indicating informational density.
- Part-of-Speech Distribution: Analyzes the density of grammatical components. High density of Adjectives (ADJ) and Adverbs (ADV) can indicate highly descriptive speech. Prominent Nouns indicate dense object reference, increasing complexity.