AI-Assisted Writing Feedback: Concepts Worth Exploring

My AI experiments for WriteDaily have progressed from conceptual exploration to working prototypes. Here’s an honest update on what’s working and what isn’t.

The Prototype

I built a small feedback engine that runs a quantised Mistral 7B model locally on the WriteDaily server. It processes an entry after save, producing structured feedback in three areas:

Pattern recognition across entries. The model compares today’s entry against your 30-day baseline and flags anomalies: “Your use of certainty language (‘definitely’, ‘certainly’, ‘without doubt’) is 3x your monthly average. This entry reads as unusually confident.”

Thematic summarisation. A weekly digest that identifies recurring themes without exposing raw entry text: “This week’s writing focused on: decision-making (4 entries), team dynamics (2 entries), personal health (1 entry).”

Tone and style observations. Sentence variety scores, reading level estimates, passive vs. active voice ratios — the kind of mechanical feedback that complements LIWC’s psychological insights.

What’s Working

The privacy-preserving architecture is solid. Everything runs locally — no API keys, no external services, no data egress. The quantised model fits in 4GB of RAM and processes a 750-word entry in about 2.5 seconds.

The structured output from the two-pass approach (extract data, then narrate) produces consistent, hallucination-resistant feedback. I haven’t seen an invented pattern in over 500 test entries.

The LLM feedback complements LIWC well. It covers ground LIWC can’t — narrative patterns, stylistic observations, cross-entry comparisons. LIWC handles the psychological profiling. Together, they’re more useful than either alone.

What’s Not Ready

Running inference on every entry save would increase server costs meaningfully. The current prototype processes entries on a queue with a ~5-second delay, which works but adds complexity.

“AI-powered writing feedback” sets expectations the prototype can’t fully meet. The model is good at pattern recognition, not literary criticism. Positioning matters.

Models evolve fast. A Mistral 7B integration built today would need updating within months. WriteDaily has always prioritised low-maintenance infrastructure. Adding an LLM dependency changes that calculus.

My Stance

I’m continuing to develop the prototype, but I’m not shipping it on WriteDaily yet. The technology is exciting, the privacy architecture is sound, and the feedback quality is genuinely useful. But “useful prototype” and “production feature” are different things.

When it ships — if it ships — it will be opt-in, privacy-preserving, and complementary to the LIWC analysis users already trust. No rush. WriteDaily has been running for over a decade. A few more months of careful development won’t hurt.