In his article "Can you read as well as a fifth-grader? Check the formula" columnist Carl Bialik discusses the readability formulas included in word processing software such as Microsoft Word, and discusses the value of the mechanical application of such formulas. He has opinions from both supporters and detractors of readability formulas, and counts both Karen Schriver, and Professor J. Peter Kincaid, one of the original instigators of the Flesch-Kincaid formula used in Microsoft Word, amongst those who question the value of the formulas.
I recently read a far more sustained attack on readability formulas, and in particular on their "dubious use" by the UK's Department for Education and Skills (DfES), written by Martin Cutts of the Plain Language Commission. In Writing by numbers: are readability formulas to clarity what karaoke is to song? Cutts complains that public bodies like the DfES use readability formulas as part of their propaganda and ignore the obvious shortcomings of what he terms "crude" tests. He notes that the main problem with readability tests is that:
[those] who apply them uncritically tend to assume that any 10-sentence [passage] with, say, 12 polysyllabic words is as good and clear as any other with 12 polysyllabic words. But its grammar and punctuation may be poor and its message muddled, ambiguous or misleading. Such findings are only likely to emerge after usability testing (not readability testing) or editorial scrutiny or both.
In an effort to offer an alternative to the flawed readability formulas, Cutts and the Plain Language Commission have published a Plain English Lexicon, available free of charge for download from their website. The lexicon helps you find out whether the words you write will be easily understood, by comparing their grade level in the US Living Word Vocabulary (LWV) and their frequency in the UK British National Corpus (BNC). Words that have low LWV grade levels and high BNC frequencies should be easily understood by readers on both side of the Atlantic, according to Cutts. As long as the spelling isn't too different over there.