text for Tessel

A Tessel library

Words, sentences and characters, the way Unicode defines them.

Split text into what a reader calls characters, words and sentences. Normalize it, hyphenate it, check its spelling, match what someone half-typed, search it, and score how hard it is to read. Every part is checked against Unicode's own test files or a Python library.

tessel add text

MIT licence · Tessel 0.18 or later · no dependencies · Unicode 17

text.words and misspellings

Don't skip it: the beam's depth is 3.5 m. Chekc e.g. the 1,000 kN case! 👨‍👩‍👧

↓ the same text, measured
What text gives
  • graphemeCount("👨‍👩‍👧") → 1, where count gives 5
  • hyphenate("reinforcement") → re-in-force-ment
  • suggest("Chekc") → Check, …
  • stem("bending") → bend
  • nfkc("file №2") → file No2

Examples

Every output below was printed by running the code with the library.

Words and sentences

Unicode's rules keep "don't", "3.5" and "1,000" whole, give each Chinese or Japanese character a word of its own, and don't end a sentence at "3.5" or "e.g.".

let note = "Don't skip it: the beam's depth is 3.5 m. Check e.g. the 1,000 kN case first! 👨‍👩‍👧 approves."
print(text.words(note))
print(text.wordCount(note))
print(text.sentences(note))
["Don't", "skip", "it", "the", "beam's", "depth", "is", "3.5", "m", "Check", "e.g", "the", "1,000", "kN", "case", "first", "approves"]
17
["Don't skip it: the beam's depth is 3.5 m. ", "Check e.g. the 1,000 kN case first! ", "👨‍👩‍👧 approves."]

Characters as a reader counts them

Tessel counts code points. A family emoji is five of them but one character to a reader, so carets and Backspace should move with nextGrapheme and previousGrapheme.

print("👨‍👩‍👧".count)
print(text.graphemeCount("👨‍👩‍👧"))
print(text.graphemes("ñ👍🏽🇵🇭"))
5
1
["ñ", "👍🏽", "🇵🇭"]

Hyphenation

TeX's American English patterns and Liang's method. hyphenateText puts soft hyphens into a whole text for a layout to break at.

let h = text.englishHyphenator()
print(h.hyphenate("reinforcement"))
print(h.parts("hyphenation"))
re-in-force-ment
["hy", "phen", "ation"]

Spelling

The spell checker from the Quill word processor: the system's word list plus English endings, with suggestions ranked by how likely each slip is.

var sp = text.speller(text.systemWordList(), user: "Tessel\n")
for m in sp.misspellings("Teh beam was stoped by Tessel's sensor.") {
    print("{m.word} at {m.from}-{m.to}: {sp.suggest(m.word, limit: 3)}")
}
print(text.autocorrect(text.autocorrectDefaults(), "Teh") ?? "-")
Teh at 0-3: ["The", "Ten", "Tea"]
stoped at 13-19: ["stopped", "stiped", "stooped"]
The

Fuzzy matching

"Did you mean", and a command palette that finds "Open File" from "opfl".

print(text.levenshtein("kitten", "sitting"))
print(text.closest("lenght", ["length", "height", "width"]))
print(text.fuzzyFilter("opfl", ["Open File", "Open Folder", "Copy File", "Toggle Fold"]))
3
["length", "height"]
["Open File", "Open Folder", "Copy File"]

Normalization

"é" can be one code point or two. nfc makes them the same, and nfkc also turns ligatures and full-width letters into plain ones.

let typed = "Cafe" + text.fromCodePoints([769])
print(typed == "Café")
print(text.nfc(typed) == "Café")
print(text.nfkc("file №2"))
false
true
file No2

Search

BM25 ranking, as in Lucene, with stemming and stopwords. Phrases go in quotes, and prefix: true searches while the user types.

var help = text.SearchIndex()
let pages = [
    "Simply supported beams: bending moment and shear force diagrams.",
    "Reinforced concrete columns under axial load and bending.",
    "Steel beam connections: bolted and welded.",
    "The moment of inertia of a section.",
]
for p in pages {
    let ignored = help.add(p)
}
for hit in help.search("beam bending") {
    print("{hit.document} {hit.score.formatted(decimals: 3)}  {pages[hit.document]}")
}
print(help.search("\"bending moment\"").count)
print(help.search("reinf", prefix: true)[0].document)
0 0.531  Simply supported beams: bending moment and shear force diagrams.
2 0.327  Steel beam connections: bolted and welded.
1 0.304  Reinforced concrete columns under axial load and bending.
1
1

Stems and readability

Snowball's English stemmer, and the Flesch formulas with syllables counted by spelling rules.

print(text.stems("The engineers were checking the bending of reinforced beams"))
let st = text.textStats("The cat sat on the mat. It was happy. Readability formulas estimate the education needed to understand a passage.")
print("{st.sentences} sentences, {st.words} words, {st.syllables} syllables")
print(st.fleschReadingEase().formatted(decimals: 1))
print(st.fleschKincaidGrade().formatted(decimals: 1))
["the", "engin", "were", "check", "the", "bend", "of", "reinforc", "beam"]
3 sentences, 19 words, 35 syllables
44.6
8.6

Checked against the sources

A script reads Unicode's data and test files, runs Python libraries, and saves what they give for the tests to compare against: tessel run tests checks it all in about seven seconds.

1,114,112code points, each with the category UnicodeData.txt gives it
3,222cases of Unicode's own grapheme, word and sentence break tests
20,034cases of Unicode's NormalizationTest.txt
42,649words stemmed exactly as Snowball's own list has them
14,000words hyphenated as pyphen does with the same patterns
400search queries scored as bm25s scores them
1,509pairs of distances as rapidfuzz gives them
10 msto count the words of 100 pages

Reference

Positions are code point indexes, the ones Tessel's substring takes. The README has more detail.

Characters and normalization

FunctionGives
codePoints(s), codePoint(ch), fromCodePoints(cps)Text to code points and back.
category(c)The General_Category: "Lu", "Nd", "Po"…
isLetter, isUppercase, isLowercase, isMark, isDigit, isNumber, isPunctuation, isSymbol, isSpace, isAlphanumericWhat kind of character a code point is.
caseFold(s), sameIgnoringCase(a, b)Comparing without case: "Straße" and "STRASSE" are the same.
nfc, nfd, nfkc, nfkd, combiningClassUnicode normalization forms.

Splitting

FunctionGives
graphemes, graphemeCount, graphemeBreaksCharacters as a reader counts them.
nextGrapheme(s, at), previousGrapheme(s, at)Where arrow keys and Backspace go.
words, wordCount, wordPieces, wordBreaks, wordAt(s, at)Words, and every piece between word breaks with from, to and isWord.
nextWordEnd, nextWordStart, previousWordStartMoving by words with Option or Ctrl.
sentences, sentenceBreaksSentences, each with the spaces after it.

Hyphenation and spelling

FunctionGives
englishHyphenator(left:, right:)A Hyphenator with points, hyphenate, parts and hyphenateText.
hyphenator(patterns:, exceptions:), hyphenatorFromTex(tex)Any language's patterns, such as those of hyph-utf8.
speller(wordList, user:), systemWordList()A Speller with isCorrect, suggest, misspellings, addWord, ignoreAll, ignoreOnce and resetIgnored.
autocorrect(table, word), autocorrectDefaults()About 140 typing slips put right, capitals kept.

Matching, search and readability

FunctionGives
levenshtein, damerauLevenshtein, similarity, jaro, jaroWinklerHow alike two texts are.
closest(word, choices), fuzzyMatch(pattern, item), fuzzyFilter(pattern, items)"Did you mean", and matching typed letters in order.
stem(word), stems(text)Snowball English stems.
SearchIndexadd, search(query, limit:, prefix:), addTerms, searchTerms.
textStats(text), syllables(word)TextStats with Flesch, Flesch-Kincaid, Gunning fog, SMOG, Coleman-Liau, ARI and reading time.