Every output below was printed by running the code with the library.
Words and sentences
Unicode's rules keep "don't", "3.5" and "1,000" whole, give each Chinese or Japanese character a word of its own, and don't end a sentence at "3.5" or "e.g.".
let note = "Don't skip it: the beam's depth is 3.5 m. Check e.g. the 1,000 kN case first! 👨👩👧 approves."
print(text.words(note))
print(text.wordCount(note))
print(text.sentences(note))
["Don't", "skip", "it", "the", "beam's", "depth", "is", "3.5", "m", "Check", "e.g", "the", "1,000", "kN", "case", "first", "approves"]
17
["Don't skip it: the beam's depth is 3.5 m. ", "Check e.g. the 1,000 kN case first! ", "👨👩👧 approves."]
Characters as a reader counts them
Tessel counts code points. A family emoji is five of them but one character to a reader, so carets and Backspace should move with nextGrapheme and previousGrapheme.
print("👨👩👧".count)
print(text.graphemeCount("👨👩👧"))
print(text.graphemes("ñ👍🏽🇵🇭"))
Hyphenation
TeX's American English patterns and Liang's method. hyphenateText puts soft hyphens into a whole text for a layout to break at.
let h = text.englishHyphenator()
print(h.hyphenate("reinforcement"))
print(h.parts("hyphenation"))
re-in-force-ment
["hy", "phen", "ation"]
Spelling
The spell checker from the Quill word processor: the system's word list plus English endings, with suggestions ranked by how likely each slip is.
var sp = text.speller(text.systemWordList(), user: "Tessel\n")
for m in sp.misspellings("Teh beam was stoped by Tessel's sensor.") {
print("{m.word} at {m.from}-{m.to}: {sp.suggest(m.word, limit: 3)}")
}
print(text.autocorrect(text.autocorrectDefaults(), "Teh") ?? "-")
Teh at 0-3: ["The", "Ten", "Tea"]
stoped at 13-19: ["stopped", "stiped", "stooped"]
The
Fuzzy matching
"Did you mean", and a command palette that finds "Open File" from "opfl".
print(text.levenshtein("kitten", "sitting"))
print(text.closest("lenght", ["length", "height", "width"]))
print(text.fuzzyFilter("opfl", ["Open File", "Open Folder", "Copy File", "Toggle Fold"]))
3
["length", "height"]
["Open File", "Open Folder", "Copy File"]
Normalization
"é" can be one code point or two. nfc makes them the same, and nfkc also turns ligatures and full-width letters into plain ones.
let typed = "Cafe" + text.fromCodePoints([769])
print(typed == "Café")
print(text.nfc(typed) == "Café")
print(text.nfkc("file №2"))
Search
BM25 ranking, as in Lucene, with stemming and stopwords. Phrases go in quotes, and prefix: true searches while the user types.
var help = text.SearchIndex()
let pages = [
"Simply supported beams: bending moment and shear force diagrams.",
"Reinforced concrete columns under axial load and bending.",
"Steel beam connections: bolted and welded.",
"The moment of inertia of a section.",
]
for p in pages {
let ignored = help.add(p)
}
for hit in help.search("beam bending") {
print("{hit.document} {hit.score.formatted(decimals: 3)} {pages[hit.document]}")
}
print(help.search("\"bending moment\"").count)
print(help.search("reinf", prefix: true)[0].document)
0 0.531 Simply supported beams: bending moment and shear force diagrams.
2 0.327 Steel beam connections: bolted and welded.
1 0.304 Reinforced concrete columns under axial load and bending.
1
1
Stems and readability
Snowball's English stemmer, and the Flesch formulas with syllables counted by spelling rules.
print(text.stems("The engineers were checking the bending of reinforced beams"))
let st = text.textStats("The cat sat on the mat. It was happy. Readability formulas estimate the education needed to understand a passage.")
print("{st.sentences} sentences, {st.words} words, {st.syllables} syllables")
print(st.fleschReadingEase().formatted(decimals: 1))
print(st.fleschKincaidGrade().formatted(decimals: 1))
["the", "engin", "were", "check", "the", "bend", "of", "reinforc", "beam"]
3 sentences, 19 words, 35 syllables
44.6
8.6