API Reference#

Language Models#

class rustling.lm.LanguageModel(*, order: int, smoothing: str = 'mle', gamma: float = 1.0)#

An n-gram language model.

Supports MLE, Lidstone, and Laplace smoothing methods.

fit(sents: Sequence[Sequence[str]]) → None#

Train the language model on tokenized sentences.

Each sentence is a list of tokens. The model extracts n-grams of all orders from 1 to the model order and counts their occurrences. Sentences are automatically padded with <s> and </s> tokens.

Parameters:

sents – An iterable of tokenized sentences.

generate(*, num_words: int = 1, text_seed: Sequence[str] | None = None, random_seed: int | None = None) → list[str]#

Generate words from the language model.

Uses weighted random sampling from the conditional distribution. Generation stops early if </s> (end-of-sentence) is sampled or if no continuations are available for the current context.

Parameters:

  • num_words – Number of words to generate.

  • text_seed – Seed text (context to start from). Defaults to beginning-of-sentence context.

  • random_seed – Random seed for reproducibility.

Returns:

A list of generated words.

Raises:

ValueError – If the model has not been fitted yet.

logscore(word: str, context: Sequence[str] | None = None) → float#

Return the log (base 2) probability of a word given a context.

Maps out-of-vocabulary words to <UNK> via the vocabulary.

Parameters:

  • word – The word to score.

  • context – The preceding context words.

Returns:

log2(P(word | context)). Returns negative infinity if probability is 0.

Raises:

ValueError – If the model has not been fitted yet.

property order: int#

The order of the n-gram model.

score(word: str, context: Sequence[str] | None = None) → float#

Return the probability of a word given a context.

Maps out-of-vocabulary words to <UNK> via the vocabulary.

Parameters:

  • word – The word to score.

  • context – The preceding context words.

Returns:

The probability P(word | context).

Raises:

ValueError – If the model has not been fitted yet.

unmasked_score(word: str, context: Sequence[str] | None = None) → float#

Return the probability of a word given a context, without OOV mapping.

Unlike score, this method does not map out-of-vocabulary words to <UNK>.

Parameters:

  • word – The word to score.

  • context – The preceding context words.

Returns:

The probability P(word | context).

Raises:

ValueError – If the model has not been fitted yet.

property vocab_size: int#

The vocabulary size (including special tokens).

class rustling.lm.MLE(*, order: int)#

Maximum Likelihood Estimation language model.

An n-gram language model with no smoothing.

class rustling.lm.Lidstone(*, order: int, gamma: float)#

Lidstone (additive) smoothing language model.

An n-gram language model with Lidstone smoothing, which adds a constant gamma to all counts.

class rustling.lm.Laplace(*, order: int)#

Laplace (add-one) smoothing language model.

An n-gram language model with Laplace smoothing (Lidstone with gamma=1).

Word Segmentation#

class rustling.wordseg.LongestStringMatching(*, max_word_length: int)#

Longest string matching segmenter.

This model constructs predicted words by moving from left to right along an unsegmented sentence and finding the longest matching words, constrained by a maximum word length parameter.

fit(sents: Sequence[Sequence[str]]) → None#

Train the model with the input segmented sentences.

No cleaning or preprocessing (e.g., normalizing upper/lowercase, tokenization) is performed on the training data.

Parameters:

sents – An iterable of segmented sentences (each sentence is a sequence of words).

predict(sent_strs: Sequence[str]) → list[list[str]]#

Segment the given unsegmented sentences.

Parameters:

sent_strs – An iterable of unsegmented sentences.

Returns:

A list of segmented sentences.

class rustling.wordseg.RandomSegmenter(*, prob: float)#

A random segmenter.

Segmentation is predicted at random at each potential word boundary independently for a given probability. No training is required.

fit(sents: Sequence[Sequence[str]]) → None#

Training is not required for RandomSegmenter.

Parameters:

sents – Unused.

Raises:

NotImplementedError – Always, since no training is needed.

predict(sent_strs: Sequence[str]) → list[list[str]]#

Segment the given unsegmented sentences.

Parameters:

sent_strs – An iterable of unsegmented sentences.

Returns:

A list of segmented sentences.

Part-of-Speech Tagging#

class rustling.tagging.AveragedPerceptronTagger(*, frequency_threshold: int = 10, ambiguity_threshold: float = 0.95, n_iter: int = 5)#

A part-of-speech tagger using an averaged perceptron model.

This is a modified version based on the textblob-aptagger codebase (MIT license), with original implementation by Matthew Honnibal.

property classes: set[str]#

Get the set of POS tag classes.

Returns:

A set of all tag classes in the model.

load(path: str) → None#

Load a model from a JSON file.

Parameters:

path – The path where the model, stored as a JSON file, is located.

Raises:

  • FileNotFoundError – If the file at the given path does not exist.

  • EnvironmentError – If the file cannot be read as a tagger model.

save(path: str) → None#

Save the model to a JSON file.

Parameters:

path – The path where the model will be saved as a JSON file.

tag(words: Sequence[str]) → list[str]#

Tag the words.

Parameters:

words – A segmented sentence or phrase, where each word is a string.

Returns:

The list of predicted tags.

property tagdict: dict[str, str]#

Get the tag dictionary.

Returns:

A dictionary mapping words to their most likely tags.

train(tagged_sents: Sequence[Sequence[tuple[str, str]]]) → None#

Train a model.

Parameters:

tagged_sents – A list of segmented and tagged sentences for training. Each sentence is a sequence of (word, tag) tuples.

property weights: dict[str, dict[str, float]]#

Get the model’s weights dictionary.

Returns:

A dictionary mapping features to their weight vectors.