cellmage.utils.token_utils

Token counting utilities for estimating token usage in LLM prompts.

This module provides functions for accurately counting tokens using tiktoken or falling back to heuristic methods when tiktoken is not available.

Functions

count_tokens(text)	Count tokens in text using the default token counter.
count_tokens_for_messages(messages[, llm_client])	Count tokens in a list of messages, with role-based categorization.
count_tokens_in_code(code)	Count tokens in code using the default token counter.
count_tokens_in_dict(data)	Count tokens in a dictionary using the default token counter.
count_tokens_in_list(data)	Count tokens in a list using the default token counter.

Classes

TokenCounter([encoding_name, ...])

A configurable token counting utility for accurate token estimation.