LLM Token Tracking & Context Size: A Complete Guide

by Marco 52 views

Hey everyone! Ever found yourself bumping up against the limits of your favorite Large Language Models (LLMs)? Maybe you've had a brilliant story idea cut short, or an API call rejected because your prompt was too long? It's a common problem, and today, we're diving deep into how to tackle it. We're talking about token count tracking and context size management for your LLM endpoints. This guide breaks down the problem, current solutions, proposed features, technical implementations, and future enhancements, ensuring you're well-equipped to build and use LLMs effectively. Let's get started, shall we?

The Problem: LLM Endpoint Limitations

Different LLM endpoints have varying token limits and context sizes, and these limits can catch users off guard. Understanding and managing these limits is crucial to avoid unexpected API errors or incomplete responses. Think of it like having a limited-size canvas for your artwork. If your picture is too large, it won't fit! The same applies to LLMs; if your prompt exceeds the allowed context, the LLM will struggle. This is the core problem we're addressing.

Current State of Affairs

Currently, many LLM applications are missing key features that help users understand and manage their token usage. This leads to a lack of transparency and control. Here’s what's typically missing:

  • No token count visualization in the UI: Users have no direct feedback on how many tokens their prompts consume.
  • No context size awareness for different endpoints: Applications often lack the ability to recognize or adapt to different LLM endpoints and their unique context sizes.
  • Users can unknowingly exceed context limits: Users may inadvertently create prompts that exceed the LLM's capacity, resulting in truncated responses or failed API calls. This can be frustrating and lead to wasted resources.

Proposed Features: Enhancing LLM User Experience

To solve the problems above, we need to improve the user experience with LLMs. This means giving users clear visibility into their token usage and context limits. Here’s a look at what we’re proposing.

1. KoboldCPP Integration: Native API Power

For those using KoboldCPP, we can directly leverage its native API endpoints. This offers a precise, real-time solution. Here's how it works:

  • Tokenization Endpoint: We'll query the /api/extra/tokencount endpoint to get an accurate token count for any text. This allows for precise monitoring.
  • Context Size Detection: Using the /api/v1/info endpoint, we can detect the current max context size of the LLM. No more guesswork!
  • Visual Indicator: We'll introduce a progress bar (dark → light) to visually indicate context usage. This simple, intuitive display immediately shows how much of the context is in use.
  • Real-time Updates: As the user types and content changes, the token count will update in real-time, so the user always knows where they stand.

2. OpenAI/Generic API Support: Flexibility and Adaptability

Not everyone uses KoboldCPP. For those who use OpenAI-compatible or generic API endpoints, we have a flexible approach:

  • Token Estimation: We'll use the OpenAI tokenization API (/v1/tokenize) whenever it's available, for accurate token counts.
  • Manual Configuration: Users will be able to add a "Max Context Size" field to their LLM profile settings. This allows for custom settings.
  • Fallback Estimation: When direct tokenization isn't available, we'll use character-based estimation (about 4 characters per token) as a fallback. It's not perfect, but it provides a good starting point.
  • Warning System: We'll implement a warning system to alert the user before sending a prompt if the estimated tokens exceed the configured limit. This prevents unexpected errors.

3. UI Components: User-Friendly Design

Effective UI components are crucial for a seamless user experience. Here’s how the token usage information will be displayed:

Token Usage Indicator

Imagine a clear, concise indicator like this:

Context Usage: ████████░░ 1,024 / 2,048 tokens (50%)

This will be visible in both document mode and writer mode.

Profile Settings Enhancement

In the LLM profile settings, the UI will be updated:

LLM Profile Configuration:
┌─────────────────────────────────┐
│ Name: My KoboldCPP Server       │
│ Endpoint: http://localhost:5001 │
│ Type: [KoboldCPP] [OpenAI] ...  │
│ Max Context: [Auto-detect] ✓    │ ← Auto for KoboldCPP
│ Max Context: [____2048____]     │ ← Manual for others
└─────────────────────────────────┘

Technical Implementation: Diving into the Code

Now, let’s get into the nitty-gritty of the technical implementation. This involves modifying code, integrating APIs, and designing effective UI components.

KoboldCPP API Endpoints: The Heart of the System

Here are the key endpoints we’ll use to interact with KoboldCPP:

  1. Token Count: POST /api/extra/tokencount

    To get the token count for a given text, we will send a POST request to this endpoint with a JSON payload like this:

    { "prompt": "text to count" }

  2. Server Info: GET /api/v1/info

    To retrieve the server's information, including the maximum context length, we will make a GET request to this endpoint. The response will look like this:

    { "max_context_length": 2048, "max_length": 512 }

OpenAI-Compatible APIs: Adapting to Different Providers

For OpenAI-compatible APIs, we will rely on the following:

  1. Tokenization: POST /v1/tokenize (if available)

    If the API supports a tokenization endpoint, we'll use it directly for precise token counts.

  2. Fallback: Character count ÷ 4 for rough estimation

    As a backup, we'll use a character-based estimation. This provides a reasonable estimate when direct tokenization isn't available.

Files to Modify: Where the Magic Happens

The following files will need modification to implement these features:

  • src/data/models/llm-profiles.svelte.ts: This file will be updated to include context size fields, allowing users to configure the maximum context length for each LLM profile.
  • src/lib/llm-service.ts: This file will house the core token counting logic, handling API calls to retrieve token counts and context information.
  • src/components/story-mode/LLMIndicator.svelte: This component will be responsible for displaying the token usage indicator, including the progress bar and token count.
  • src/components/pages/settings/LLMProfiles.svelte: This file will be updated to enhance the LLM profile configuration UI, providing users with an intuitive interface to manage their LLM settings.

Acceptance Criteria: Ensuring Quality and Functionality

To ensure the quality and functionality of these features, we have set the following acceptance criteria.

KoboldCPP Support

  • [x] Auto-detect max context size from KoboldCPP API
  • [x] Real-time token count using KoboldCPP tokenization endpoint
  • [x] Visual progress bar showing context usage (dark to light)
  • [x] Accurate token counting for all content types

Generic API Support

  • [x] Manual max context size configuration in profile settings
  • [x] Token count estimation (API-based preferred, character-based fallback)
  • [x] Warning dialog when estimated tokens exceed configured limit
  • [x] Clear indication of estimation vs exact counting

UI/UX Requirements

  • [x] Token usage indicator visible in Document Mode and Writer Mode
  • [x] Progress bar or gauge showing context utilization
  • [x] Tooltip showing breakdown: "Story: 512 tokens, System: 64 tokens, Repository: 128 tokens"
  • [x] Warning states when approaching or exceeding limits
  • [x] Settings UI for configuring max context per profile

Error Handling

  • [x] Graceful fallback if tokenization endpoint unavailable
  • [x] Clear messaging about estimation accuracy
  • [x] Handle network errors when querying context info

Future Enhancements: Beyond the Basics

We don't stop there! After implementing these core features, we'll have a solid foundation for further enhancements:

  • Token usage analytics/history: Providing users with a history of their token usage can help them understand their spending habits and optimize their prompts.
  • Smart content truncation suggestions: If a prompt is too long, the system could suggest ways to shorten it, such as summarizing or removing unnecessary text.
  • Context optimization recommendations: Offering suggestions on how to structure prompts to maximize the LLM's performance within the context limit.
  • Support for additional API-specific tokenization methods: Adding support for tokenization methods specific to different LLM APIs will improve accuracy and compatibility.

That's the plan, folks! With these improvements, you'll have better control over your LLM interactions, making them more efficient and less prone to errors. Stay tuned for updates, and happy prompting!