6d ago
Implement message trimming/compaction to prevent context overflow errors in long conversations
When a conversation gets long, some models start returning errors and stop responding.
This seems to be because the entire conversation history is sent every time, eventually exceeding the model's context window limit.
Models using the official Anthropic or DeepSeek APIs don't have this issue, likely because those APIs handle context management on their side. Other providers lack this feature, so the request fails as-is.
The existing `pruneMessages` function removes old tool call results, but the conversation text itself is never trimmed. As a result, longer conversations still exceed the context window limit.
Please consider:
- Trimming or summarizing older messages once the conversation exceeds a certain size.
- Adjusting the amount of history sent based on each model's context window limit.
In Progress