[REQUEST] Can't Set Context Length via front-end

[zapshe]

OS

Windows

GPU Library

CUDA 12.x

Python version

3.11

Describe the bug

There's no way for the front-end to set the context length of the conversation. Either the context is set via max_seq_len in a config file or there's no context size control at all.

This is terrible for tools like Cline where you want it to be able to set a custom context when running the model and it may differ with each run.

Reproduction steps

Running Cline with context set to any value.

Expected behavior

Tabby should run the model with the context length ("max_seq_len") that's requested.

Logs

No response

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[kingbri1]

ADMIN NOTE: Please title your issues properly to make things easier to categorize

You should not be setting max_seq_len on request. According to the OAI spec, the maximum length of a prompt + space for generation should be reflected in max_tokens which is less than max_seq_len.

TabbyAPI sets this to the configured max_seq_len as a default since we're assuming the user wants to fill up the context as much as possible for that request.

If cline doesn't expose this value (it ideally should and I'd recommend opening an issue there), you can read on sampler overrides and create a preset based off of the sane defaults one and configure accordingly.

[zapshe]

Thought I added a title, my bad - it's there now.

Cline, Kilo Code, Cherry Studio, Open-WebUI, and some other front-ends I've used. I've tried setting max_tokens for all of them (and other settings), it never changes the cache size allocated.

WARNING:  The given cache size (36000) is not a multiple of 256.
WARNING:  Overriding cache_size with an overestimated value of 36096 tokens.

I use non-multiples of 256 so it will definitely display the cache size it will allocate. I've put -1 for the TabbyAPI config file's "max_seq_len" and set ""max_position_embeddings": 36000" for this particular model's config file. This has been the only way to set the actual context.

Default config values (and sending "max_tokens" shows:

max_seq_len is undefined. Overriding to 4096 tokens.

The point is to determine how much VRAM is allocated for the model for that particular session via the frontend - which seems impossible right now.

As I understand, limiting "max_tokens" limits the output the AI can generate, not the context length - and TabbyAPI does not seem to be using max_tokens for max_seq_len unless there's something I'm missing.

The config files for AIs do not contain "max_tokens" either.

As far as I know - OpenAI's backend doesn't seem to allow for custom context lengths. Other backends do - like llama - via variables like "ctx_size". If TabbyAPI is supposed to be using "max_tokens" for "max_seq_len", it's not (and shouldn't?).

[turboderp]

cache_size refers to the total size of the paged cache, limited only by your VRAM and how much memory a particular model uses per token. You want it to be as large as possible: you cannot run a job with a total length larger than the cache, but if there's extra room in the cache it's used for prompt caching and concurrency.

The value is set once when the model is loaded and then the cache persists until the model is unloaded. If no value is given in the config, Tabby will use max_seq_len as the default, but this is still a different parameter.

max_tokens is the maximum number of output tokens you want to generate from a given prompt, passed along with a particular request to the server. Generation for any one request will end when this number is reached or the model emits an EOS condition.

If max_tokens isn't sent with a request or the value is zero, the default is computed from max_seq_len - context_size. This means a request by default uses up to the model's "useful context length" or whatever max_seq_len is set to, possibly capped by the server capacity (cache_size) if necessary.

For your particular case, you should be able to set cache_size = 36096 (or higher if you have the VRAM for it and also want the benefits of prompt caching etc.) and max_seq_len = 36000 while omitting max_tokens from the request. If you want to specify the 36000 total length per request then I believe currently the only way to do that is to use the tokenization API to get the context length of each individual request, subtract that from 36000 and pass the result as max_tokens for the completion request.

As for extending the OAI spec with custom arguments like ctx_size, that's always kind of a slippery slope and usually doesn't lead to anything good. It could make sense to have a max_total_tokens argument, maybe? I'm not sure what would be the "most standard" way to implement it, or why it's really needed anyway?

[zapshe]

If you want to specify the 36000 total length per request
why it's really needed anyway?

The point is not per request - but when loading the model into memory. If I'm having a simple chat, then the default context size is probably fine. But if I'm using the AI for coding, I may want a certain amount of context (perhaps the maximum my VRAM can give). These affect VRAM usage.

Currently, the only way to control how much VRAM is allocated for context is by manually changing the "max_position_embeddings" variable in each model's individual config file (after setting -1 in the global config file).

There's no way to control the context size via the front-end, so setting context size in any front-end does absolutely nothing for how much memory is allocated locally.

Maybe these front-ends don't even send context sizes (even when set) due to the OAI spec, but I can't really see what they're sending out so I can't be sure.

I already have to add a custom header: "x-api-key" to give front-ends my admin key so they can start up models on TabbyAPI back-end. So even a custom header for context size might not be a terrible idea.

But I do see this might be an OAI spec issue that has no direct solution - but that depends on what front-ends are actually sending out when a context size is selected which I can't confirm.

[DocShotgun]

The k/v cache is allocated when the model loads and does not subsequently get resized.

If I'm understanding correctly, you're trying to have the cache dynamically resize to consume more or less VRAM using a param specified by the client with (chat) completion requests? This would introduce all sorts of problems, especially in a multi-user setting or with continuous batching. For example a single request with too high of a context specified client-side could crash the server with CUDA out of memory. I'm not aware of any inference server that allows this as a feature.

[zapshe]

I'm not aware of any inference server that allows this as a feature.

No server really does this - but this is for a local PC. Ollama does this - or at least supports this. When using Cline or otherwise using Ollama, the context size I set determines how much VRAM is allocated.

As you can see, there's the initial initialization for the model followed by an initialization for the context. Screenshot shows loading the model twice. The first was for 20000 context (you can see initial loading then loading extra for context). The second was for 5000 context, and you can see how the VRAM allocated is dramatically lower.

If this isn't possible with TabbyAPI as it's meant for servers rather than individual PC usage, I understand. The OpenAI spec clearly is meant for servers and not individual computers. Ollama is more built with a single PC user in mind as far as I know.

[DocShotgun]

Yeah it wouldn't play nicely with concurrent requests or more than one user. As for ugly workarounds, you could try:

1. Something like llama-swap configured with the low and high cache size versions of the model assigned different aliases, and this would essentially kill and restart tabby with your new settings on swap.
2. Create another directory in your models directory, symlink all of the individual model files into that new directory. Make a new tabby_config.yml with the different cache size settings in that folder. Name the original high context and the symlinked clone low context or something.

[zapshe]

Unlucky, Tabby was nice since it could run EXL3 models which Ollama doesn't support. EXL3 models seem to be superior than the GGUF models as long as you have the VRAM. Since Ollama is local PC friendly, it makes sense they haven't leaned into EXL3 models.

Leaves poor me with trade-offs to make as this makes it harder to completely come off Ollama and switch to TabbyAPI.

[frenzybiscuit]

I'm confused. Why do you want this feature?

You can already change the context length when changing the model with inline model loading... to the best of my knowledge. As such, just create a duplicate model. One with less context, and one with more. Then, when you want to change context length you can load the new model.

[zapshe]

You can already change the context length when changing the model with inline model loading... to the best of my knowledge. As such, just create a duplicate model. One with less context, and one with more. Then, when you want to change context length you can load the new model.

Yea, but that's rather rigid - not as streamline of a solution as what Ollama does.

I may need to have several of the same model listed on my front end just with different context sizes. Maybe another program is taking up VRAM and so I need to allocate a little less VRAM than usual - now I have to edit config files and create another duplicate (symlink would be the best solution there for DIY). You could imagine upgrading a GPU down the line and now you have to edit every single config file of all models and their differently-sized contextual cousins to adjust for the difference in VRAM.

Instead of setting the context length in the front end on demand, I need to prepare ahead of time models of different context sizes as necessary. I don't think this is a deal breaker at all - but it's just easier to use Ollama's GGUF models.

A programmatic solution most definitely exists here - even something as simple as Tabby overwriting the max_position_embeddings variable in the config file when given a special parameter would solve the issue. It's quite easy to add custom parameters to Tabby via most front ends (like Open-WebUI, Cline, etc..).

It shouldn't break anything for anyone else as it's a feature that must be utilized via a custom parameter. But would make Tabby so much more accessible for those on local PCs and want an alternative to Ollama.

[frenzybiscuit]

I believe exllamav3 implemented cuda graphs, so I don't know if its possible to implement a on-the-fly cache change. I'll let the other, more experienced people answer for you.

Ollama does not, as far as I know, have cuda graphs.

[zapshe]

Well it doesn't have to be on the fly at all. It would be on model startup. Even Ollama doesn't do it on the fly as far as I know. If you want a different cache size, just unload and reload the model with the new parameter.

For me at least, unloading/reloading models is a small time penalty. Creating different symlinks with different configs for every context size I may need would be a much bigger beast in terms of time penalty alone.

I'm envisioning a very simple logic: Receive parameter -> edit config -> load model. It would be a great addition to the Tabby API for anyone on a local PC.