alt text
Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”
Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”
What does it need to “reason” for 5 seconds on a sentence like that?
The model loading.
Weird that it counts those together. Pi shows “Waiting…” for model loading and prompt processing, and “Thinking…” for reasoning
It’s Llama swap and I barely know how to use it.
I use Pi (the frontend) with llama.cpp in router mode. You put it in router mode by passing
--models-dirinstead of-mI’ll have to figure this out. In llama swap the model unloads after a time to live which I’ve set to 300 seconds. I don’t know if the model consumes more power just sitting on memory or not but it makes sense to me to just let it stay loaded until I need to use my computer instead if it.
No harm in keeping a model in memory, unless you plan to use that memory for something else. Unused memory is wasted memory
I haven’t noticed models using power idle, just a ton of memory.