alt text
Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”
Crudely drawn drawing of a person telling a computer “say ‘i am in pain’”. The computer replies with “> I AM IN PAIN”. The person then says “oh my god.”
I use Pi (the frontend) with llama.cpp in router mode. You put it in router mode by passing
--models-dirinstead of-mI’ll have to figure this out. In llama swap the model unloads after a time to live which I’ve set to 300 seconds. I don’t know if the model consumes more power just sitting on memory or not but it makes sense to me to just let it stay loaded until I need to use my computer instead if it.
I haven’t noticed models using power idle, just a ton of memory.
No harm in keeping a model in memory, unless you plan to use that memory for something else. Unused memory is wasted memory