Never tried that. I worry about the damage and lifespan reduction that could cause to my SSD drives. If we go vanilla, I get the following performance on my PC:
This is via the KoboldCPP backend. Things are much faster than they used to be, a year ago this would have taken at least 10 minutes. This is a 35b MOE model.
Specs-wise, my machine is…
7965WX - STR5 workstation CPU ($1,500, used)
32gb of RDIMM DDR5 5600 ($600, used)
RTX 4090
0000000
Within a few weeks, I will have the machine worked on to fix a teething issue and to add at least a second GPU. My machine has device enumeration issues, which I believe to be caused by the PCIe SATA card. I am figuring that eliminating all non-motherboard SATA devices from the build will end the issue that I am having.
In this case, the computer can have 20+ minute boots, with bootloops or getting stuck. This only happens with booting, Windows is just fine. I consulted with my local AI about the patterns and error messages I was seeing, we eventually homed in on likely points of contention for the system.
This also lines up a prior experience with a SATA card some 15ish years ago. Silly me for expecting a modern SATA card to not suck.
What kind of token rate do you get if you try to run them in swap space?
Never tried that. I worry about the damage and lifespan reduction that could cause to my SSD drives. If we go vanilla, I get the following performance on my PC:
000000
Generating (2562 / 8192 tokens) (EOS token triggered! ID:248046) [11:17:40] CtxLimit:2656/32768, Init:0.10s, Processed:94 in 0.83s (113.12T/s), Generated:2562/8192 in 47.51s (53.92T/s), Total:48.44s
000000
This is via the KoboldCPP backend. Things are much faster than they used to be, a year ago this would have taken at least 10 minutes. This is a 35b MOE model.
Specs-wise, my machine is…
7965WX - STR5 workstation CPU ($1,500, used)
32gb of RDIMM DDR5 5600 ($600, used)
RTX 4090
0000000
Within a few weeks, I will have the machine worked on to fix a teething issue and to add at least a second GPU. My machine has device enumeration issues, which I believe to be caused by the PCIe SATA card. I am figuring that eliminating all non-motherboard SATA devices from the build will end the issue that I am having.
In this case, the computer can have 20+ minute boots, with bootloops or getting stuck. This only happens with booting, Windows is just fine. I consulted with my local AI about the patterns and error messages I was seeing, we eventually homed in on likely points of contention for the system.
This also lines up a prior experience with a SATA card some 15ish years ago. Silly me for expecting a modern SATA card to not suck.