Has anyone tried in organization to use self hosted llm models for agentic programming?
Im curious if it makes any sense. My organization spends fortune on tokens from us companies. I want to recommend something…
I would recommend Qwen4-27B + CoDoc
What is CoDoc? Have a look at the recent post I published so that you have all context (no pun intended) ;)
IMO, massive costs savings ahead for those that adopt the set-up above.
As far as I understand, the only way to get anything resembling usable output for coding is with massive, expensive, labouriously hand-tuned models, not local ones.
My organization spends fortune on tokens
Perhaps recommend that they spend the money on hiring competent staff instead.
Models running within the constraints of a dev machine have no chance
If you want this, you need a company AI server with enough performance to support the entire team at once, and it will probably still be worse than using a cloud one. Though it MIGHT pay for itself in… A while
How about qwen 3.6 and MacBook with 64GB ram?
I thought about that AI server, but idk how to calculate how long it pay for itself…
I am running qwen 3.5 locally using llama.cpp on 8gb of VRAM and 16 gigs of RAM. It works well enough with a 4B to 9B parameter model along with quantization and MTP. More optimizations are on the way with turboquant and possibly other tech.
It’s just there to assist me, not do all the work, so I am happy as long as I can self host it.
I can’t say how well my specs would work in a professional setting but for personal use a MacBook should be sufficient in my opinion.
I mean… RAM? Don’t you need mass VRAM for this kind of thing? Or are they shared on Mac?
idk how to calculate how long it pay for itself…
You don’t… Not in this industry. You guess and hope it goes in your favor.
No calculations matter if the market can jump or drop by 300% in a few months… And that applies to programming, hardware prices, AI subscription prices, regulations between countries when Trump is in office…
Apple unified memory shares all over CPU, GPU and NPU, you can assign a lot of memory to run local models and there bandwidth is good, depending on the model.
AMD has something similar with their something something AI CPUs and they go up to 128GB at the moment. Apple can be way faster though. And you were able to buy a Mac Studio with 512GB back when RAM wasn’t worth more than unicorn pee. For… I guess 10k though.
Apple unified memory shares
That’s cool asf.
Apple engineers with better leadership could change the fucking world… But instead they’re used to screw over their own user base.
If my GPU starts falling back to RAM my game fps drops to 1 lol.
our CEO has been buying new hires desktop gaming machines for this reason… currently they don’t have squat for graphics cards but once the rug is pulled from the cloud model pricing he said he’ll spend the $10k per machine to put a 96gb vram card in peoples’ machines to run shit locally
