Rolling your own agentic harness is a test of fortitude
It's been quite a year in the world of Agentic AI and we here at Vroom Performance Technologies have been here for it. We took the long road to agentic AI in an effort to fully understand the inner workings of LLMs. The models, hardware, configuration, recipes and the many MANY pitfalls.
The toddler harness (late 2024)
We started with ONNX using small models on local consumer hardware. This is a fascinating route, because even with outdated hardware, you can do things with (in our case) C# that you simply can't do with the functions afforded to you by a compiler and a set of standard libraries: summarization. Seriously, I don’t think many people think about what a paradigm shift that is. Ask yourself: "how would I summarize this 2-page message into a cohesive paragraph using my favorite programming language?"
So, the goal here was super simple, write a simple C# console application that will summarize text and so was the stack: Various laptop CPUs and an Intel ARC Integrated 4GB GPU.
We rolled this into a simple chat app with a few tools, which you can download below.
The awkward phase (mid 2025)
We went through the normal phases of Ollama, llama.cpp, etc. using a myriad of hardware configurations. We were serious at this point, but the goal was learning.
Hardware
- Nvidia GeForce GTX 970 / 4GB
- NVIDIA GeForce GTX 1080Ti / 12GB (yes, THAT 1080ti!)
- A test workstation with a PSU that we'd be ok with it of exploded.
- A fire extinguisher.
Software
So once again, we broke out the trusty C# (this time with WinForms, but I promise we'll get to APIs later) and started coding. The goal this time was exercising a context, pushing to failure, tweaking knobs to see if we could make it stop repeating itself, reliably call a tool.
Verdict
Yeah, I know we just got through the tiny software section - but there's a reason we are calling this one here: we came to the conclusion that any model that will fit in 12GB of VRAM just wasn't a good test case. Every failure mode could so easily be traced back to the model just not being clean with its output. The tiny models combined with high-level of quantization just make for a bad day.
Sure, it's cool to look at - but it's really not going to do any real work - so we aren't going to waste your time here.
Goodbye GTX, hello RTX
So, up until now we had been running brute force. Either on the CPUs or on the GTX 3d pipeline. What we needed was CUDA. So, we grabbed a few RTX 3060 with 12GB of VRAM each and started plugging! I mean, that's 24GB of video-ram and all of the best inference engines say right on their labels that we can span GPUs. Clearly, we have this in the bag! (We did not).
You see, the RTX 3060 has a bandwidth of 360 GB/s, but the PCI bus is only running about 32 GB/s both ways, and to make it worse - we only had one 16x slot. So, the token rate was about as fast as a 5-year-old reading their first Dr. Seuss book. (that's not fast)
Growing up (2026)
Ok, going forward we will be discussing actual implementation details, lessons learned, pitfalls, and the joys of running the agent loop. The hardware is real this time too... so buckle up!
Hardware
- Nvidia GeForce RTX 3090 24GB (blower type)
- NVIDIA RTX 6000 ADA 48GB
- Dell Precision XEON 20-core, 128GB RAM
Stack
- C#, Dot-Net 10, Blazor UI, Windows 2025, IIS
- Inference engine: vLLM (Ubuntu)
- Model: Qwen/Qwen3-14B-AWQ
First real harness - but it was not great
So, the first thing that we did was develop a web UI for chatting. Seemed like a logical first step. This was intended to be a place where we could ask questions, paste code for review, and just throw things against the wall. We quickly found that it would be much more useful if we could work with actual files, so we added GitHub integration. That was all fine and good but now we had to go through GitHub on our local machines to make changes and interact with the model.
So, we decided it would be cool if we could just upload and download files in the UI which the chat session had stored locally. Well, that was a pain!
What we really needed was a way for the service running on our web server to interact directly with the files on our machines.
I don't know if you see where this is going, but the web is not the place for local work (it's so obvious now). If you are going to work on local files, you can have your tooling locally and keep your agent loop in the service (that’s your secret sauce).
It works - locally!
So, we built an electron app, using the agent loop defined in our APIs to drive the local tooling. Finally, we could point the app as a folder, ask it to make some changes and watch it go. If you are following the flow, then you probably guessed that it didn't - yet.
You see these models are kind of stupid, they give up - or they are too verbose and burn up the entire context musing about the color of the webpage they are tweaking.
Often times they even report that changes were made without ever even making them - and this my friend is where the secret sauce really comes in.
You see, anyone with a reasonable dev background can talk to a model - but driving that model, making it verify itself, nudging it to keep going when the harness knows it's not really done and forcing it to plan are where the magic is.
The landing
In the end, with nearly 2.5 years under the belt we finally got this thing down. Not by writing it from the ground up as a golden state tool - but by fundamentally starting from the bottom and working the way to the top. Never progressing until we understood each piece.
So, we ended up with a cool tool - but a near-literal treasure chest of new knowledge about how models work, how to talk to them (technically and through prose), their limitations and a real understanding about when and where they can make life easier.
Specific takeaways
- Stay in VRAM.
- Use large contexts.
- Allow agents to compact their own context without affecting the "parent" context.
- Offload classifications to a separate GPU.
- Ditch RAG encoding. A decent tool set for discovery and sifting is far superior.
- SQL Server search vectoring? Nope! It only supports 1,998-dimension which restricts you to all but the smallest encoding models.
- Plan decomposition? Skip it! It's an extra lever of abstraction between your requirements and the model. Like playing telephone with a team of developers.
- Build a pipeline that forces the model to stay focused.
Alesia Masharskaya | https://www.vecteezy.com/members/alesia-masharskaya483427