Video: Agent Quality & Token Optimization | Duration: 3592s | Summary: Agent Quality & Token Optimization

Transcript for "Agent Quality & Token Optimization": Hello, everybody. Thanks for joining us today. We're gonna give it another minute or two while everyone joins. And then if you want to in the chat, just go ahead and drop where you're joining us from. Awesome. We have people from everywhere. That's great. Alright. Let's get ready here. So hello, everybody. I'm Katie. I'm a customer success architect here at GitHub. And today, we're gonna be covering agent quality and token optimization. So just a few key points. One, I'm joined by a few of my colleagues here, so please drop questions into the chat to keep them nice and busy. And then, also, if you hear any snoring at all, I promise no one except my dog has fallen asleep during this presentation. So let's get into it. The motivation for this is, of course, GitHub switch from premium request to usage based billing. So this has raised a lot of questions about token consumption. Customers are asking, how do we improve our token spend? But focusing purely on cost can actually diminish the value you get out of Copilot. So the better question to ask is, how can we make the most out of the tokens we spend, which actually comes with token optimizations as we will learn. So we're gonna cover three parts in this presentation. So part one, we're gonna go over agent quality. So we'll explore why optimizing for quality beats optimizing for cost alone and how quality improvements naturally reduce token spend. Part two, we'll go over some foundations. We're gonna cover the fundamentals of large language models, agents, and context windows. This foundation will help you gain a deep understanding about optimizations and quality, but also help you evaluate the hundreds of token optimization tips circulating online and form your own judgment about what works for you. Then we're gonna cover quality and token control. So we'll dive into the practical controls you can employ yourself from model choice to prompts to agent configs with actionable customization tips. Don't worry. We will also include a live demo so you can see some of these tips in action. The goal for this is to help you be better with less. Now let's start with why agent quality is the better focus. To understand this, look at how the industry is operating today, how each of us is probably acting. We're using agents in what I'd like to call a gambling system. Now imagine if rockets were cheap. NASA could just send 20 rockets roughly in the direction of the moon. If one of them landed, great. If none of them landed, let's just send off another 20. So this maps precisely to how agents are often used today. You give only a little context. You write a prompt, but don't put too much effort into it. You send the agent on its way. If it comes back with a good result, great. If it doesn't, you send off the next agent. The problem is that this gambling approach is no longer sustainable, and it never really was. It was sustainable when you only had two, three, or four agents a day. It's no longer sustainable in the direction we're moving where dozens, if not hundreds, of agents get dispatched by every single developer every day. Those agent sessions are becoming longer and longer. This is also not sustainable from a financial point for us at GitHub because we used to bear that cost. And now that we're switching to a usage based model, it's also no longer sustainable for you as a customer. So I'd say it's definitely time for a change. But if you purely look at this from a cost angle, the question becomes, how can we make the fuel cheaper? That's really just asking how can we continue to gamble, and that isn't the right thing to do. I also don't think it's what you want to do. Instead, we should work on decreasing the number of rockets we send out because more of them will actually hit their target. In other words, we need to increase the value of each and every agent. We need to increase the quality of the agents we send before we send them out so that more of them will reach their target. Overall, we need fewer agents, and fewer agents automatically means fewer tokens spent. We need to start working on a better return on investment for our agents. So one main reason why focusing on quality is more important is the compounding error problem. Large language large language models are nondeterministic. They have an error margin, and it will never be 100% accurate. In multi step agentic workflows, this can create compounding errors. So here's the math. At 99% accuracy per step, which is optimistic, 50 steps reduces your overall accuracy to just 61%. Drop to 95% per step still sounds good. Right? And you end up at 8% accuracy over 50 steps. That's not good. This doesn't mean every agent will fail, but it means every quality improvement dramatically increases your chances of success. And consider the cost. Every agent miss wastes those tokens requiring bug fixes, reviews, and additional agent runs. You might even have incidents caused by low quality agent output. In classical software development, we address this kind of problem with the shift left movement, shift left quality, testing, security. All of that becomes even more true in agentic systems. So to sum things up, instead of counting tokens, let's make every token count. Yes. Reduce token size, but not driven by cost, but by quality. You want to send few rockets with high accuracy, which will automatically optimize the fuel. So to increase the quality of agents, we first have to understand how all of this works. The LLMs, the agents, the context window, and how it all ties together. So a quick high level refresher. An LLM is a text in, text out machine. Under the hood, it's a word probability machine. Given the input plus the training data, it predicts the most probable next word, then the next, until a sentence is formed. Encoding, same mechanics, just predicting the next instruction or statement instead of prose. Of course, models got better and better over the past years, and a lot of bells and whistles got added for more compute, higher accuracy, biases towards software development, etcetera. But the underlying principle and capabilities are still very much just this. Now with agents, the previous rule matters even more because it's no longer a single back and forth. The agent talks to the LLM on your behalf dozens of times before it comes back to you. An agent is just an app. Code. There are many harnesses offered by GitHub. We have GitHub Codespaces chat, where this all started. We have Copilot CLI, Copilot Cloud agent, and even third party harnesses like Cloud Code and Codex. The LLM itself is just the model, GPT, Cloud, Gemini. Now what we need to understand here is that the way an agent interacts with it is not magic, although it seems like it. It's still just text and, critically, stateless text. An LLM does not store conversations. Having a conversation actually means resending the entirety of inputs and outputs in order every single turn. So this causes tokens and context to compound. As you can see, the harness in for itself already has a large role in agent quality, but you still have ways to influence it. So some levers you can use to influence the agent are your prompt that you're sending, the files in your projects, agent configs, such as instructions, skills, and MCPs. So first, let's look under the hood at context windows. On every loop, the agent sends the entire conversation to the LLM again. So here we have loop one. We have your system prompt plus the tools, which are descriptions of available functions like read file, write file, plus your prompt, plus file references equal the input tokens. Then the LLM response is the output tokens. We have loop two, and that contains all of loop one plus previous responses plus new input equals even more tokens. So we think about it in terms of token per word. So one token is about three fourths of an English word. Let's just think one token, one word. So context limits vary. Smaller models can handle 50 to 200 k tokens, whereas larger models like Claude Opus or GPT 5.5 can handle 1,000,000 tokens. Now if we think of 1,000,000 tokens, we're thinking Lord of the Rings trilogy plus the Hobbit. It's a lot of words. However, don't get caught up in tokenization details as you don't have that much control over it anyways. Think at a higher level. Prompts, files, responses, all consume tokens, and they compound with each loop. Now before diving into controls, we need to understand two major problems with context windows. First, we have lost in the middle. The model favors content at the beginning and the end of the context window, not the middle. This is usually fine because the beginning contains your instructions, your goals, and your plan. You want this prioritized. The n contains the current work stream, but this is also important. However, the middle is past work, so that's less relevant. The problem hits when you switch tasks mid session. So you start with a bug fix, then mid conversation you say, now let's implement a feature. We've all done it. As the window grows, the model may suddenly switch back to the bug fix because it biases the initial statement over recent ones. Solution for this, Use a new context window for each distinct task. Then we have the recency bias. So above 50% capacity, the model starts favoring only the end of the conversation. It forgets your system instructions, custom instructions, and original prompt. The model drifts, doing things you don't understand based solely solely on recent context. The solution for this? Don't let your token window grow beyond 60 to 70% unless you have to. Dividing and conquering tasks from the get go is the best way to counter this problem, followed by potentially compounding a conversation. Lastly, don't take this as something that will always happen. Bias doesn't mean the model will forget everything in the middle. Don't deal in absolutes here. There are, of course, scenarios where you'll need to go above 50% of the token window, and that's fine. It's just something to keep in mind that you can use to optimize your usage. So, hopefully, this gives you a good foundational understanding of tokens, context windows, large language models, and agents, and the first ideas on how to optimize your usage. With this knowledge, let's now look at the practical guidance and things you can do starting today to improve the quality of your agents and your tokens. Okay. Let's do a quick tour of where quality gets lost and tokens get burned. These are the usual suspects we see in bloated copilot sessions. I'm not going to read over each one of these on the slide as we're going to come back and dig into them individually over the next several slides. From model choice to imprecise prompts, to oversized instructions to missing guardrails. So think think of this slide as the map before we walk through that territory. So the rule of thumb I want you to hold on to as we move through each of these individually is this. If you can't explain why Copilot needs a piece of context, it's probably costing you. One more thing worth mentioning here. Since we won't have a dedicated slide for it, think about how your repository itself is structured. If you're working in a monorepo, and most of you probably aren't going to restructure that overnight, the practical move is to subfolder by application layer and keep application wide context in its own separate folder so it isn't dragged into every single task by default. If you do have the flexibility to choose your repo structure going forward, a poly repo or multi repo approach can get you that same isolation from the start. So how does this actually come together in practice? This is what we mean by becoming agent native using the right mode with the right model and the right context at each stage of the work. Now look at the workflow on the slide. It's broken into five stages, and each one calls for a different mode and a different model. And for deeper work, this is where it gets powerful. Copilot can bring multiple models or agents into the same workflow. One to spec, one to code, one to challenge assumptions, and one to serve as risk. That's how AI moves beyond faster output and starts helping teams create better software. So in spec mode, you switch the harness to its explicitly research and or plan mode depending on your desired outcome. This is planning and architecture work, so you most likely want heavier reasoning model here. Then we have challenge. So you can invoke this with the rubber duck command, which is available in the Copilot CLI. This is our classic rubber duck debugging, the approach in when you deliberately use a different model family than the one that wrote the code or the plan. And this matters since a model reviewing its own work is prone to rubber stamping its own blind spots, just like we are. Now we have the code. This is the implementation of the plan you just built. So you want to pivot to more of an orchestration model here, say, SONNET five, since the heavy reasoning already happened in the spec phase and now it's about executing it well. Things like fleet or orchestrate in the Copilot app help to execute the plan with the proper sequence and parallelism, ensuring that you divide and conquer, and with it context, into digestible pieces for one or multiple agents. Then we have the review. So this is used with the review command, so running your tests, linters, and Copilot code review. GPT 5.5 is a good one here. And, lastly, we have secure. This is invoked with the security review in the CLI for code security. This all ties back to something we touched on earlier. Model choice is one of your biggest levers in token optimization. The common pattern we see is defaulting to the biggest reasoning model for everything, including typo fixes, and the ways from that is significant. The cost difference between a mini model and a frontier reasoning model can be 24 x. Model choice drastically impacts both token cost and quality, and bigger doesn't always mean better. This is exactly what this five stage workflow builds in on purpose. Match the model to the stage, not just the default to the biggest one out of habit. So this is the next step in the auto story, not just a default model picker, but a more intelligent routing system that can balance quality, speed, cost, and task fit. Our original implementation of the auto model picker was based off avoiding you all becoming rate limited. Then back in June, with the advent of token based billing, we enhanced it to focus on task intent. Now we have the next version of the auto model picker coming known as auto v two. The key point for our customers is that auto is getting broader and smarter. It can route to a wider set of models, including premium options when the task warrants it while still respecting costs and latency tradeoffs. We also expect more tiered choices in auto, so customers can make a conscious quality versus cost decision without needing to manually pick a model each time. That matters for mixed workloads where some tasks are quick and lightweight, while others need more depth, more context, or a more capable model. Another important improvement is token aware and long session aware routing. The more a session grows in complexity or duration, the more the system has to think about context quality, prompt efficiency, and drift. Auto v two is designed to make those decisions more intelligently rather than treating every task the same. In customer terms, this means less manual tuning and more consistent results. The system gets better at choosing the right model at the right time, which helps improve both quality output and token efficiency without forcing users to become model experts. You cannot memorize them all. Now it's worth pausing on how all of this actually applies depending on what you're trying to do. Different use cases warrant different modes and optimizations. So the real starting point isn't a checklist of tips. It's your intent. Look at this look at the table on the slide. Three common use cases, what good looks like for each, and the levers that move the needle most. Debugging or incident fixing. Good here means root caused fast with no new bugs introduced. So the levers that metal matter most are reasoning model and deterministic guardrails like tests. You want the model thinking carefully, and you want tests catching it immediately if it introduces a new problem while it's fixing the old one. Then we have feature implementation. So good here means correct on the first try with minimal rework. This is where that phase research, plan, challenge, implement flow we just walked through really pays off, combined with your agent configs and deterministic guardrails doing the heavy lifting throughout. And then we have the research intensive slash unfamiliar code. So good here means the agent finds the right files and keeps its context scoped instead of pulling everything in. The levers here are the sub agents in the scoped context, solid context engineering, and custom agents are skills that know how to navigate that part of the codebase. Now the point here isn't to memorize this table. It's to get into the habit of asking, what does good look like for this task and which levers actually move it? Before you reach for a model or a config. That's what starting from intent means, and it's the mindset that carries into everything we're about to cover. So your prompt is very important. Don't optimize prompts for fewer tokens. Optimize to steer the agent correctly from the start. Prompts, systems, and tools are always in the context window at the beginning, giving them outsized influence. So be precise. Don't just say fix this bug. Say issue number 45 describes a bug where x y happens. Fix it. And you wanna include stop signals. Once the bug is fixed and tests passed, stop. This prevents the agent from continuing with unnecessary work, like git commit, push, linking more files. And you wanna add known context. So if you know where files or other context lives, don't let the agent find it. Provide it. Same is true for documentation websites you want the agent to fetch, skills to invoke. Whatever you can put there from the start will improve your experience and reduce the cycles and tokens used for the same outcome. So let's look at what prompt discipline actually looks like side by side. A real example of the same task prompted two different ways and why one costs you far more than the other. On the left, the expensive prompt. Look at my repo and figure out why users are getting 500 errors on login. Check the auth code, database layer, middleware, and logs. Fix it. That prompt isn't expensive because it's long. It's 19 words. It's expensive because look at my repo. You're telling Copilot to go read a lot of files, which means a lot of tool call results. Tokens come back into the context window and compound. Then on the next turn, all of those get resent. It rescans many files, reads large chunks, and burns tokens before it's even narrowed down what's wrong. On the right, the efficient version points directly at the specific file and function where the bug lives and describes the exact failing case. It costs one prompt to produce a targeted fix because Copilot doesn't have to go hunting for the problem first. One function, one case, one fix. If you know the information, provide it. So we've spent a lot of time on prompts and models. Now let's talk about guardrails because having the right ones is what actually prevents agent misses early instead of catching them after the fact. This is the same shift left principle we mentioned earlier in the talk, just made concrete. Four guardrails worth having in one place. Tests. I cannot stress this enough. They guide agents to implement the right behavior and ensure regressions get detected early, rather than discovered days later. Remember the compounding error problem earlier? Accuracy degrading step after step over a long agent session? Tests are the single best counter for that because a failing test gives the agent an unambiguous, deterministic signal to stop and self correct instead of continuing to build on a broken foundation. Then we have linters. This keeps your formatting consistent, which makes the codebases easier to read and reason about, and that benefit applies just as much to agents parsing your code as it does to humans. Good architecture. This means clarifying to the agent what goes where. When the boundaries are clear, the agent wastes less time searching for the right place to make a change, which increases the chances of success and reduces the tokens burned along the way. Then we have quality and security stands. This gives you feedback on both dimensions as early as possible. Catching a quality or security issue at the moment it's introduced means no rework later versus finding it downstream where everything's broken and the fix is more expensive in both times and tokens. The common thread across all four is that they are deterministic. They don't rely on the model getting it right by chance. They give the agent and you an unambiguous signal the moment something goes wrong, which is exactly the nondeterministic models need to stay on track, and your coworkers will also thank you. So everything we've covered so far has been about personally what you do in your prompts, your instructions, and your configs, but you don't have to carry all of that guardrail work yourself. GitHub helps you out. So we also have a set of guardrails that reinforce a lot of what we talked about without costing you extra prompt engineering. So three of them that are worth knowing. Copilot code review. This is an intelligent, automated, first pass review of your changes that catches the most prevalent issues automatically before a human reviewer even looks at the PR. This is the same idea as our challenge stuff from earlier, just running natively on the platform. Repository rule sets. These can be used to automate the invocation of Copilot code review on every one of your PRs. And then you have advanced security. CodeQL scans give your agents viable feedback before things become a real problem, catching issues like injection vulnerabilities as they're introduced. Push protection actively prevents secrets and tokens from leaking in the first place, stopping the commit before it ever lands. And then we have code quality. So this scores repositories across maintainability and reliability, so you can see at a glance where quality is slipping across your organization. The common thread across all three is that they're deterministic controls just like the tests and the linters we talked about earlier, except these live at the platform level instead of in your repo. They catch what your prompt, your instructions, or your custom agents might have missed, and they do it consistently on every PR without you having to remember to ask for it. Now we have persistent instructions. These are in the context window for every agent sessions, every interaction. These are something that you can do for extra guardrails. So you wanna make sure they're concise and small. Don't put your entire documentation or human readable guides here. Think of it as you're human in a loop, proactive guidance for every agent. So what belongs here? We have project guardrails that every agent must follow. You wanna add any correct reoccurring errors, like wrong testing framework, wrong build command if an agent runs that accidentally. You wanna be concise. You wanna drop niceties. Say only return code. Output tokens are the most expensive, and trimming them matters. So research shows be concise yields the same results as the 50 line caveman skill. You don't need the agent to explain it to you. You want it to be concise. So some best practices for generating these agent and copilot instructions are don't use AI to generate instructions. This is your chance as a human to guide agents and fill gaps AI can't know. Now write them yourself based on real agent behavior in your project. Now you wanna iterate. It doesn't need to be perfect from the start. You can add corrections as you observe agent misses and then recreate them. What also helps is recreating them often. Models and your project changes constantly. For reference, the Copilot CLI team throws away their entire instructions every three months because they might be outdated, no longer relevant, or no longer contain the required information or even compound useless information. Consider this a living, breathing document. No. Have you ever if you've ever inherited a bloated Copilot instructions markdown file from a previous project, this slide is for you. The main thing to understand here is that this file gets injected into the context window on every single turn. It isn't documentation you write once and forget. You need to think of it as a high cost configuration you're paying for repeatedly on every prompt for every developer using it. So on the left of this slide is what a type file actually looks like. You have the project context, standards, and a short list of explicit do not suggest rules. That's the whole thing, and it comes in roughly at a 150 tokens. Compare that to what we've seen in the wild, teams dumping their entire onboarding guide, a full architecture overview, and a style guide into the same file. That's more like 3,000 tokens, multiplied by every turn by every developer on the team. Under usage based billing, that's real recurring money for content that mostly isn't earning its keep. Just throwing away tokens there. The keep versus cut breaks down on the right is an actionable takeaway. So standards, these are things the model can't infer from the code, and explicit do not rules earn their place in this file. Onboarding, essays, art architecture diagrams, full style guides, and anything the model can already infer from the code does not belong here. Link them out instead or leave them out entirely. Now beyond persistent instructions, there are a few other agent configs worth knowing since they're all part of the same context engineering toolkit. We have custom agents, skills, and MCP servers. So custom agents force an agent to adopt a specific persona or a way of working. They're best used as something manually invoked by you as a human when you want to orchestrate a workflow and have an agent behave in a very specific way. For example, a test driven development agent that's scoped to only implement red failing tasks. You invoke it with a slash command, and the harness retrieves that custom agent file and adjusts the available tools accordingly. Now be intentional about which tools you define in the custom agents YAML front matter because they do take up space in the context window. And if you don't need them, don't give them to the agent. The real benefit here isn't token savings as much as preventing agent misses. If you only want an agent to read an issue for a spec, not write or update it, you prevent that entirely by simply not giving you access to the tool. Now we have skills. So skills are close to custom agents, but not quite the same. A skill is something you offer to your agent based on the task it's doing, and it's loaded dynamically rather than always on. The harness offers the skill description to the LLM, and when the LLM detects a task matching that skill, it loads the full skill file along with any reference files or scripts. So that makes skills a great way to offload context that isn't always relevant, pulling it in only when it's needed. Don't overdo it though. Every skill's description still has to go into the context window, so hundreds of skills add up. And a skill describing something that the model is already highly proficient at can cost tokens without adding much value. Now lastly, we have MCP servers. So this grants agents the ability to pull in information from external sources and use it as context, exposing dynamic tools once activated. You wanna be intentional here. An MCP server offering 20 tools when your agent only ever needs two means you're paying for the other 18 in every single turn. And it can also lead an agent to call tools you didn't want it to use, like updating an issue when you only wanted the agent to read an issue. Now deactivate MCP servers you don't always need or scope them into custom agents instead of just leaving them all on globally. For example, the Playwright MCB server is a good example of one to use sparingly. It's powerful for web front end work, but screenshots and page reads consume a lot of tokens, and images attached early in a session get reprocessed and build again on every subsequent turn. So everything we've covered today is guidance you apply proactively, but you should also be observing your usage after the fact because that's how you actually find out whether any of it worked. There are three levels to look at here. Your current context and cost, your usage and behavior over time, and higher level metrics at the org and enterprise level. So first, we have the analyze current context and cost. AI credit consumption is surfaced in each harness at the end of every prompt execution, so pay attention to it in the moment rather than waiting to review it later. In the Copilot app, you can use context and insights. In the CLI, you can use context and usage commands. In Versus Code, you can use the context view and the agent debug logs. The debug logs, in particular, let you inspect a session after the fact and see exactly where your tokens were spent. Which turns, which tool calls, which file reads were the expensive ones, and then you can take them out and modify. That's often the fastest way to spot a rediscovery loop or an oversized context pull we talked about earlier. Then you can analyze your usage and behavior. So with the chronicle command, this is available in all harnesses to query your overall usage. It's not a Versus Code only or a CLI only thing. So it works by digging into your actual session logs, the history of your prompts, your turns, your tool calls, and your token usage, rather than relying on you to remember what you did because that's impossible. The one to focus on for today is Chronicle cost tips. I will demo this later. It gets you tips scoped to your own prompting behavior and the model selection patterns grounded in that session data rather than a generic checklist. There are a couple of other purpose built sub commands worth knowing too. We have chronicle tips that services workflow improvements you might be missing, and chronicle improve looks for a pattern in Copilot's reoccurring mistakes and generates custom instructions to fix it. This is directly useful for the persistent instructions we just talked about. Now we have higher level metrics. So now these mostly have to be set up or access has to be granted first by your enterprise admins. A couple different methods available are OTEL. So Versus Code offers open telemetry options, so admins can pipe usage data into whatever observability stack they're already using. And then we have GitHub Impact Dashboards. This shipped in July, and these help assess the overall impact and ROI of agents at the organization level. We also have agent session streaming, currently in private preview, but this streams every action an agent takes, thinking, writing code, running tools. As an event, you can subscribe to giving admins visibility into sessions as they happen live rather than only after the fact. Now the point of the slide is simple. Don't apply these optimizations once and assume they're working. Go look at the data at every level, confirm it, and then change what change what action actions you're doing. So everything up to this point has been things you do. Let's close out with a few things you can just defer to us because we're continuously shipping improvements on the harness and model side that give you quality and token efficiency for free without any change in your behavior. So first, we have harness improvements. Caching, tool compaction, and more are the kinds of things our harnesses are best positioned to optimize, not you. Every release of Visual Studio Code in Copilot CLI tends to quietly ship a handful of these and add up over time. It's one of the easiest wins available precisely because it costs you nothing to get, but the value only reaches you if you actually stay current. An out of date harness means you're leaving free quality and token savings on the table every session. Keeping your harness updated is genuinely one of the highest leverage, lowest effort things you can do. And now we have auto mode v two. We already spent real time on this one earlier. Remember, model choice is tedious, especially for smaller tasks, and that's exactly the decision auto mode v two is built to take off your plate. Routing intelligently across a broader set of models based on task rather than making you pick every time. And lastly, we have MAI code models. Microsoft is investing directly in coding specific models here, and it's not just a promise. Code one is already live in GitHub Copilot and Visual Studio Code, and the newer code 1.1 Flash introduces higher quality code at roughly 25% greater token efficiency and about a quarter of the cost of what came before it. That's the model layer itself getting more token efficient by design, which stacks on top of everything else we've talked about today. So the common thread, harness improvements, smarter routing, and more efficient models are all things that get better in the background. You don't have to chase them. Just keep your harness current and let them do the work. Alright. Let's get into it. Enough slides. I'm gonna open up Versus Code, and we'll take a look at some live demos here. Okay. I'm gonna zoom this in to make sure it's a good size. So what I did here was I already ran three pairs of prompts ahead of time so we don't have to sit and stare at them while I think. So we're gonna compare what the agent returned, how much guidance it had, and what that means for total tokens spent. So right here, I have two prompts. I have a bad refactor prompt and a good refactor prompt on the right hand side. This is the prompt that I sent over. I have the chats open side by side. So the first prompt, the bad one, says only make it better. Better could mean readability, performance, error handling, documentation, or a larger redesign. We don't know. It does not say which behavior must stay the same or whether I want code and explanation or both. That's what makes it a bad prompt for the task. Not because it's short, but because the agent has to guess what success means. Same thing as you told this prompt to a human. So now both models produced a correct list comprehension. The difference is how they got there. I used a heavier reasoning for this, probably unnecessary, 8.2 credits. Over here, I used GBT five Mini, only point six credits. Huge difference. Now just for fun, I also ran the same bad refactor using GPT five Mini, and it was still 1.2 credits. So not bad. Double what the good prompt was, but it also is asking me more questions because it's needs more clarification. So now this is not an apples to apples model benchmark in these two prompts because I did change the prompts and the model. This just demonstrates a workflow choice. So when the task is precise, like the one on the right, a smaller model is easily enough. The visible difference is 8.2 credits versus point six credits, but prompt quality is only part of the gap. Model choice is the larger lever in this example. Okay. Let me open up two more. Okay. So here I sent a vague prompt and a precise prompt. I used the same model. The bad prompt didn't even produce any code yet. Right now, it's asking me to clarify what the functions to calculate, and I'm already at 12.7 credits. This one complete at 13 credits because I was clear. I was concise. I defined the function, the rules, the error behavior, and the test so the agent can implement it immediately. Now the important part of this is the test acts as the specification. It tells the agent exactly what success means here. So now the vague prompt creates another turn before work even starts. My answer and the next model call carry the earlier conversation forward again. A little more useful context upfront avoids that growing clarification loop. Let's open up our last prompt. This is a debugging example. Let me make sure you can see all of this. Okay. So in this comparison, I also use the same model, and the immediate token spend is actually going to be very similar. So now the better prompt over here on the right, it still matters because it gives the symptom and the known root cause because I already know this. The agent can focus on the fix instead of rediscovering information that I already have. So this example is about reliability, not a dramatic one turn saving. One run may cost about the same. We're looking at 12.9 credits versus 13.2 on the same model, but the benefit appears when this pattern repeats. Fewer wrong diagnosis and corrective turns. And think about it. Within a long session or a very complicated problem, these tokens are going to compound. Every avoided turn also presents prevents more contacts from being resent in the chat. So don't judge efficiency just by the prompt length alone. Judge the complete successful workflow, model choice, output, clarification, retries, and correction. So the simple pattern is say what you want, include what you already know, define success, and tell the agent when to stop. Better prompts improve one interaction. Larger work also needs a better structure, so irrelevant context does not follow every phase. Next, I'm going to show kind of a research plan implement example. So this demo that we're gonna look at, this builds a markdown link checker in three separate sessions, research, plan, and implement. So I want you to watch what each phase passes to the next in terms of context. So I'm gonna open up the research notes and then the research prompt. So a little bit longer. But so now in the research, I asked it to output the research to this file. So the agent read the sample markdown file that I showed you earlier. It checked the file. It created a small to do list, and it wrote the research notes. It took several tool calls and maybe about one or two minutes. 2.3 credits. That's it. I was very precise in this prompt telling it exactly what files to read. And research is where broad exploration belongs. The next phase will receive these short notes, not the entire tool history. So the optimization here is not zero tool calls. It is containing the noisy exploration in one session and carrying forward only the useful findings into the plan session. So if we look at the plan markdown and the plan prompt, you'll see how specific I was in the plan as well too. I told it when to stop. I told it what to read, and I told it what to do. So now over here in the plan file, just a few little pointers. We have scope, the tests, how will we prove it works, definition of what done means, what's out of scope, what's in scope, public API. So now the generated plan is detailed, but the audience only needs these four anchors. It turns the research into a contract for implementation. You're accepting this plan. Now if we look at the token spend, we use a little higher reasoning model, and we use 56.9 credits because planning is some of the most important things. Now use expensive reasoning when it changes the outcome. Once the decisions are written down, you can just use a general purpose model to execute the plan because most of the planning went into the plan. So now we have the implement method. So once again, I told it what files to read. I told it when to stop, and I gave it very explicit instructions, and it reads the plan. So we can see that the test pass and ran all of the tests. Now there was a little issue. The editable install was not available in this in in this environment, but the agent adapted by calling pytest and rough directly, then it completed the plan implementation. Lightweight model, 4.3 credits. So the path was not perfectly smooth to get here, but the plan and the checks kept the task bounded and used very little tokens. Now test and lint return some additional tool output, but they usually cost less than speculative speculative debugging, review churn, or another agent session. So now takeaway for this, explore broadly once in research. Turn the findings into a plan that you agree with that's concrete, then implement against tests. Each phase gets only the context it needs. So this structure improves one task. Now let us make the good behavior repeatable so future agents can start with the right repository context and hand focused work to the right specialist. Give me one second while I close these out and we open up our agent. Okay. So here I have the agent's markdown file on the left. This gives every agent essentially the same repository context before it starts working. It can explain the project structure, the correct test commands, and important guardrails. So without it, an agent may spend time searching for those answers, choose the wrong command, or repeat a mistake we already know how to avoid. Now this turns repeated explanations into concise, reusable project guidance. Without it, it'll just look forever. Maybe not forever. But always on instructions are recurring input, so every line in this file must earn its place. One accurate command can prevent several orientation reads or a failed command cycle. Now over here, we have our custom agent. This is a read only code reviewer. This solves a different problem. So instead of giving the main agent every possible responsibility, we create a specialist with a clear role, a clear model, tool access, and output format, as you can see right up here. The model's specified, the tools are specified, and the description. So the reviewer has one job, read code and return a short prioritized list of meaningful findings. It uses a lightweight model and read only tools, so it cannot drift into rewriting the code because that's not what we want. A smaller model and a smaller tool surface reduce spend on every invocation. The main agent receives the useful conclusions instead of carrying the review's full working context. Let's look at the complete code review for this code reviewer agent. Simple. 2.8 credits, lightweight model. We have it listing out the top three concerns. We have one high and two medium, and there we see that it just finds and prioritizes. The main agent or us as a developer can decide what we want to fix. Now the extra sub agent session is not free. It is useful because a smaller model handles the boundary review and returns only a compact result to the main workflow. Now as you've seen, we've now made two improvements proactively, repository guidance and a focused agent creating a or containing a specialized task. But let's take a look at Chronicle. We discussed this in the earlier slides. Okay. So haven't used this repo too much, but Chronicle reviews my recent sessions, and it highlights patterns that may be wasting credits or context. The command does not save tokens by itself. The real save comes from acting on a repeated suggestion, for example, starting fresh sooner or adding one concise repository instruction. And then another neat little feature we have is the session window. So this gives us a little summary of what's going on here. We have the session cost. We have the context window. We have the tool definitions and the messages and tool results, and then we also have a compact conversation. So we went over a couple of things here, but across the three demos, the pattern is consistent. Improve the signal, reduce irrelevant choices, and create faster feedback. Let's finish up with a few slides here. So now I know we covered a lot, but here are the five biggest tips we can give you to improve your agent quality and token spend today. Split your tasks. You saw it in the demo. We did plan, research, implement. Choose the model or use auto mode. Provide clear guidance and goals in your prompts. Provide deterministic guardrails, think tests, linter, security scans, and maintain a concise human written copilot instructions file going forward. Shake anything out of this talk. It's right as little context as required and as much as necessary. So I just wanna say thank you to everyone who joined. And on the next slide, I'm going to show our QR code to complete the survey because we really do take your feedback seriously. Thank you, everyone. Have a great day.