Name: Agent Quality & Token Optimization (EMEA Friendly)
Uploaded: 2026-05-21T15:30:28.500Z
Duration: 36 min 44 s
Description: Agent Quality & Token Optimization (EMEA Friendly)

Transcript for "Agent Quality & Token Optimization (EMEA Friendly)": Well, hello there. So, today, we are gonna go through, an awesome webinar on agent quality and token optimization. So I'm Andrew Weiss. I'm the field CTO for our regulated industries business here at GitHub. We're gonna talk about ways in which you can optimize your use of GitHub Copilot, across the tool chain within GitHub. What we're going to talk about is we're going to talk about ways where you can optimize your token spend across the platform itself. I'm going to give you some foundational insight into large language models, how they work, how you can use agents across the software development lifecycle, how context windows work, and provide quality and token controls and practical optimization tips. And that's what we'll cover today. So we think about how we actually use agents and consume tokens within the software development life cycle. We have to understand that we can't necessarily gamble on how we use and consume agents and tokens. We can't necessarily just assume that whatever prompt we provide the agent, is going to be the end all be all for a given task, because tokens aren't necessarily cheap. Agent accuracy is, something that we need to take into account, because when tokens are cheap, agent accuracy is not necessarily as important. And that requires extra engineering work. And so we need to think about, more consistently how we actually consume tokens across the SDLC. Because we think about how quality impacts the ROI, the value of the agent output is going to necessarily address the ROI issues that we have across the spectrum. Because by increasing the value of the agent output, that's honestly going to reduce the token costs over time. Because when we think about agent error rates, these issues are gonna compound over time, because agents and LOMs are nondeterministic, because over time, agent errors are gonna compound. So for instance, let's assume that an agent error is going to an agent's gotta be accurate 99% of the time, but 1% of the time it's going to have errors. And so when we provide agents into the SDLC, those errors are going to compound. So let's say we start with one agent to do a single task. The next agent is going to potentially have additional errors that occur. And over time, these errors are going to compound. And so we need to keep this in mind as we go through our agent execution pipeline. And so rather than counting tokens, we need to make sure that every token counts in our SDLC. So let's talk about large language models, agents, and context windows for a second. Large language models are simply prediction engines. They've seen patterns over time. And so in this instance, we give it a prompt to get a copilot of most worlds most widely either adopted or AI developer tool. And so it's simply going to predict based on patterns it seemed your next suggestion. So when we think about this in the context of using GitHub Copilot, we wanna provide as little context as possible, but as much as required. If we don't provide any context for GitHub Copilot, it's going to struggle to produce a suggestion. But if we provide it a little bit more detail, it's going to give you a more accurate prediction based on the patterns that it's seen. So this is some the example in which I use GitHub Copilot today. One of the ways in which I use it is to analyze my past meetings and conversations that I've had and combine that with implementation tasks that I have at hand. So let's say I wanna build a value map example for a particular customer. I wanted to analyze my team's conversation with that customer that I've had that's been recorded. And I wanted to go in and actually implement the design that I have in mind based off the conversation that we've had and examples that have been provided to me. And so that's an example of a prompt that provides a little bit more context, but doesn't necessarily give it too much insight into the given task. I didn't give it give it necessarily design specifications, or I didn't even give it a technical stack that I wanted it to implement. Although understanding the technical stack is definitely going to be key. So when we think about how we actually work with an agent day to day, you, of course, need to understand the project that you have at hand, the different ways in which you can interact with the agent, define the agent's behavior, the capabilities that the agent has such as sub agents, custom instructions, so on and so forth. The interaction endpoints that you want to provide with your agent, such as the chat endpoint, the CLI, the cloud agent, so on and so forth. And also understand that it's simply just text that's being used across your agent interaction chain. And then it's also prudent to understand the LLM that's being used by the agent itself. You don't necessarily have to use the latest and greatest frontier models to provide context to the agent, but some models are going to perform better than others based on the task at hand. So for example, if I'm working on documentation tasks, I don't necessarily want to use the latest and greatest Frontier models like Opus four seven or GPT five five. Maybe I wanna use one of the mini models to support that task. So how do we actually interact with an agent? What does the context window look like? And how do tokens actually flow through this context window? The one thing to understand is that every interaction that you have with an agent is going to compound. So let's say you provide it with a system prompt and tools, that prompt is gonna include not just the prompt that you gave it, but any additional files and context that it captures, such as the code that you provided. It's also going to include the output tokens, which is of course the response. But every iteration is going to compound. So all of those input and output tokens are going to be cached but not guaranteed in subsequent prompts and subsequent loops. So that when you execute another interaction with the agent, it's going to build upon the stuff that you provided previously. And so we think about this from a context perspective, you don't necessarily want to just maximize your context window with stuff that you don't necessarily need, because most of the context is actually going to be lost in the middle of the chain. Your best bet is to provide context, both the latest context tokens that you provide and then also the earliest tokens that you have within your window. Because model bias tokens models will bias the tokens from the end of the context more so, and also at the beginning of the context window. So that's something to keep in mind is that you're very effective at prompting the agent at least at the outset and then also towards the end of the, conversation. So let's talk about quality and token controls for a second. When you're actually providing context to an agent, if you do so in an ad hoc fashion, those initial prompts aren't necessarily gonna care about optimizations. As you deploy more agents across your SDLC, and use maybe things like fleet mode within the CLI, the effect of the optimization that you provide is going to compound over time and could be more impactful to your token spend. If I try to fully optimize for my agent infrastructure, but I'm simply ad hoc requesting things from the agent itself or asking simple questions for things like documentation, the effect of that optimization is not necessarily going to be seen versus when I actually implement optimizations at scale and I provide context to multiple agents across my SDLC such as the Cloud agent and also the CLI engine. So what are the two biggest levers that we have for optimizing token quality and context windows? Of course, we can choose between different models. And like I said earlier, we don't necessarily have to choose the frontier models for any and all tasks. We also have mid tier models that we can take advantage of, and lower tier models like the mini models as as well. And one of the cool features that we have in the platform today is an auto mode function. You can think of the auto mode as the lazy default. Auto mode today is going to, direct your task to the model, that is optimized to support the response. Today that's based on availability, but soon that'll have an intent driven capability such that, when you provide it in intent or a prompt, it's going to figure out the best model to support the response for that. So if I ask it about generating documentation, it may choose a mini model versus the Frontier model to support that versus if I ask it to rearchitect something more involved, it's going to choose a frontier model. And so when you think about this from a context perspective, like I said earlier, you wanna provide as much as required, but as little as actually necessary to support that interaction. And context engineering is ultimately being used as your guidance and upskill potential. The better you understand how to provide context and manage your context windows, the more efficient you're going to be in your interactions with GitHub Copilot. And there's ways in which you can optimize that too. So you can compact your sessions, as needed. You can execute things like chronicle within the Copilot CLI to understand your usage patterns and get feedback for how you can optimize it over time. And you can definitely clear your sessions, of course, and that's something we recommend over time as you're using the CLI and some of the other tools to ensure you have a fresh context window if it starts to get poisoned or stale, with too many tokens consumed. So let's talk about how you can actually guide the agent. So we start with the systems and the tools. Of course, those are always available. And your prompt is always something that you're gonna be able to provide to the agent itself. You, of course, wanna be precise, and you wanna add stop signals. So if the agent, for instance, is going in a direction you don't necessarily want, you can say, hey, stop this task, resume, or provide a condition. So let's say I am trying to re architect parts of my system stack, and I don't necessarily want to focus on the UI, I just want to focus on the API. I can simply say, hey, focus on the API. Don't make any changes to the UI. But if you end up having to make changes to the UI just to support some feature, stop in that path. So that's something that you have to keep an eye on depending on the condition that you're providing it. And of course, you wanna provide known context. So known context could be, files and folders that you have available to it. That can be your Visual Studio code session. That could be a web based chat within github.com. You wanna make sure that you're providing that context beforehand because that's gonna be key. And there's also different ways in which you can evaluate your prompt, using the AI itself. So there's a function within GitHub Copilot CLI, which we call the rubber duck mode. And rubber ducking allows you to plan and verify before you actually execute the implementation tasks that might consume the most tokens. And the rubber duck mode is actually going to take your prompt and analyze it against different large language models. So for instance, I'll use the documentation example again. As I'm asking it to document my project, I can say, hey, before you execute the rearchitecture, I want you to verify my documentation steps, against another model, and the rubber duck mode will do just that. And so planning and research is really going to be key from the outset rather than just focusing on the implementation, because that's going to give you some better insight into how the agent's gonna actually execute the implementation steps. And, of course, Fleet Mode is definitely something that you could take advantage of. So you can execute Fleet Mode to, run different sub agent tasks within your local system using the CLI. And Fleet Mode will automatically analyze the task that you have and determine the best way to allocate the substeps of that task across different, agents, running on your local machine. And like I said from the outset, you wanna really avoid compounding errors because every agent execution is going to introduce potential errors into your context chain. So for instance, agents being non deterministic, if the agent is correct 99% of the time, but it errors out 1% of the time, every subsequent agent task is also going to incorporate those errors such that if you execute a fleet of 10 or 20 agents, by the time you actually finish the task, the error may might be up to 50% or 60%. So you want to be deterministic in the way in which you interact with the agent itself, and provide verification controls into that context window so you don't necessarily waste the ICD cycles in minutes, and, of course, AI credits to support that task. So for instance, you can build in unit tests across your, rather, for example, if you're trying to use an agent to verify unit tests, and code coverage across a different project, you can make sure that you have that context provided from the different source code files. But if you don't necessarily have unit tests in there and you're trying to write unit tests and it fails per se or your code coverage alignment is not necessarily what you want it to be, it's potentially going to consume more tokens across that session. So let's talk about all the different ways in which you can actually configure the agents themselves. So, of course, there are things called Copilot instructions, which allows you to define how the agent behaves within a given session. That custom instruction is simply marked down, and you give it a persona within a given project or you can set that at the organization level, and that will guide the agent every single time that it accepts prompt. You can also create custom agents, and custom agents are effectively personas that you wanna give the agent itself, and it's effectively a role that you wanna give it. So you can have a custom agent that acts as your unit testing and verification agent. You can also have a compliance agent or documentation agent. Skills are similar to instructions in that, they guide that agent's behavior, but in a more tailored fashion. We'll talk more about skills in a second. MCP, you can think of as the glue between all the disparate data sources that you have and how the AI interacts. MCP allows you to integrate all of your data sources within GitHub Copilot, and you can have MCP servers that integrate, of course, with GitHub data. You can have MCP servers that integrate with things outside of GitHub. Sub agents are basically the roles that you provide, excuse me, the agent itself to execute more scoped tasks given that role. So for instance, if I wanted to find a custom agent that acts as my unit testing agent, the sub agent is going to potentially use that persona to go in and execute a tailored task and then provide a summarized output back to the main agent. Great for research and tasks and so forth. We'll talk more about that in a second. Scoped instructions, of course, are similar to persistent instructions in that you can define more conditions in how every agent execution takes place. Prompt files are just reusable prompts, and then memory is a way to share persistent context across multiple sessions. And so we'll talk about a little bit of each of these in a second. So let's start with persistent instructions. So persistent instructions, reside at the root of your repository. You can also define persistent instructions for, not just every agent interaction, but for the agent itself. We embrace open principles here, of course. So agent MD is a way to define a common instruction layer for the agents themselves. This is something that you wanna put in all of your projects and don't necessarily use AI to generate all those custom instructions. The purpose of the custom instruction and precision instruction is for you to guide the agent on every task. And so if you let AI just automatically generate the custom instructions every time, you're not necessarily gonna get the desired output. So there's still a human in the loop here that's required to support the custom instructions. And it's key to consistently keep them up to date, iterate, maintain, and potentially recreate them as you see fit. So if we talk about custom agents, custom agents, like I said, are personas that you provide the agent itself. You can create a custom agent, like I said, as a unit testing agent, you can complete create a verification agent. And the custom agent is going to execute the task based on the instructions that has been provided for its given persona. It's going to encapsulate all of the system tools and props that you provided it previously along with those custom instructions that you might have. It's going to basically give that agent a lens to execute that given task. Custom agents, again, are simply markdown files and plain text that you provide the agent to guide it in its, execution check chain. Like I said earlier, skills are ways to define behaviors for the agents themselves. So you can combine, of course, custom agents with skills. So if you have a custom agent that acts as your unit testing agent, a skill can give it specific behaviors such as for all the unit testing tasks I want you to do. I want you to really focus on the API component of this project and that's going to be your best skill per se as you are the master at developing, all things around unit testing for, the API endpoints in my application. Skills can, of course, be combined with custom agents and then all the instructions that you have. So you can see the the context window starting to compound here, but you're providing conditions so that you can optimize subsequent executions of the tasks that you have. Like I said, MCPs are ways in which you can automatically pull disparate data sources into that context window. So let's say I want to, have mine. I'll go back to the example I've been using is I want my unit testing custom agent that's gonna be focused on the API portion of the application to pull in all of my blueprints and architectures that reside in, m three sixty five. And so if you're not familiar with WorkIQ, WorkIQ is a way to automatically pull down the m three sixty five graph and connect your data sources from M three sixty five. Within the Copilot CLI, I can simply prompt my custom agent to analyze the architectures that reside maybe in SharePoint, pull those down, and automatically add that into my context window to execute the prompt that I've given it. And there's tons of different MCP integrations that are available to you, today. Ones that we provide and officially support and also integrations outside of the GitHub ecosystem. I mentioned sub agents already. So sub agents are a way to tailor the agent to go out, do a little research, and provide a summary back to the main agent. You wanna be cognizant of how you're going to effectively use sub agents. Like I said, it's a great way to research a task for a main agent, but you don't necessarily want to spawn sub agents for anything and all the things, simply because it adds additional context and unnecessarily, consumes AI credits from your pool that you don't want it to consume. So that's something that you'll want to iterate over time and figure out the best way to use sub agents for your given use case, and we can talk more about that, towards the end. And there's other agent configurations that you could take advantage of. There are scope instructions, of course. So these are conditional instructions based on things like file path patterns, or file syntax. These are, of course, nondeterministic. So they're offered up to the agent the same way as, skills, and other instructions that you provided. Of course, prompt files are reusable prompts that you can share across your organization and within your repository. And they can be used to trim the tool sets you have and invoke potentially custom agents. They're a great way to start, but you'll have to iterate on them over time. These are things that reside within your source code repo alongside all of the other source code. And so you wanna keep in mind that the prompt files are not necessarily the end all be all. There's something that you're gonna wanna iterate with as you do the rest of your application stack. And, of course, Copilot memory. This is a powerful feature because it's effectively always on instructions that are made available to your agents. And we're improving the ways in which you can manage memory and state across your different sessions. And they're used across Copilot services uniformly. And it makes sense to regularly check how you're consuming memory across your session. And there's different ways that either the CLI, for instance, will automatically checkpoint your session state in ways in which you can manually do that as well across your, agent supply chain. So all different configurations that are made available to you, it's prudent on you as a developer to understand all the different ways in which you can prompt the agent itself and understand all the different configurations, that, of course, AI can use. You can use AI to help you define, but all the different ways in which you can manage your interactions with the agents themselves. So as a power user, let's say you are an expert at managing your, interactions with the AI. You understand all the different nuances of the CLI itself and all the different endpoints for managing your interaction with GitHub Copilot. You wanna think in terms of optimization tasks. It's not just about focusing on the implementation steps. You wanna understand all the different, augmentations you can provide the CLI, for instance, to optimize that context window. So you can consider using the CLI versus MCPs because if you use MCPs for every single task, it might add unnecessarily an unnecessary bloat to your context window. You may not necessarily need all the data that an MCP integration, while it provides all that data, may not necessarily need all of that for given tasks. Going back to the unit testing example, I may not necessarily need all the blueprints on architectures for my application stack in order to actually implement a given set of unit tests. And there's also different ways in which you can collapse your tool calls. There are open source projects out there. One such example provided, that can be used to collapse and compact all the different ways in which you're interacting with the agents themselves. So you don't necessarily have to bring in all of these external sources every single time. We mentioned auto mode, in that will automatically direct your request to the right model to support a given task. Super powerful feature. That's something I use pretty regularly anytime I interact with GitHub Copilot. But these are all things that you can take advantage of as a power user as you start to get more comfortable with the interaction endpoints within GitHub Copilot. So as long term guidance, how are ways in which you can or what are some different ways in which you can optimize, your context window? And how can you kind of evolve your understanding and get a Copilot to support token optimization tasks? So the coding itself, the implementation details are really the last thing you necessarily need to be worried about. These are tried and true patterns that AI is really good at implementing. I can't remember the last time I opened up my ID to actually start coding. I simply rely on the AI to do all of the coding and implementation tasks. But my role more so is spending time actually defining the behaviors and understanding the domain that I want the agent to work in. So if I am building an application stack for a verification system for, let's say, to support a regulatory audit, my time is going to be spent more so on understanding the regulatory audit guardrails than the actual implementation of the agent and its execution task. By understanding that domain and understanding that problem statement, I can more effectively prompt my agent so it optimizes for the given task that it has at hand. That goes hand in hand with understanding good architectures, Understanding things like domain driven design, CQRS, event driven design. These are all things that are prudent on me as a developer to understand so that I can more effectively prompt the agent. And if I'm not necessarily sure on different parts of the architecture and ways in which my organization is trying to implement these architectures, I can simply ask the agent, or interact with other teams on top of GitHub to get that context for me and, of course, use AI to automate that data flow. Because at the end of the day, it's gonna give you stronger guardrails. It's gonna avoid excess agent sessions. It's gonna avoid all of that back and forth that you maybe have with that agent anytime it executes a task or it has to constantly debug, and so on and so forth. And like I said earlier, it's key to continually iterate on the prompts and the agent configurations that you provide. Chronicle is a super far powerful command that we provide within the CLI itself, because Chronicle is going to be a way for you to analyze your users panels with the CLI and figure out the best way in which to execute tasks going forward. That's something I use pretty much every single day that, I'm using the CLI command available to me. So what if there's some things that you can start doing today? First and foremost, understanding which models you're going to leverage within your organization and understand which model is the right model for the right task. Like I said, you don't necessarily need to use a Frontier model for every single task that you provide an agent. Automotive is a great way to start doing that. But as you start to get more comfortable with the models, you'll find that some of the more lightweight models are very powerful for doing the majority of your work. I use a lot of the mini models, for instance, to analyze and document a lot of my code and work through my day to day tasks. But I'll use the frontier models for more involved rearchitectures and so on and so forth. It's important to also be clear and consistent in your prompting. That's something we already touched on. And spending time researching and planning is really, really key. Like I said, I never start directly with the implementation task. I always guide my agent to focus more so on the researching and planning upfront. So that way when it gets to implementation and already has that data, it already has the most optimal way to actually do the implementation. And I'm not necessarily spinning cycles, doing additional research tasks after or when I actually execute the implementation. Mentioned this already, of course, providing deterministic guardrails. So any ways in which you can provide verification checks harnesses into that context chain is definitely going to be key. And of course, security scanning is is a big part of that. And providing human written instructions is really going to be the best way to optimize your interactions with GitHub Copilot. If If I simply use AI to generate any and all the things, then there is no longer a human in the loop. Then there's no way for me to verify because I don't know what the agent necessarily generated for me to verify against. You end up in this loop of unmaintainable artifacts that your AI is using to do the important work that you have in front of you. So those are some top level things in mind. I think that was the majority of the content we had today. And I think there's some questions. I'll pull them up here in the chat. Let me see here. Let me take a look at the questions. Can I provide you provide instructions, preference for model, doc, and coding? You certainly can. Yeah. So while there's you can provide all of that information in your custom instructions. There's no guarantee that the custom instructions are actually going to use that exact model. But the best way to do that is is, of course, when you execute the prompt, you choose the model that you want it to execute against. Let's see here. Best way to handle agents on MD, small human written info. Yes. You don't necessarily wanna bloat the agents on MD file. It's something that just exists alongside your code base and your version control like anything else. But there's, all sorts of agents MD optimization tasks that you could take advantage of, and that are provided within the agents that MD open source community. And we provide examples as well. Will or does Chronicle have access to token spend data? Not necessarily. It's just going to figure out how you've actually interacted with the agent itself, previously and provide guidance, and potentially recommend different ways in which you can approach your session task. But it doesn't necessarily have token spend data, at least at this point. Is there a Versus Code Copilot chat equivalent of Chronicle if GitHub Copilot CLI is not installed? Not at this point. So Chronicle is specific to the CLI itself. In which OS does the model Copilot CLI work best with? It works best with all of the operating systems. So we support, all the OSes, and there's no one OS that works better than the others. I see bosses answering a few extra questions here. Yep. So it looks like there's a good understanding of some of the concepts are concepts already talked about. Is every user power user separate login, or license? So all these functions require I get a Copilot subscription, from the outset, unless you're bringing your own model into the CLI. Ultimately, all of your users are gonna be power users as they get more comfortable with how they actually interact with the the tooling. As you get more comfortable with prompting it and optimizing your context windows, you could argue that everyone will be a power user within your organization. Let's see here. Scoped instructions on the pre by the way, these questions are all fantastic. I love it. Scoped instructions in the previous slide, is that like personalization and normal n three sixty five Copilot? Kind of, although it's specific to GitHub Copilot itself, but it does work similarly to how you can personalize m three sixty five Copilot. The more appropriate equivalent is actually gonna be a custom agent, because a custom agent is basically giving the agent a persona. It's giving it an identity, and it provides a better way to guide the agent for tasks that are more involved, such as acting as a researcher or a unit testing agent. CLI only. Yep. On a rubber duck mode. Great question from David. You mentioned sessions could be poisoned and need to be cleared. What is the hygiene actions DevOps should perform and how often? This is really gonna be dependent on the use case that you have, in front of you. So sessions can be easily poisoned, in the event that you're starting down a path that you didn't necessarily want to go down. So the agent's kind of going off the rails, and going back to that example I provided earlier where you really wanted to focus on API tasks, but it's starting to make changes to the UI and you're having to stop it and start it. Those are all wasted tokens that you otherwise could have used for different tasks. So, of course, Chronicle is the best way to do that and simply starting a new session. Figure out the reason why it was executing a task the way it was executing it in and just simply start a new session. Compacting conversations does not itself consume tokens. Also a good question. How can we really improve our agent configurations? So you can do that, of course, manually as needed, but using AI to optimize the agent configs themselves. So you don't necessarily have to bog yourself down in the different configuration syntax. But AI is a great way to kind of at least get you started off with some good agent configurations. But reviewing them manually is definitely gonna be key. Memory is not necessarily is not shared within the team. Let's see. I feel like sometimes to reread instructions, perform some tasks. Yes. So sometimes it may miss the instructions that are provided to it, although it's usually pretty consistent at inserting them out of the gate. It really depends on how you're prompting it and ensuring that those instructions are there, upfront. From my understanding, the CLI comes with built in harness and modes. Is it beneficial to use custom agents or skills? It really depends. You don't necessarily need a custom agent for any and all things, because if you don't necessarily spend the time upfront to craft how that custom agents that executes its tasks, it may go in a completely different direction and consume tokens you don't want it to consume. And so starting off with plan and autopilot mode is is definitely favored over digging right into a custom agent. But it it really depends on the the use cases. Does MCP another good question here. Does MCP use more input output tokens compared to skills? It definitely can. If MCP is configured to execute all instructions, for all the different things that it can connect to, I do not necessarily have a guardrail on that. Then, yes, MCP could potentially use more input output tokens compared to skills. Any tips on how to keep the, instruction files small, you should definitely start with the KISS principle. Keep it super simple. You don't have to blow your custom instructions in order for them to be effective. I work on 20 repos in a given week. What is the best price to keep the agent config consistent across repos? Ideally, version control, the custom repo configs only where needed. Great question. And this is where the GitHub platform comes into play. Because when you manage all of these things within the GitHub platform and use things like the GitHub MCP integrations that we provide, you can pull in that context automatically from all those other repos that you're working within. And with that context, you can ask AI to ensure that the custom instructions and the agent configurations that you're using within one repo are consistent across all of them. And you'll start to identify drift. So if you have a number of different repos that are starting to or you're implementing things in a little bit of a different capacity and some of those other repos than you are in the main repo you're working within, you can start to see that drift take place and you can ask the AI to address that for you. Yes. So you'll be able to see your AI credit consumption, within the, within the UI itself. Skills and custom agents, like I said earlier. So custom agents, again, are personas. Skills are behaviors that you can give that particular persona. So, going back to the example I keep using here, I can have a custom agent that acts as my, unit testing and verification agent, But I can give it a skill such that it only focuses on the API portions of my application stack. And it and it can be really good at just that rather than some of the other aspects. Let's see here. These are all I I love all these questions. This is great. A lot of them are being answered here. Yep. Session will be recorded. With that, I think I'm going to end the session for today. Otherwise, really good questions. Appreciate everyone for taking the time out and joining. And I think we'll call a rep from here.