Video: GitHub Copilot Billing Is Changing ——— AMER/EMEA Partner Office Hours: Usage-Based Billing | Duration: 3316s | Summary: GitHub Copilot Billing Is Changing ——— AMER/EMEA Partner Office Hours: Usage-Based Billing | Chapters: Welcome and Introduction (3.12s), Pricing Model Changes (101.05s), Usage Tracking Dashboard (397.69s), Dashboard Features (686.08s), Billing Report Details (813.77s), Pooled Usage Budgets (1029.875s), Usage Forecasting Tools (1099.865s), Billing and Tracking (1332.285s), Billing Metrics Insights (1473.575s), Model Pricing Details (1614.285s), Adoption Patterns Analysis (1735.66s), Model Router Updates (1927.19s), Metrics and Feedback (2123.78s), Budget Notifications (2239.34s), Azure Subscription Restrictions (2365.325s), Regional Data Residency (2496.445s), Customer Feedback Challenges (2597.23s), Auto Feature Timeline (2836.315s), Q&A and Closing (2897.235s), Closing and Resources (3071.925s)
Transcript for "GitHub Copilot Billing Is Changing ——— AMER/EMEA Partner Office Hours: Usage-Based Billing":
Hello, everyone, and welcome. Thank you for joining us today. I am Irina Smigelski, and I lead partner scaling within the GitHub Learn team. I'm very excited to welcome everyone to today's office hour. So we're glad to have you with us. This is gonna be an open office hour where we answer your questions on usage based billing. But before we get started, just a few quick notes. First, please feel free to submit questions through the webinar chat throughout the session. We're gonna do our best to address as many questions as we can, and that's gonna be either during this discussion right now in the next hour or as follow-up afterwards. And this office hour is also being recorded, and we are gonna post the recording to the partner portal's usage based billing landing page. And with that, let's go ahead and get started. Today, we have Robert Freeman, who is a strategic co pilot business lead, and he's with us to help answer your questions. We also have Geraldine, Camilo, and Margarita on the partner team in support as well. So to start us off, we actually have had several questions that were already presubmitted before the webinar. So we're going to start off with a few of those, before addressing your questions through the chat. And as we address those, please feel free to start, throwing your questions in here. So with that, I'm going to turn it over to Robert. Robert, go ahead. Appreciate it. So, we already have a handful of presubmitted questions. We'll, we'll start with those, and then as mentioned, feel free to, toss additional questions in. So for our first question in fact, I'm I'm gonna cover, first two because they're they're basically related. The question is removal of the enterprise bonus. So in the previous model, enterprise users received a thousand PRUs, more than tripled the business plan's 300 PRUs. Why is this high volume bonus being eliminated in favor of a one to one credit ratio? The second question, like I said, they're they're related as cost per unit increase, Power uses the shift from a thousand peer use to $39 in credits represents a significant reduction in total monthly AI capacity. So let me let me hit on the first piece. And and like I said, they're they're fairly closely linked. The reality is this is this is purely cost driven. Anyone that's, followed this industry, the AI industry, is well aware that these plans have been heavily subsidized. When we originally launched GitHub for Business and GitHub, for enterprises or Copilot for enterprises, we lived in a a pretty different world from an AI inference standpoint. A, an AI request was primarily code completion. PRUs were primarily one off, requests. Reasoning models were just kind of starting to roll out, and the idea of tool calling and engineering didn't exist. So a PRU, something that from an overstandpoint, we charge 4¢ for currently. At that point, you know, usually ran, somewhere somewhere in the neighborhood of about two x that, somewhere around, you know, $8.08 plus cents, to provide from a cost standpoint, from a COG standpoint. Now, a, a PRU or request can cost tens or more dollars per request. So from a long term stability and sustainment standpoint, like, can GitHub continue to offer Copilot as a viable product? We are, we're we're forced to make this change to tie the actual requests from end users to what it actually costs us. As far as the erosion on enterprise bonuses, you know, that the the reality is that increased spend didn't justify the significant increased discount. We are in a promo period, for the first three months of this change where enterprise still gets a cost advantage. So instead of, 39 and dollars, you know, there's $70. During this period, we will reevaluate, those, those, those values, and determine if, if we're capable based on margins to, to expand the usage pool, for for the enterprise SKU. But as of right now, you know, those will remain 19 and 39, coming, starting on September 1. Post promotion strategy, this guy just kinda dovetails in. After the temporary $70 promotional credit expires, will enterprise customers receive a permanent multiplier discounted token rate to restore the previous 3.3 x throughput advantage? The the short answer is, no. The longer answer is kinda what I alluded to. We are evaluating if there are additional discounting that can be done at the enterprise SKU to provide, more, inference value. Right now, the two SKUs other than the inference value, are are roughly identical. Alright. What is the rationale of GitHub introducing AI credits when providers like OpenAI and Anthropic already use tokens as a standard unit of a measure? So a little bit of reframing on this. OpenAI and Anthropic, don't entirely use tokens as you measure. And by that, what I mean is have they used currency? So you you pay in currency just like we are, moving to this, AI credits. And the reason we're moving to AI credits versus a straight currency is that we operate all across the world, and we can provide, basically, a universal, translator for, for how, credits are bought without having to worry about currency fluctuations. We are still measuring the back end by tokens. The reason I mentioned that that tokens, are not necessarily the unit of measure, the the token times the model is the unit of measure. So different tokens, different types of tokens, whether those are input tokens, output token tokens, cash read tokens, or in anthropics case, cash write tokens. And then the model themselves all dictate what the what the rate on each one of those is. So AI credits directly tied to the US penny or 1¢, and and that will, allow us to kinda stabilize, what those rates look like. Great. Robert, we actually have some questions that came in. Let's go take a few here, and then we can go back to the presubmitted list. So the first question is, around trying to track usage with the dev team. That's pretty difficult. The best way we can get it is exporting out an Excel file, which gives us arbitrary numbers. With AI credits, this will become a huge problem. Are we making the dashboard easier to those who use the credits, and what will management of the caps at the user enterprise level look like? Yeah. Hi. So a handful of questions there. So let's start with, with with the data part of it. Right? So at launch on June 1, what would be available, somewhat similar to now as far as being available as a CSV file, which can obviously put in be put into Excel or directed into your, you know, whatever your enterprise reporting tool is, as well as via the Cobalt metrics API, will be, line level, and by that mean effectively request level detail on all requests that will tie to individual users, it will tie to their, their cost center, it will also tie to it will include the, the token information, so tokens in, tokens out, cash, tokens, cash right tokens, as well as the the model. So you'll have fair bit of fidelity on individual oh, and and individual developer, maybe who made the request, into, all, requests. Now what will not be included in that is what the request was. So intent, that type of data is something that, we are trying to figure out, if if there's demand for us to capture that historically in the AI space, there's been strong demand not to store that information, so that has been our default in the past. But as people look deeper into not only the the, the amount or volume request, but also, looking at intent, that's something we're looking to figure out if if that's if that's data that's, being asked for and and worth the, you know, the privacy concerns with us storing that data. Now, that being said, from a budget control standpoint, budgets are, executed at the enterprise, the cost center, and the user level. The new construct here, the new piece that's being introduced is user level budgeting. That previously did not exist within GitHub, and so a a standard user level budget can be applied across all users. And then, for exceptions, power users, those can be, either overwritten directly in the GUI, or via the API that can be programmatically done. We are what currently, does not exist is cost center specific user level budgets. That would have to be done via, the API. That is a a feature that is on a road map and is for fast follow. I don't have a date, for that, but ideally, that will be, relatively, quickly. Some nuance to this, this is important for setting for budgets, is right now the budgets are, so any budget that that currently exists at the cost center enterprise level, those will automatically transfer over. Since usual budgets don't exist, those would have to be created. The nuance to this is the user level budgets are all usage, so, pooled, any pooled entitlement usage plus any overage. So, for example, if you were to set a user level budget to, $50, so the budgets are in dollars, if you were to set that, then, the user would be able would use, would use both entitlement and overage to get to that, depending on what's available, if there's entitlement, left. At the call center, the budgets are purely for overage, and at the enterprise level, it's purely for overage. And so you would set the budget based on how many dollars, or or AI sorry. There are a lot of dollars and how many dollars that you are allowing it to exceed, your your current spend. So, those, again, those transfer over from previous, budgets, but use it all the budgets will need to be set. The reason I reiterate those, is that, because we're moving to a pooled model, an individual developer's, entitlement is no longer protected, and so you could have a small group of power users, consume a significant amount of new entitlement if there is not a user level budget set. Great. Another question on the dashboard. Do we have a date for the availability of the new dashboards and also a little bit more detail as far as what kind? Yeah. So on on June 1, there will be, again, there's a there's a data export for usage itself. There's also a dashboard that's going to show individual users, or so it'll look it'll show your your your enterprise and your cost center budgets and where you're tracking toward those as well as your your, total entitlement pool and how that's tracking. It will provide notifications at 75, 100% of entitlement pool, as well as of overage budgets. You will also see individual line items for each user and where they're tracking against their user level budget. So we wanna set, that will include the ability to override, by clicks. So being able to go in and say, oh, well, Robert is approaching his budget or he is at his budget. Let me click on that and increase his individual budget, because he's blocked. Or let him know that, hey. For the rest of the period, you you've extended it. What this is always a follow-up question, so let me ask answer this in in case someone has this question is, what happens when you reach your budget? Currently, the way that, GitHub works is you would, rotate to auto mode, and auto mode, because there's no budget, would automatically default to no cost models. There's no longer any no cost models. So if you have reached your entitlement or budget limit, a user would not be able to use, any of the features that require, effectively AI inference. The one exception to that is code completion or inline, completion will, is unlimited, so that will continue to function, within your, you know, your IDE of choice. K. Great. Next question. When, how, and where can we expect to see or get the sample billing report using the new pricing specifically for the April usage? Yeah. So that is supposed to ship, this week. I'm looking to see, on my Slack and see if that it's that that's gone live yet. But the the target is, for end of the week. And that so. that what that will include for anyone that's not aware of that is that will include a poll of April data that's mapped between PRUs and tokens. And so you will be able to see effectively, like, what your April bill, was or is, have mapped to what it would have been under the new, usage based billing model. That will also include that same aggregate or sorry, that same, low level detail that I mentioned previously. So you within that, you also be able to see individual u user usage. The key part of that is what we found is in most enterprises that a very small group of users, you know, oftentimes 5% or under, represent, you know, 50 to 90% of, of actual over juice. And, oftentimes when, accounts are looking, your customers are looking and seeing, a significant increase in overall price, a lot of times that, you know, some amount of of coaching of of that small group of users. Oftentimes it's, you know, sometimes we have a thousand seat accounts so, you know, kind of kind of a medium, to small size account that has two or three users that are, consuming, the the bulk, of usage, for various reasons. Now that that can be various reasons. Right? Like, that that can be, that because they're doing things that are adding value to the account and and the spin needs to stay there or you want to stay there. That can also be any number of other things like working on side projects or inefficient use of the tool, inappropriate model selection. I shouldn't say inappropriate model selection. Not not ideal model selection. The majority of overages, about 75% of overages are driven by, the OPUS family of models. A study that was done by, NVIDIA and Georgia Tech, late last year found that 60 to 80% of tasks could be completed with, small models. Small models are, significantly less expensive. Also at the frontier levels of OPUS level and the GP, five forty five level, you know, we've reached mostly, intelligence parity on theirs. So what we found is, oftentimes, a good portion of those tasks, using a model, a competing model will, reduce cost significantly. Some of that is driven by the idea that, and and a lot of people are not, aware of this, but, while Opus four seven, four six, and four five are more expensive than the likes of GPD five four from a per token standpoint, they're also significantly less token efficient, traditionally using up to five x the number of tokens that are used by, GPU, five four and even five five, which has price parity or actually slightly more expensive than Opus, uses again significantly less, tokens. So the cost can, can be significantly less as well. So balancing, availability on on models and using models, that are appropriate to the task, can significantly reduce, spend. Okay. Okay. Great. Another question on budgets. How does the pooled usage and budgets work when users are matrixed across multiple different cost centers? Yeah, so cost centers, so the only, piece that controls pooled usage is user level budgets. So, the budgets for cost centers and enterprises currently are only for overages. We've gotten a lot of requirements, that are looking to say, hey, we we wanna also, budget at the enterprise or at the call center level, for well, more more specifically at the call center level, for, for the the pool, component as well, or the entitlement. And so we're we've we've kinda identified that as a as a requirement and are looking to to scope and identify when that can ship. Ideally, that ships very fast, but right now, the the require or the the, way that you would accomplish that is the use of, of user level budgets. Mhmm. Okay. K. Great. So this question is around kind of preforecasting. Is there any way to calculate usage now and how that might translate to the new cost depending on models? We're trying to get a sense of how much additional spend we may have. Yeah. So, this is brought up a a few, a question or two ago. We are shipping a, a kind of two phase piece. One is, a a data export. It exports as a CSV file that shows, actual usage for all of April, both in the existing, billing system, or or construct of PRUs as well as the the underlying tokens component of that. That will give you, an estimate of, you know, what your April, bill, your your most recent bill would have would have looked like, under the new construct. There is a, if you are not porting that to, you know, your own, BI tool or business intelligence tool, there's also what we call the sidecar and that is a temporary application that we, that we built to allow folks to basically, ingest that CSV file, and kinda slice and dice it and show what that impact would be very quickly. The reason that that is a kind of a sidecar application is that it is temporary. So it did not make sense to ship in the core product when it would just go in when its usage is only relevant for the next, you know, couple weeks. But that that'll allow you to see kind of point in time this is what people look like. I would, caution with you and any of your, your own customers that are using these tools, to take that just as a wholesale and saying, like, oh, there was this bill. That's what it's gonna be next month. The vast majority of our customers, their their usage as they mature, from an from an AI developer standpoint, is is increasing fairly rapidly. So it would not surprise me depending on the customer if that number can could grow anywhere from, you know, 10% to, exponentially high, you know, a 200%. We we've definitely seen groups that have gone from, you know, say, again, that thousand seat customer where 10 users are using the majority of inference, and then the next month, you know, 20 or 30 more users have been trained and enabled, feel more comfortable with the tools, and and significantly up their usage, and that that can ramp up cost fairly quickly. K. We're heading over to limitations. So can we set limitations of usage, I think it's, of the model, per user per model? Limitations of usage per model per user. So currently, you can set what we call SKU specific, limits, but those are set at, the cost center and the enterprise level. So currently, we cannot set a usage for a specific SKU, so a specific model, by, by user. So we would have to limit it by currently either, limiting the models they have access to. Mhmm. Hey. This group particularly does not need access to model x or y, or, limiting their total spend. Now if they were to use more expensive, less efficient models, they will hit that, potentially hit that budget, quicker. The, there there has been a a requirement lodged around per user per model, limits. That's not currently on on the road map. That doesn't mean it won't appear on the road map, and and be implemented. Mhmm. K. Okay. Lots of good questions coming in. Thank you so much. Keep them coming, please. The next question is, tracking and separation of billing. So how is GitHub planning on to handle tracking and separation of billing for a agentic workflow based on consumption of AI versus interactive user based consumption? Given that some agentic workflows can be set up to run on schedules, will those schedule based usage able be able to be billed with predictable batch based processing? Yeah. It's a good question. We are seeing more and more use of the Copilot SDK, to enable, some of these, like, async processes. Currently, Copilot SDK still requires a specific user account to be tied to it. So you would see that the SDK was used, but it would tie to a specific user. So that that's how the data comes over now. There are certain, agentic features, specifically CCR, so Copilot Code Review, which can be done at, that can be a bill to an org specific budget, that will continue and we're looking at providing, more agentic features that can be built, toward, an org or a call center. But as of right now, usage, would still, while it would show up it as as, as SDK, as the, so additional piece of the the data that comes over at the end level is the kind of, the point of interface. So whether they used CLI, whether they used DID, whether they used Web GUI, whether they used Copilot SDK. So that that data is there. But it probably, at least in its current form, would, take some amount of logic to tie back, to, especially if the user was using SDK for something other than that. Now you could create a account only for that. Most accounts, are are starting to move into Ophirgen anyways, and so, it you know, there wouldn't be, the the as the as the entitlement would move into the pool, you wouldn't you wouldn't necessarily have wasted, wasted spend, but it would depend on on the scenario. K. The next question, our customers would like an easy way to gather billing metrics based on the new AI credit model. Will insights or other metrics enable them to see AI credit breakdown? So, basically, they want an easy understanding where costs are coming from and how to control them. Yeah. So as mentioned, the the data they will have access to via Copilot metrics API, as well as the CSV exports will be, request, at the request level. So they'll see be able to see the user, what the interface was, the number of tokens in, out, the number of, cache, cache recash write in in the anthropos case. So all that all that data will will, come across as well as, like I said, the surface that was used. What currently is, as I mentioned before, is, any, like, intent based, data that that would require, exposing the actual, protect prompt or or or actions. So that that doesn't currently exist. One thing that we are working on internally, and which will become available to users in the future, is enhanced, enhanced metrics that'll come to the metrics API as well as to, to dashboards that also allow, data around code attribution, and kind of value and velocity. So being able to tie, specific code, creation, both lines of code, as well as, code reductions, back to, repos and commits, and then even to be able to track that back to understand, you know, did those commits result in, you know, code quality issues, was the code better, was it worse, were there more vulnerabilities, did it end up being did that, code end up being committed, what was the timeline on on the commit, sorry. Not the commit, but on the, on the PR and the and the PR merge, you know, did that merge get back out to that level of data so you can not only see what you're spending, but now tie that spend directly to, to Impact. Okay. One more question, I think, before we can go back to the pre the resubmitted list. Is there a way to see a list of cost per model now not based on usage? There's a request for the full metrics price list. I can see calls for about yes. So the again, the that line level detail will show, you know, how how many tokens, which model, and and cost. And, that that obviously gives you the access to the data to say, okay. Well, what is what is a call center or what is this group of users or what is an entire enterprise? Do we spend on, you know, Opus? Do we spend on GPD five five? What do we spend on Croc one fast or or Mhmm. whatever you're looking at, you'll be able to see that line of detail as well as, again, that additional detail, like, well, was that majority of that was read? Was that output? You know, was that, was that was how are we doing as far as taking advantage of cash, to help reduce spend, that data will all be there. As far as the price list, on a from a model standpoint, I think I've seen that question kind of pop up a couple times, we are using standard MSRP, Frontier Lab API rates, there's no extra load or anything like that. That will be, on our docs page, it will be maintained. So, if there is a price increase, then because our prices go up, they'll they'll, reflect in that model. If there is a price decrease, that will go down. That will include promos, you know, so oftentimes Frontier Labs offer promos, you know, Opus four service releasing. We're gonna half the price of Opus four service for the first two weeks. That that will be passed on as well. K. K. If you wanna, I think some of the questions that we have presubmitted actually were similar to the ones that we already asked. If there's another one or two while we wait for a few more questions to come in from the group here, we. can do that. Yep. So let's see. Spanish hold on. I'm working through here. Yeah. So, has GitHub conducted an analysis or forecast on Copilot adoption patterns, specifically identifying which customer segments are likely to increase their usage under the new billing model and which segments may reduce or limit usage. So yeah. We we, you know, we we, look at this, almost continually, both from a, kind of customer, kind of a notional standpoint as well as from a, from a research and statistics standpoint. And, the model itself will obviously put significant pressure on customers as they see increased, you know, AI spend, globally. Our model aligns to effectively all the other models that, you know, the the idea of a request based model, it doesn't make sense, it doesn't tie to the reality of the underlying inference costs, and so everyone has kinda moved to this, token based, model. So I think what this is leading and what we're seeing is more price sensitivity on model selection, on context management, so efficiency, and optimization, usage. At the same time, the actual usage were were, still seeing significant increases because the, the capability of the models, how the tools, what tools what work the tools can do, and the velocity that increases, and as well as the maturity of the developers themselves. I think that's the biggest piece is on large large large enterprises, so enterprises with tens of thousands, or more developers or users. Oftentimes, the maturity curve is is still very, is very new. So what we have is we have a significant number of power a small number of power users, usually like I said five to 10% depending on the the group with that increasing kind of month over month and as that, maturity level as they get more comfortable with tools and and they, work those kind of, built skills around specific use cases and automations, the the usage is is naturally increasing. With the increase in prices, and the increase in spend, the expectation is that, you know, more aggressive, budgeting, will occur, and some of that, some of that like, we're seeing a 100% month over month growth on on just from a usage standpoint. So not from a necessarily from a seed standpoint, but just from a usage. Number of PRUs, amount of tokens, those are all skyrocketing. That will continue. My expectation is that some of it will, will abate as we as as people get the bills for it. Yep. On the billing calculator availability, I I I think I already covered that. The expectation was to ship this week, and and so I then that would go out, as kind of a a a broad, a broad announcement for GitHub when that's available. Exactly. Yep. Okay. Let's switch gears back to some, other questions that were just submitted as well. So one of the pathways for customers is to use the auto mode from model selection. How can this function work, and how can customers and us, the partners, evaluate, the quality of GitHub model router for cost and KPIs. Yeah. So prior to June 1, a new version of model router, or auto router, will, release. It's called I don't know I don't know if we have an official name for it, but, effectively, it will be intent driven as it pays to the way that the model router currently works. And so when a, when a request is submitted, the the intent based router will be able to determine, you know, kind of effectively based on a complexity analysis, whether that should be routed, to, you know, a larger, oftentimes more expensive, reasoning, model or a less expensive non non reasoning model. The idea is to keep this fairly simple, and driven by effectively economics, with with the idea of kind of defaulting toward, quality, if if there's a if there's a decision point. So if you're sitting if the model rider looks and says, hey. This is, you know, this is somewhere right about in the middle, it is going to default to the, the the more capable model. That is by design. It's not a it's not a cost driver piece here. That is a focus on, the cost is cheaper if it does it once rather than having to be, reprompted and more tokens being added to the initial request. That, right now, auto mode has a 10% discount, attached to it off, and it's 10% off the number of PRUs. We are evaluating that. Ideally, that number will go up. There's no expectation it'll it'll go down, by the way. We're evaluating it for increasing the amount of discount. But even if there was no discount, which there's not going there would I'm sorry. Which there will be a discount. Sorry. I don't wanna rumor get started that we're eliminating, that, but there's still value in the fact that, a user can kind of avoid the cognitive load of having to sit there and, like, what what model am I gonna use? Also that will continue to work within the confines of approved models. And so what I mean by that is if you have blocked, certain models, those will still be blocked from automotive. We'll not be able to select them. Currently, the, so Opus, the Opus family is not included in auto mode. The expectation is that will will continue, but as the maturity of, auto mode, continues, that'll be reevaluated to determine, hey. Is are we at the point where, like, it's making this decision, only when it's necessary to use these models that are, that that are on the most expensive, side of things? Great. So next question. Will the additional interface detail include identification of Copilot requests initiated from PR comments in other channels that trigger Copilot cloud agents other than PR reviews. In the metrics data, currently, you would see the, so if you had a a CCR, it would show you it was done by Cobalt code review. That will be included. If it is if you comment in the CCR review to, to to revise, that will continue to show up as CCR. If you comment to assign a task in, in a in a in a PR that Cobot code review so not a Copilot code review, finding, would be that would show up as, if you're assigning a task, it would show up as CCA. So Cobot, coding agent. Mhmm. So that's not true. But it would not show up as this was done as part of PR comment. They're they're not currently, that level of I can't say the word. Specificity. Yeah. Right. Yeah. Yeah. Got it. Got it. I I'm gonna fix that because that is the the question, is appreciated. I'll take that down and add that as a piece of customer feedback, Yeah. because I I don't know if that that, that specific scenario has has really been talked a lot about. And that's a that's a really good point you bring up, Robert. Please keep the questions coming, but also keep feedback coming. If there's something that you already heard from customers that we should know about, we'd love to hear those. We got some feedback actually last night, with our APAC session. So please continue to submit that as well to take under consideration. This other question is not overspend. I think you touched on it a little bit. But the question is, will there be a forecast notification or any kind of warning for the users, for the orgs, or for the enterprise about potentially overspend on their budget? Yeah. So as they approach, the, the the budgets, the approved budgets, they will get 50 per sorry. Notifications that so this is the enterprise admins. The enterprise admins, will and or call center admins, and and call center sake will get notifications at 75% of budget, burned, 90%, and a 100%. And individual users will get the same based on their user level budget. So if a user has a budget of a $100, they would get a notification at $75 or sorry. That'd be 7,500 AI credits, you know, at 9,000 AI credits. And then they would obviously get another notification if they didn't become very aware, had that they'd hit, hit their hit their limit. So those those notifications will, will be, pop ups and banners, within the the website as well as well and the various, ID, or no. Sorry. The the coding services, sort of like CLI, ID, ID of choice, or the web, as well as, email notifications, if configured as well. Okay. Yeah. Oh, I there's also there's also notifications, and these are these are configurable because this can be mildly overwhelming depending on the size. The the admins can also get notified when they and I don't know if that these are set by default or not. I'll see if I can track this down before we do the call, but, and they won't get notifications when individual users hit their, enter budget. So they can look and say, hey. Do I need to unblock this user? Or, you know, is there is there no more budget? That that, you know, that will potentially be a a a reality as well. Okay. K. Great. So this, question it's interesting question. It's about Azure. So we're a Microsoft partner as well as GitHub partner. As a Microsoft partner, we get Copilot benefit via Azure credits. Recently, we found that we cannot use the subscription we provision for payments. Is this intentional? Is there a reason why Azure subscriptions are filtered? And then why does GitHub care what Azure subscriptions I use to pay for Copilot? I can't answer that question. The Azure subscriptions are the primary method with which people. buy Copilot. yeah. Get a Copilot. The Nuance on individual subscription or a partner subscription, and that that that's some ideally, someone else maybe on the call that's, on the partners team can answer that, but but I do not know the answer to that. Sorry. Yeah. Margarita, actually, if you wanted to come on stage, I think you can answer that question. There you are. Let me see if I can do this right. Okay. Hi, everyone. I can answer that question. So, there's been some changes in terms of, trials, like free trials with Copilot that have, you know, happened internally. And, the reason why is because there was abuse of those trials, and so we needed to control a little bit how they were set. So right now, the type of, Azure subscriptions that you can configure for those trials have completely been redefined and they're still being redefined as we navigate this whole new UBB release. The expectation is as soon as UBB release, goes GA, then everything should go back to normal. But until then, we can only, leverage, like, production Azure subscription. So that implies that no benefits that you're getting from Microsoft are actually, like, good subscriptions to be used. That's still to we define the actual date when everything will go back to to normal, but it's all linked to this whole GBV release. Thank you. Thanks, Margaret. Alright. Will so next question. Will we see a general offering of keeping JHCP data within a region without GHDR and not tied to a specific model? Let me read that. Is that on the oh, you're right. I wanna see a general offer. I have to be I don't have to reach my mark. Oh, so, so, so the the question is effectively, like, can we limit get a Copilot's data, of where we save it. You know, this is very tied to mostly, obviously, we're talking to EMEA, so this is most relevant within, within, Europe. But the idea is, like, can we use a specific, regional dataset, instead of without using, the GitHub product that's data that's data that has data residency could get a data residency? The answer is no. So currently, if you want to use GitHub and tie it directly to, you know, just Europe or, just US or or another region where it's available, you would have to have, GitHub, with data residency, in the in the region that you are in, and, and have a Copilot subscription, which is tied to GitHub data residency. So those those are available, but they are specific to GHDR. Okay. We have two pieces of feedback that have come in, so I wanna make sure that we capture them. So the first piece of feedback is, so they, the customer, has difficulties in budgeting in the first place. The customer needs business approvals, in the first time to set the budget, and they do not have the data and realistic estimates to even ask for the realistic budget. Yeah. This is this is a question that comes up a lot. Right? And this is doubled by the fact that this, you know, admittedly, is a fairly short window from the time of announced till the time of effectiveness. So this is putting a lot of a lot of a lot of customers in in in a tough spot. And this is doubled down by the velocity with which the AI space is moving. There is not a universally predictable, like, kind of acceleration path from a cost standpoint. The best we have right now is looking at your April data, and showing you how that maps to the new. But as far as what that looks like six months from now, that's really very account specific, you know, plans on how they're planning to roll out the tool, what potential budget controls they wanna put in place, the maturity of their existing offering. We have, you know, some customers that are very, very mature. They have a high level usage across, their developers. They're using our our newest tools and their, you know, their their acceleration, from a usage standpoint has has kinda slowed. And then we have other entities that are you know, they have that model where, you know, one to 5% of their users are the primary consumers, of their AI tools, and each month they're adding, you know, another, you know, percent or five to that, you know, that active user, that power user set. And so their bill is accelerating much more quickly. I wish we had better guidance, around that, but right now, you know, the the best data we have is to be able to or best data we can provide is to be able to look at historicals, and then for an individual account to to say, okay. Well, where do we think we're going? Definitely lean on your, GitHub account team as well as your Microsoft account team. They can potentially provide, some insight on what we're seeing, you know, more broadly across either, you know, the the kind of, the the space in general or the specific, industry that that customer is in. K. Okay. And then another piece of feedback, which I think is interesting, so it is about cost centers. So we need to be able to change license sources or the orgs mid month because people change cost centers all the time, not just at the end of the month. The current cost center model doesn't work for this purpose since we need the org boundary for licenses due to model constraints and other settings that are organizationally based. Yeah. I would I would love it if we had more fluidity across call centers, but especially at a at a, non unlimited model. So a model that ties, you know, users to some amount of usage, Mhmm. the it it breaks the model to take someone that maybe has, you know, exceeded or or or, you know, burned a significant amount of let's say a cost in our budget and then move them to another mid mid month and then try to reassign what what the costs look like, especially when we talk about also just the base subscription fee. So that's not something we, we we currently offer. I know that that feedback has been given a number of times, and we're trying to figure out what the best way, to map that is, so that it's kind of, fair between the cost centers. But right right now, that's, that's not. on the road map. So another question here, regarding auto. Forgive me if I missed it, but when can we expect it to select the best model for the tasks? The docs say coming soon. Yeah. I'm gonna, this is recorded, I think. I'm gonna go out on a limb and and throw throw a date out there, but this is what I I'm gonna we have this a bunch. This. is the target, so this this could slip. And I say date, and I've decided I'm not gonna say it exact date. But I'll tell you right now that it's mid month, so, which is, you know, couple. couple a week or so away is where we have mid month. So this is this is, approaching, quickly. I've it's staff shipped, so it's it's shipped internal, to GitHub, as well as Microsoft, I believe. So it's testing for quite a while. I I don't see, an expectation this would slip, but, you know, things happen. But, yeah, that that that will the expectation is that will ship before, UVB becomes, effective. Mhmm. K. Great. At this so we have eight minutes left, so please feel free to, put some more questions in. At this point, I believe we're all caught up with the questions that have been submitted. So if there's any others, Robert, from the presubmission list that we can go back to. Let me see. I think we we covered, you know, a lot of, the the questions in the in the chat, covered those. Let me there was one more. Let me see if we're well, let me let me hit this just one more time. I think I've answered this kind of question a a bunch of times, but this kinda takes slightly different take on it. But is GitHub planning to introduce a high granularity API for coupon usage reporting, maybe access to raw data endpoint, especially input cache output tokens to enable customers to analyze, optimize their usage under the new billing model. So as I mentioned before, the the, the very first thing that you'll have available to you, is the kind of historical data, so being able to look back at, April. With, when UBB becomes effective, the Copilot, metrics API, will include all that data plus, plus more. So and and that API, we're actually shipping basically weekly enhancements to that. A lot of those enhancements, so obviously, all the other things that were mentioned, input cache, output tokens, you know, who requested it, that obviously, model, that's all bare minimum, that that's, there. What the enhancements I'm talking about are are, tying that more specifically to, to, like, code attribution, and some of these other features that that will allow you to do metrics not just on kind of optimization, but also on measuring, impact or value to to justify that that those budget requests. Right? If you are seeing increased usage, we see a lot of times now where, where leadership is saying, okay. Hey. Like, I now you know, two years ago, it was it was a lot of, what do they call it? YOLO, where, like, I had to or FOMO, I guess. FOMO, where I just had to, like, I had to have AI, and there was a lot of enterprise spending, with without any real, ROI analysis. And now what we're seeing is is there's a lot of, expectation from, decision makers to say okay well if I'm spending this much what what's the value I'm getting out of it? So tying more and more the Copilot Metrics API to be able to answer those type of questions. That that will be let me let me nuance that a bit. That would require, for for some of these things, for overall attribution, it would require that data being stored in GitHub. So if you are a a user that their their data is being stored in, one of our competitors, obviously, we we don't have that data, because it's not being submitted to us. K. Another question came in. Is there a change log somewhere regarding the new metrics in the API? Yeah. The, I have the the Copilot, change log includes those when they ship. And so a number of those, especially around tokens, haven't haven't shipped yet. So, but it will be in the, it will be in the, the changelog. And I think there actually is a Copilot metrics specific changelog, but I have to check on that. Okay. Well, while you check, I'm gonna make a last call for questions. We have about four minutes left. If there are any other questions, last minute feedback, please keep them coming. While some hopefully, some more questions come in here, I just wanna make sure, that everyone on the call here, has been to our usage based billing page in our partner portal. So there, we're trying to do our best to post as many assets and pieces of content as we as we possibly can for our partners to consume. So thus far, we have a partner guide. It has some FAQs in it. We have a customer conversation guide there, partner offering kit. Earlier this week, we did post a customer workshop for understanding budgets with actually some really good, speaking notes in that as well. So please take a look at that. We have various call recordings in there as well, that we've done under over the past few weeks. And we had an office hour last night, for our APAC audience. We're gonna post that recording in there as well. So if you have the time, take a look at those, recordings. Maybe there's a few other questions that we didn't get to or that weren't asked that would be pertinent to you. And we'll post here the link to to the partner portal, for easy access as well. The answer is it it's just a part of the it's a part of the GitHub, change log. So you can track the GitHub change log. Great. That's where they drop it. Perfect. And then any any changes that have happened, they've also updated the, the API documentation. Perfect. Okay. Alright. So while we wait for we have three minutes left. We'll take a second here to wait for any other last minute questions. And I will point out too, if other questions come up between now and June 1 or even after June 1, please do get in touch with us. We'll post a link here, for consistent feedback, because we know some of the conversations here are very difficult to have. Scenarios come up that we didn't answer. So definitely open line of communication, to get back to you all, with any additional thoughts or questions. K. Wait another minute here for anything else. And thank you, Camelo. That's the link here to our usage based billing page. It's on the partner portal, so please do check it out if you haven't yet. Yes. This is being recorded. We're gonna post the recording of the transcript in the partner portal. You're welcome for using the portal to to update you. Okay. Someone's looking forward to the bill calculator. I think we all are, so that's great. Alright. I think we're good. And with one minute left, I think we're gonna close it out. Thank you so much for coming, for asking such great questions. Again, I'll reiterate, open lines of communication. Any other questions, comments, thoughts that you have, please keep them coming, and our team will be happy to, to respond. Thank you again. Hope everyone has an amazing rest of the day.