Name: Strategic AI deployment: Secure, scalable, and ROI-driven
Uploaded: 2025-08-07T10:30:53.206Z
Duration: 58 min 40 s
Description: Strategic AI deployment: Secure, scalable, and ROI-driven

Transcript for "Strategic AI deployment: Secure, scalable, and ROI-driven": Hello. Good morning, everybody, and thank you for joining us for today's live session. Delighted to be, joined by you all. Thank you for making the the time today to join for session that we're really excited for, all around strategic AI deployment, secure, scalable, and ROI driven. We have a seriously all star cast today. You're gonna meet the palace in a second. Glean, AWS, and Anthropic and their senior leadership teams joining for the discussion, all focused around a few key talking points. So Genetec AI and new interfaces, AI security and ethics, scaling AI and ROI, and the number of AI applications that are gonna get covered in the discussion. The talk today is about forty five to fifty minutes. After that, I'm joined by Alex Jones, head of EMEA Partnerships go to market, live from London. He's gonna jump on for a live q and a session. So please do drop your questions in the chat on the right hand side. Alex and I are gonna get to those at the end of the talk in about forty five minutes. But for now, over to the panelists. Thanks for joining. And, yeah, do let us know where you're tuning in from, and drop your questions in the chat. We'll hear from you soon. Alright. Thank you all so much for coming. Welcome to a taste of innovation. First of all, isn't this an awesome space? Makes so much to AWS for, that does come in here and to send AI law for such a cool opportunity. And, today's gonna be super casual. So we're just gonna have conversation about AI, talk a little bit about, what each company is doing, how we're working together. I think our endeavors to go half the time talking and half the time answering questions from the group. So if you have a burning question, please, feel free to ask it as we get into that. Let's make it super accurate. Brian is back exploring the CMO of Fleet. And I'm just gonna quickly introduce our our awesome panelists today so we dive right into the content. So why don't we start in the middle with Adam? So Adam Salzman is our host, so only collaborate with him first. And he is the developer experience at AWS and has had just incredible career, the developer evangelist programs connecting with the developer of The East for AWS. And before that, Google, Salesforce, Microsoft, if you've seen the inside of about every cloud, it's in clip. Yeah. Yeah. It was before you called the cloud in some cases. Thanks everybody for coming. I guess I'm one of the exec sponsors for the our global set of, GenAI ops at AWS. I think this is probably the the anchor. We've been, we've had the lot for years. We really fired up the plans five, six months. Been doing almost one event a day. We really wanna make this the center of generative AI innovation in San Francisco. Our great partners like Glean, Anthropic. Glean is a little further away than Anthropic is, like, practically next door. So we'd love bringing the startup together the startup community together, our customers together, the developer ecosystem together. And, so excited to have you here and and looking forward to to this conversation. Awesome. Alright. It's my boss. It's Michael Gerstenhopper. He's VP of product at PropTech. And before that, it was a long time that product had dated us. Thanks for joining us. Absolutely. Thanks family, everybody. I'm excited. Yeah. And over in the end is Tamari Hoshua. So she is the president of products of technology, I believe, at our engineering and product team. And for that, a similar role at Slack, was at Google before that, and the investor in between there for that. Yep. Really great to be here, and you missed that before I was at Amazon. Oh, I know. Yes. Alright. Well, Tamara, why don't you cut this off, and explain for those that don't know a little bit on, Google leaders and why the three of us are up here. Yeah. So Killeen is a work here platform. We index all the content of your organization, make it searchable and enable a chat like interface. So you can ask questions about anything in your organization. And it's also a platform that you can build AI apps, prompts and apps on top of the platform so that you have essentially a knowledge graph of your organization. So why are we here together? We have a close partnership with both AWS and Anthropic. When you are taking the content of your organization, you wanna make sure that it is very secure, and you can host it in your environment, in your AWS environment. And we also work closely with Anthropic that you can use Anthropic as your reasoning engine, hosted on Bedrock, again, to keep your data secure. We think that this combination of the three is really the most secure enterprise solution for companies. So let's dive in to a couple of topics. So we thought we'd start with just the evolution for Gentek kind of properties within AI tools. At Glean we started as very much enterprise search company and a lot of knowledge management tasks. It's a lot of people that chat with their enterprise data. And as they move along, kind of the journey, they start to use AI for more and more tasks. And one of the exciting things is that kind of bridge to not just Q and A and kind of knowledge discovery, but starting to graph content, starting to even do basic automations in their daily lives and roles. And so, why don't we start with you Michael. In Anthropic world, Claude is just excels at resync and it's an area of a lot of evolution. So maybe give us a little bit of kind of how things work, where you see things go? Yes, absolutely. Thanks for the question. We we came out with a a model well, so first of all, Anthropic is an AI research lab. We produce, models and and starting with, CLOB three back in March, we we produced a a model family that has three different sizes on the same underlying substrate and with a lot of the same properties so that you could use prompts, in equal measure on each. We're also very, very focused on on safety, which is why the relationship here is quite so important by running in Amazon Bedrock. You do keep your knowledge in in the perimeter and your your completions within the perimeter. Nothing leaves the AWS ecosystem, and we don't see any of those prompting completions. I wanna talk a little bit, I guess, about agentic behavior and how it's evolved even over these eight months since since cloud three. Right? At the beginning, everybody was already talking about it. There was assistance APIs and and bedrock agents, and there are various ways to construct, like, application layer agents. And certainly, industry was trying to build their own as well. A lot of customers in, in finance were trying to do very complex multi step resuming tasks. My old company or my old company. The the company I used to work for, Datadog, was was trying to solve, the the question of whether or not an event is an incident in DevOps, things like that. But they were doing it themselves and working around the model intelligence at the time. Right? You would use Haiku to go and and collect data from a lot of places in parallel very, very quickly and then pass it to Opus for reasoning. You'd have to orchestrate a lot of things. Agentic behavior looked a lot more like, prompt chaining or or just application development, honestly. More and more, today, we're seeing the reasoning capabilities of the models, mature. Right? In our most recent Sonic 3.5 version two launch, as recently as a week and a half ago a week ago at time moves, funding this industry, we were able to launch, what we call computer use. Computer use is where the model, can decide can can get a virtual machine and decide when to take a screenshot by itself. And so when given a description of a task and the canonical one we used at work was order me a pizza. Right? It can bring up a browser and produce x and y coordinates in the output to to to to simulate mouse button clicks. It can bring up a browser, go to Papa John's, order two small pizzas. So it still does make mistakes because it doesn't know nobody orders small pizzas. But, it was able to enter real credit card credentials and checkout all in one work stream. Right? And so as the model gets more and more and more intelligent, we're gonna see behavior like that generalized back to enterprise tasks and and, and and and processes that we all do every day. And it's gonna become more and more relevant, I think. We were just talking about how it's hard to explain. I like how you have everything from x y coordinates and token and tokens and reasoning to order of popcharts. I do my best. We ate the pizza, a real pizza came. It was great. So Adam, one of the nice things from your vantage point is you see people go in all sorts of stuff on Amazon in the ad space. So are there interesting use cases you've seen with people kind of pushing for genetic behavior that you think are interesting? Yeah. The thing we see over and over again is, to get real value, they our customers don't just wanna demo. They wanna see do real work, go like a pop set, save somebody time, produce better output, you know, improve coding tasks, like, with our our queue developer assistant product. And, real work takes, you know, that great model capability and it also needs enterprise data. Glean is a great way to to get that. Like, you know, you you don't just draft an email for a sales call or a pitch deck or a first call deck or something. It's context of your company's products and your company's messaging and all this to get put together and synthesize the like, you know, you get a synthesized account plan. You need to know a lot about the account that lives in your corporate system. So, your AWS data sources, like our, data stores you keep your data in, all the sources that glean get pulled together and search for are are super are super useful for that. So I think we're kinda learning, like, a lesson we've heard a long time ago. It's just like to get value out of the cloud. You, like, you need to, like, really get your business in it and run it. I think the same thing is true with generative AI. If you really wanna get value out of it, you gotta, like, connect it to things and give it the benefit of your corporate data. Guys still follow the same rules also. Hopefully, we'll talk about that in terms of security and stuff. We'll come to that. But, like, those foundational lessons, I think, are still there. Like, real return requires real context and to do real work and that lives in your real business with real data and information for around your airports. Yep. For it to be useful at work, it has to understand the data, people of work. Cool. So one of the examples you brought up was, a sales rep maybe prepping for media and getting context. That's an interesting use case to set it off on our platform for sure. As you think how often you have to pull together all these different data sources and then get a single picture and then walk in the beam and do repair, just huge huge use case there. So tomorrow, one of the things that that we've launched fairly recently was a capability called advanced prompts and just kind of stepping into more and more automation and multitask automation. But you can talk a little bit about that, Jerry. Yeah. The jury of Agen to growth was really interesting as Michael was saying that at first, people were talking about it a lot on Twitter, but nothing actually worked. And now you see that there's certainly to actually work. And, thanks to a lot to the capabilities of models. And so when we started testing out agentic behavior at Glean, we started with simple prompts where we just have a prompt, save it, be able to share it with the organization, and then we can move to more advanced prompts that we've been testing in beta with our customers. And this is multiple cost to the LOI to tap into the reasoning capabilities so that you can do much more advanced things. And you need still a level of expertise on how to craft your prompt exactly then. So I think there's, like, a small percentage of the organization that really understands how to do it, but then once you create it, you can share it. So my go to prompts that when we were testing out this this feature is, what gives the latest status on a feature and its release? And so instead of just looking at our launch cal, which gives a day, it will look at launch cal, it will look at the Slack messages or Teams and say, what are people saying? Is an engineer sent that there's an issue that really hasn't been reflected anywhere else? What's in the documents? What what Jira tickets are there? Are there any Zendesk reports from some of our beta customers who can pull all of this together to give you an accurate status of the feature, including who is the DRI who you should contact. And so this is an example of a multi pronged, prompt that calls you out multiple times and could also take action because we have that now that's a priority to me, and objective behavior is also being able to take action on your behalf. And then that can be shared with the organization so everybody can use it. And this is one of our top use prompts right now within acclaimed inside the organization. So that's been really exciting to see once you give tools to people to use that they might not have been able to use. We have to do before that it unlocks so much for them. I agree. It's been super cool to see just everyday people take advantage of automation. You know, there used to be the RPA market. That was something that you had to go code. It was for automating, like, the most complex business processes. This is light for everyone. Right? And so just seeing people bring their creativity to, making their work more efficient. So Adam brought up a good point earlier just around that connection enterprise data and security, and as you start to bring more and more private data with these models, how you guarantee security. You know one of the things that that Amazon has been a leader around in Bedrock is just creating an environment that allows people to move fast in development and bring the models to bear, etcetera. Maybe getting us off on that topic a little bit around just building fast food security and that cash flow that rose data. Yeah, But everybody faces this challenge, right, how to go really, really fast, but then in this complex world, how do you keep your customer information safe, your business safe, protect your investors and your employees and everything. And so we provide that capability with the AWS cloud. And we want to be the most reliable, most secure, most performing, most features, run-in the most regions, just really get the capabilities our customers and our developers need just have a really great IT environment, just lets him go faster than ever before and do it safely. And And so we thought about generative AI, we decided our focus is gonna bring how we bring those same principles for generative AI. Inference is amazing and training is amazing, but this kind of compute and the rules of physics, the rules of security kind of don't change. You still need security. And, so we offer Bedrock. So our customers get access to the world's best models, like all the quad models are available in Bedrock. It's a fully managed serverless environment. So you can just instantly, generally instantly spin up and start consuming hit fricks, fine tuning even in in some in some cases. And, we provide additional services around it like guardrails. There's rack built in the ability to call things like Lambdas for tool use. But again, the principles are like the same principles as any other kind of compute. So, like, use I'm like fine grade permission and access controls in logging. And I just to to close that, I'd like to translate what it means for a business person is like, you you want to be able to run your generative AI workflows just like you run any other capability in your company where the compute is there, you know, our IT staff, your IT staff can manage it. All the logging is there. You know where your data is. Your data is not getting sent all over the Internet to some provider where you don't know what they're we're doing it. It's keeping it inside the AWS envelope. And, we're finding our customers really like that. Rigid industries customers really like that. It's allowing them to take advantage of generative AI faster than they would be able to. I don't trust all the new little startups in the space, candidly. I don't know. It seems like a pretty dynamic space. And, you know, if those companies might make decisions, it might not be the way, you know, large customer would really trust them to make decisions. So So we're trying to bring our partners together on Bedrock, anthropically, are both there inside AWS. So our customers have really trusted secure environment to push forward with this generative AI workflows. I mean, we think about safety all the time. But when we think about safety, we think about all of our customers. Right? Which is more broad than is useful for each individual. Each individual has their own compliance regimes. Safety means something different for each enterprise and their brand and and that sort of thing. So, so that's where, like, on one hand, we can we can refuse, prompts that that ask us to do, like, genuinely harmful things. But then then we do rely on AWS to keep prompts with with an AWS. So we can easily say that we've never trained on customer data. But, again, with AWS, we don't even see the prompts at completions. And then we also delegate a lot of the permissioning and and the guardrails to the client developer also. And that's where Glean comes in. They have access to your data. They have access to your data on AWS or across the Internet on SaaS tools. And they can manage permissions and and the data itself gets permission. The service account gets an identity. Right? And so we don't train on data. But then at the application layer, there are also a lot of tools like I'm which is underappreciated in this, in in protecting your data. Right? And these are tools that have always existed, on any any sorts of data, and that's no different in this in this world. So we go to market for fintech. We we we go to market for for health care. We sell to the to the, DOD and intelligence communities. We get there because of the relationship with Amazon and their knowledge about their customers and their knowledge of those compliance regimes in order to keep that data safe and protected. And maybe tomorrow give us a perspective of that as we bring these tools and this in the largest organizations. You've seen this in your career a few times bringing new tools like Slack, lately and into organizations that may be more used to them from a security perspective. How does this kinda work with the base companies of getting them over the number of little trust in it? It's been a very interesting experience. So I was talking to, an engineering leader who I worked with before. I was trying to recruit, took leave, and he's like, how the hell could get past CISOs? You're like, adjusting all the content of everyone's organization. And he was just so amazed. And I because from day one, we took this really seriously. And from the beginning of Glean, even before GenAI, when Glean started, it it was enterprise search, and it was indexing all the content of your organization and making it searchable. And so from the beginning, the cofounder said, if we don't get security and permissions, right, this is never good work. So that was since the beginning of the top priority. It's how you understand the permissions accurately. How do you understand the SaaS tools and which what the permission is for each tool? Because the offering product has a different permissioning model. And so you have to understand those so that you never give information to somebody who shouldn't have it, and you have to keep it really secure. And this is why we said from the beginning, we're gonna be single tenant, and we are gonna allow people to host in the ring back there because you're not gonna feel comfortable giving all of the content to us in a multitenant SaaS environment. Now there are pro a lot of products out there who are multitenant SaaS, but the sensitivity of the data isn't as high. And so this is how we get CSETs comfortable is because we show them that they have control. And with, again, AWS and better app, it is secure. So exact combination of the security that they trust through AWS, the, security and permission that they trust through Anthropic. And then for us, we're never gonna show you a document that you don't have access to. So if you, if you try and gain access to ask a question about a document you don't have access to, you're not gonna get an answer. Yeah. It's it's obviously just a huge education for the largest customers out there. And I think, you know, people not only underestimate the challenges of that, but also just scale. You know, we're at a point now where the largest wing customers have over a billion documents index. And not only is that enormous, there's the real time aspects. People send a message, they wanna be able to see that Slack message they sent a minute ago as fast as possible. Right? So just the scale of that becomes very interesting. So you know, Michael brought up something earlier with, you know, just the computer use. I think another cool topic or exciting evolution right now is just the ways people are active these tools. And, you know, if you rewind, you know, two years ago almost no chat to be moment. You know, like, the channel interface really opened up people's use of this and kind of democratized use of these tools profit. But now in a lot of ways I think we started to feel like just being able to only have that channel interface is limiting the text interactions that are possible. And so Quas has done some amazing stuff in terms of new ways in our back with our backs and that could ease. Maybe we can dive into that a little bit and how do you think that's going to change over time? I mean, I definitely think a lot of this is going to become more and more and more proactive. We definitely see it with with things like workflows. Right? One one person at a company can implement a workflow. Many, many people can benefit with, from it. And the model case is that information comes to that person when they need it. They are helped in following enterprise processes. They don't have to initiate the process in order to do something that the computer could be doing for them when they need it. So I'm very excited about things like, like artifacts, which is a a a document builder that allows you to see the results of of of your authorship, of your code, whatever it is, and then iterate it with it in line. It's still fundamentally a a chat interface, so it's interactive. I am more and more and more excited about the possibility of of predicting what you are trying to accomplish and helping to accomplish it or even preventing you from having to to do it to begin with. And I think we're we're getting there with long running agentic processes, rather than jobs that just, like, run to production based on one input. That's especially true with having more and more access to data. Right? Like, you can't help somebody without knowing what they are trying to do, the context of who they are in the organization, what they're allowed to do, things like that. And and and and for that, right now, it's it's complex prompting and rag. And I think it it requires a lot of engineering and and more and more as context windows grow, as as as recall stays high accuracy, as rag techniques, mature and other retrieval techniques mature, it's just gonna get more powerful. So, we're very excited about all the integrations that are possible through through workflows like like with Glean. Yeah. Yeah. I think you're not a key element there which is also just getting people down maturity curve. And it's it's been interesting to see how, in one way the the search chat interface is great because everybody else had to do it. You know, they they were probably the search twenty three years ago. But it pretty quickly becomes limited in discovery and the advanced capabilities of the tools. Some more people work on that lot in terms of new ways to interface and and kinda show people promising these in their daily lives. Maybe expand on that a little bit. One of the questions I get asked a lot, are you a horizontal tool? Are you a vertical tools? And they say both because you have to be both. So we're a horizontal tool. You can ask anything. You can go to the search interface. You can go to the chat interface. You can ask any question, and you're gonna get an answer that's relevant for you personally to Michael's point. We understand who you are, get a chance to who you report to, who reports to you, so we can get the right documents and give the right answers based on that information. But we also need to understand your role. If you're an engineer or marketing or sales engineering, we have to understand your role and what you're trying to accomplish. So we've started, one, having more vertical applications like we launched recently something called lean assist, which is agenda major for a, customer support agent that is customized for customer support, understanding those workflows, understanding what you need to answer or take and then take action. And then being able to if you're, in marketing and you want to create a block for this, you don't have to, like, write a first draft. You can say, what are all the documents that product handling engineering have written on this feature? If you don't think that's fine, then you're just saying, what are they? Give me the first draft. We did that recently, and it's and it's really interesting what it comes up with is that it gets you just so much faster to what you need to do. So what we did is we built a prop library that's available on our website, a generic one, for any customer for every vertical. So any kind of line of business, you can have a prompt and prompt library. A new customer gets that prompt library for default, and then we work with the organizations to build specialized prompts based on your data sources for your company. And, again, now we share them. And if you're an engineer and you open up, you will see in your zero state prompts that other engineers in your team agrees. And that's a really powerful thing. Just like autocomplete in search tells you what other people are searching for, This says, oh, there's a there's a prompt for PR reviews to automate PR reviews. I can know that. I'm gonna try that out. And we launched that pretty recently, and we're seeing really nice pickup of that. If you think you have an intuition how prompting works and how to best cope with results out of the JERA AI tools, you're probably wrong. And don't trust your intuition because physics works really differently. I play an LLM, and, I got exposed to the anthropic LLM prompting guides really early. Totally surprising and different. Has some connection to how the models are trained. This is my breed. I'll I'll refer to the experts and, learned a ton and and and surprised that my a lot of intuition expect the same thing inside the Glean and the and the ability for your employees to find prompts that work and get best practice in Glean and then share them. It's gonna be super valuable. Like, it's a totally different kind of skill. It doesn't work like you might think it works or you won't get the best results without learning some of these new stuff. So definitely, like, take advantage of the resources that are out there. I was just gonna say my favorite is when I've heard that if you tell the LMM I'll give you a tip, it actually gets you a better answer. We got better JSON from an early cloud version if we said please. Yeah. There are actually, like, a lot of tools just like a lot of code now is written by LMS through through code gen. I think a lot of bumping is also going to be done in the very near term feature from generating prompts just like any other generating code. And so, like, hopefully, the skill level comes down in this sort of thing because you will be able to describe a task to to a a a prompt generator, and it will be able to structure the output of a prompt. You'll be able to see the best practices in that. There are a lot of guides. There are a lot of, like, meta prompt techniques in cookbooks and and and and in the industry, but I I I really do hope that that comes down as well. It's really good content on the anthropic website. They're prompting a really good idea. I think the other exciting thing is just to watch how people evolve, how they work with these tools. And I think often people come to it and they think, okay, I do a certain set of tasks, how can this be you up in that gap? And as they start to understand the value of AI and and kind of what enables, eventually you learn to work differently with it. There's things you can do that just more possible before, and so there's whole tasks and and ways that we're pushing it that are different. That's where the real power comes up. I wonder if, like, the most popular Glean shared problems inside a company will be the from the youngest least experienced. I I won't say laziest, but, like, most most excited for automation to to do the work for them, you know, like, no no preconceived metal model who has to do what and just are willing to just go all the way in and invent this this really and maybe have used it in higher ed or elementary or high school or whatever. But I have a feeling that, like, the the, the spark of creativity that gives us the next generation of really amazing generative AI powered apps and lead prompts and things may and Bedrock apps may may come from, like, our least experienced workers just don't carry any of the baggage how we all think I all as a as a as a grizzled veteran in this industry, oh, everything has to work this way, and I'm totally wrong. Absolutely. I mean, I I I'm the father of few high school students right now. It's just over the last year, it's been the shift from you shouldn't use these things to, okay, we should get you to use these things. And then the the whole mental model is starting so early. Alright, maybe one last question that we'll open up to questions for all. So, a couple of debate that we hear a lot about right now is around ROI. Right? Are the investments in these tools or am I getting return? I actually have the question. Is that even the right way to think about it right now? You know, is ROI what we should be measuring at this early stage? So just talk a little bit about how you see customers getting through results and how they should think about that framework for how to spend to get those results. Yes, I mean, I guess I ask this question all the time from customers. I am fairly comfortable with early investments that will pay off as long as people think they'll pay off. So anybody who started building a product knows that it it takes time to get that product to production, to start making money on that product in a very real way. So I'm I'm fairly comfortable when people tell me that they're experimenting with it. But it's very important for me, obviously, as the provider of the LMs that these do provide value for customers and that they're not just science experiments forever. And we are definitely seeing that. Like, all day long, definitely, in specific verticals, it's it's sort of an easy one to talk about is is customer support and customer experience in in legal research. Almost every large legal research company in the world and every small legal research company in the world and every legal research startup is starting to make money, from this. CodeGen is is a pretty big, ROI generator. So there are definitely verticals that are easy to talk about. And then everybody's experimenting, and I think we're gonna find more and more and more, more and more business application out of it. Most yeah. Anyway, I I I it's it's hard to get in specific, in detail about individual customers just because I don't know who I'm allowed to talk about exactly. But but we definitely see it moving from this stage of experimentation a year ago to development earlier this year in the summer into production and producing, more money than the tokens cost, certainly. But it's probably the biggest question I get when in customer calls with CIOs is how do you measure the value? And so we work with the customers and with their teams to measure the value. As as Michael said, on customer support, it's pretty obvious, like, okay. How long did it take to resolve a ticket? And you see ticket resolutions coming down significantly or deflection. Engineering, you look at what does it take to resolve an incident, what's the average time to resolving an incident. And then we also just look at subjective time. Say so we do a lot of surveying before and after. How long did it take you to get an answer to this? How long before, and after, and then how often do people come back and use the product because that's a good indicator that it's getting value. So we work closely with our customers to see how they wanna measure the the value and then make sure that our tools can help give them that data. Makes sense. And I I think one of the, most challenging areas to measure the value I found is with engineering teams. Yeah. Because oftentimes with sport or whether you look at call lag, then I'll and have you seen anything deep there about how people think about, engineering with these tools and the benefits there? Yeah. We use technology from Bedrock for a number of products, including Cloud inside a developer system tool called Amazon Q. A Q is like an expert to help developers do their jobs. And we started that product a few years ago, and the technology has gotten leaps and bounce better. Some of our engineers tried it early, and we're like, it's kind of useful, but not crazy useful. So I had a key a principal in here on my service team. It's like our highest bar of engineers that are, like, really go deep in technology, really deep dive and and have a, it's like neutral dispassion view on things. And a key in my service team is not willing basically to use generative AI to to help him do his job. But I finally convinced him to use Amazon Qube. It's powered by Quad. And he just came back and he said, I I did it, a hundred and sixty hour task in twenty hours. Like, just just like, this is my task. It my size is a hundred sixty hours. I did it in twenty. That winds up what we see. Yeah. Yeah. And it continues to get better. So this is something I didn't wanna do it, just candidly. But then it's now, like, quite an advocate for internally. It's helping people. And we're learning about prompting and everything. So, yeah, we're we're seeing real ROI with Bedrock customers. We're seeing real ROI with, Q and Glean customers. Customers like Intuit do, billions of tax transactions using Bedrock to flag them for humans, if they need to get escalated and saving, you know, millions of hours of labor. They they've talked about the numbers publicly. And they're in prod and they're prod on Bedrock, and they're they're using technology from Anthropic and others. And, so the the the ROI like, to understand ROI, you have to send the r. And as best we can tell, we don't understand the bounds in the r yet. We don't have to, like, land all the r and measure all the r yet, but we also don't see bounds on the r, the upper bounds in the r. Computer use means not only will the agent be looking up things in Glean and synthesizing, but also starting to take action, you know, in the in the future. It's early, I think, but in the future. And then action is human labor. So, like, we don't know what the upper bounds are. It's super exciting. And and I think that's why it's for you, dear, and that's why there's so much of the innovation happening in the in the category right now. Hi. I'm Adam. I'm a research scientist here in this building in Amazon ads. My question is about hallucinations. So these things aren't always perfect. Sometimes they return with what sounds right but isn't correct. How are you folks measuring that and start trying to minimize it, especially for, like, support use cases where the information is super important? Since we're a rack system, and then we use the LLM just to do the generated from the answer, but all of the content is coming from the content you're in a patient. It's all grounded. So you with billing, you see exactly which documents it came from in the answer. And so it's where we've been able to limit hallucination significantly by making sure that every answer is grounded. We actually use LLMs to validate and evaluate our LLMs, and groundedness is one of the top things that we look at and we track. And that's a common pattern for you too, right, because the resources are there. Someone might use the summary for the synthesized output then also click in and read the source concept. Yeah. Correct. Because it tells you, here's here's if you wanna find more, but also if you just wanna verify, you could go easily go into it. And we try to lighten the load on the application layer. Like, we actually have trained the model, to respond to requests for grounding from the context window. So we'll add APIs to make that more semantically obvious, but it's already trained to do that. And we think it's very, very important that application developers take advantage of of features like that and and take advantage of grounding data. And and that just so everybody understands, to make sure I understand this, that's, like, cutting edge research and that's new. So if someone used NLM a year ago and asked questions about hallucination a year ago, is that the type of it's a highly it's highly likely that with your research, this is actually getting Yes, sir. Yeah. Yeah. We focus on this. Exactly. How you doing? I'm Brian. So first of all, for everyone in the room, if you see if you ever put deep on the ROI topic, you can go through, like, usage metrics and all the examples you ever get. But if you're really sent in a rabbit hole by your CFO, one tip is to say, ask them what the ROI of G Suite or Office 365 is and how they meticulously measure that. It's kind of a show stopper. But the second point is I've used a lot of different enterprise tools like Glean, including Glean and I find it fantastic. And first of all, I'll shout out to Glean for the security data privacy and legal teams love Glean versus some of the newer players and especially since it's been around longer. But the question I have is, they really rely on high quality enterprise data and spoiler alert that doesn't exist in every enterprise. And so how do you see that as a limitation going forward, particularly with agentic workflows? So if we take the example that you gave tomorrow with the launch date, what's the source of truth? Is it the official plan? Is it a Slack message that someone has sent? You know, what is the source of truth? And can when can we be confident to say, okay. Yeah. Fire it away. You know, I I trust the machine to to got this. So So a lot of things that we look at is we look at all different kinds of signals. So we look at activity data. So is there a document that has more activity on it because more people are looking at it? It's gonna be more authoritative that a document that somebody wrote that nobody's ever read or looked at or commented on. When was the document written? And also who is commenting. So we we have a sense of we understand who are experts in a certain domain through we have your org chart. We know what document you've written. So is this person an expert on prompting? Because they wrote the prompting guide, and they wrote all the documents about prompting. Well, if they're commenting on it or it's a document they wrote, it's it's probably more authoritative. This is part of the secret sauce of Glean is that because we have this three sixty view and we have somebody who's answering questions about a topic in Slack, they're gonna be more authoritative. So that's what we use to understand. Now, obviously, if there is no authoritative information, then we can't find it. But we can suss out, and we can even, in our answers, explain that, well, this feature, it says it's gonna attract to this, but this person has some issues and concerns. And then you can and, also, that's kinda like you broke the prompt for it. Find it so you to to the point early about understanding that prompt. But that's that's a lot of the trick of how do we get this information from the tools and how do we use it effectively? How do we make sure those signals are used in the right way? I mean, you took the words right out of my mouth, first of all. Like, we're doing some research into this, but that's exactly how we think about it. One thing I just wanna add briefly is that the answer to this isn't deterministic. You heard all of the ways in which you can think about expertise and recency and and how to weight these things. And this is particularly good for LLMs because it can do some prediction and interpretation about all of that unstructured data and all of that mass out of all of those sources of data. So but y'all now have me wondering what my internal Amazon, like, Uber score is. Oh, fisting. Doesn't always rank with hierarchy. It's not always who's seen Hi. I'm Bill Lasier. I'm part of a working group at the Digital Economists. It's a think and do tank. So we're looking at responsible AI. So how do you see the enterprise embracing responsible AI in guardrails and guidelines? And the reason I asked, we're doing a study on return on investment of ethical AI as a component of responsible AI and how you measure it, you know, back to your ROI question. Do you have any views on that specifically? So you can fly you responsible for that. Well, everyone defines it differently. Some narrowly, it's privacy, it's security, it's it's governance. Broadly, it's what's the broader impact of people on society. So I'll I'll take the it within the enterprise, we spend a lot of time looking at the practice and security and making sure that we understand who can see things. And one thing I didn't talk about yet is a lot of times the information is not accurate within the enterprise. So I said, well, we're gonna make sure that we give you access to documents that you have access to. But a lot of times, things are not locked down in the way they should be. So HR teams may not lock down all their documents, and we somebody might ask when's the next RIF and get an answer and when they shouldn't because the document didn't have the right permissions. So we spend a lot of time on the responsible part of that, making sure that we can work with our customers, and they can say, red list these kind of documents, screen list these kind of documents. Never show an HR document to anyone no matter who they are, no matter what they ask. So we're very we work very closely on that and the privacy and the and the security of that and understanding what people can can look at. Or maybe Michael wants to comment on the other part of it. I mean, we certainly think about safety. We still think of existential threats. We think of narrow threats and enterprise threats. We talk about this in terms of alignment a lot of the time also, and alignment is very important to the ROI for any user, enterprise or not, because nobody wants to put a tool into production or use a tool that's gonna lie to them. It's gonna cause harmful behavior. Like, a lot of this has a virtual virtuous rather virtuous cycle where where, the safety makes it usable at all. I think going back to the hallucination question, people talk less now about hallucinations than they did a year ago. We still talk about it, obviously. I'm just saying, on a relative measure. And and when people used to talk about hallucinations with me, they used to use very emotionally charged words like it lied to me. Right? It would it would describe, emotion and and motionlessness to it. And I think that's very telling. I think I think there's no, like, cost to thinking about this ethically or or thinking about this in terms of alignment, and it's actually, like, quite, conducive, to to being able to put it in production and and use it at all. So you mentioned that, like, you scan through, like, conference Jiras and everything. So do you actually, scan through the reports and, for example, like, Adobe Analytics and, Adobe reporting, target reporting, Tableau, and other stuff as well? Yeah. So we have over a 100 connectors that we connect to your enterprise data and that we are able to adjust and index that information. If there are bespoke data sources that a customer has, we also have an indexing API that's very easy to use that they can connect to to ingest their information. We can help with that. And sometimes there are you put numbers that have, have a new data source that is on our list or that we want to build for other risk first as well. So, yeah, we have, we have a a very strong pipeline of connectors. Yeah. Other question is around, like, in conference, like, you have diagrams, like, sequence diagrams, flow diagrams, whatever. So if there are engineering questions on, like, these kind of things, so do you actually scan those diagrams and then, like, give responses based on that as well? Yes. We have the ability to look at, let's say, a PDF and understand it. We do depend on the models of how accurate they are in understanding them, but, yeah, we do look at information within PDFs. So given Glean recent focus on becoming an AI infrastructure platform, how do you plan to differentiate the offerings and creating specialization from potential competitors like perplexity entering into the enterprise searching search space? We're very focused on our customers and making sure that we're that we are developing the best thing for our customers. And we've got five years of building a RAG engine to understand your organized porting sanctions data to deal with scale as as Cates was saying earlier with customers with over a billion documents, a lot more than 500. And so we make sure that we can index everything. And we make sure that we're enterprise ready, that we work in your enterprise. So I'm very focused on what are our customers asking for and meeting their needs. And I think that we've got a great solution. So so but I just said but before I could jointly, I was briefly a a VC, and I was on the deal team for a lead. We did a lot of customer calls, some with maybe people here. And, that's what that's what differentiated Glean to me is the customer satisfaction with the quality, the results, and the ability to find information. The connections enterprise data are really hard. And I see a number of solutions today saying, okay, so if someone had the connection center provides data, will allow people to upload some data, maybe a couple 100 documents, etcetera. And it's a kind of general to that, right, you have to manage them independently, you basically never do all the security reports off here. And so that is putting some big pool and you've script away the permissions. And so getting that right at scale in real time is, a big part of Seagrassaults. Yeah. One thing that's interesting is how our customers a lot of our customers use Glean through the APIs because the APIs are very robust so that they can build their own applications on top of the the information. And that's something that, again, we're seeing a a lot of people tonight. Hi. I'm Mimi, and I'm a product manager at SAP. And I was curious about the extent of your collaboration across the three organizations. Like, do you go, do go to market activities together, like sales operations? Do you potentially solve use cases together so you coordinate road maps, you know, that that kind of thing? Yeah. You're you're at one. So welcome. Is that something that is a great concept? Yeah. I I maybe I'll send an AWS perspective and then and then the partners can share our partners can share from there. Yeah. We we AWS is like the center of a large ecosystem. And we work with partners as customers and as go to market partners and critical feedback providers for our servicing roadmaps, our product roadmaps. And, we have offerings at the areas of Marketplace where our partners can actually our customers can transact partner solutions with AWS. We do a lot of enablement work with partners to make sure their solutions, work best, align road maps, and then do a lot of go to market together, including account level coordination to make sure we give the end customer the best experience. And we do that even when we have competitive problems. I just wanna, like, stress that. Like, there's we AWS is a really big tech. And so even if we have a competitive product that, like, maybe does some of the features of a partner, we still happily, equip and, and and daily sell with partners. Yeah. I would, I would echo what what Adam said about really great partnership. So we have we're selling in AWS marketplace, and that's been really valuable for us because it reduces friction for our customers, and it helps tell the story together. So we do go to market activities, sales activities. We also look at engineering integrations. I actually, this week, was in Seattle, meeting with the Bedrock team to say how how can we integrate more closely together. Yeah. And then the last thing I'll say is I think, you know, speaking for all of us, like, we are all better together. And so in order to provide you like, tomorrow you mentioned the customer experience. Like, that is so important for all three of us. And so in order to do that, we really have to partner together, and we're all best at different things. And so kinda you hit on it is, like, we all care a lot about this partnership, and that's why we're all up here. Hey. My name is Benjie. Question for you guys about coding. My engineers have been using ChatGPT to write code. And I read, you know, I think a day or two ago that Sundar said 25% of Google's code is now AI generated. Anthropic, you know, seems to have the best coding model, but there's sort of this minimum to get started. We had already started with, ChatGPT. So I'm curious both for all your organizations, like, how are your engineers using LLMs to write code? And then how would you advise me to stay, like, to what tools should I offer my engineers so that they're always on the bleeding edge? There's also Canvas. There's GitHub Copilot. How should we because it feels like that is one of the most powerful productivity use cases in play right now. So I'd be curious to hear how each of you are using LLMs for your own developer productivity. I don't think we've mailed it yet. We're we're kinda you know, we we say we're an AI company and we're using AI tools, but we're also figuring it out. And so we use Copilot in, internally, and we're all the new IDEs that are, the AI AI powered we're testing out, we're using. We played around with Devon and Cutvision and saying, how can we use that more effectively? So we tend to try and play with and test everything. The thing that's we're using the most effectively right now is probably Copilot, but we also use, I mean, I I'm saying this, but they self serving, but we actually use clean quite a bit, not for code generation, but for answering questions on code. Because it's kind of a compliment of, like, how should I approach design to this? Because it reads Glean reads in all your code. And so, well, like, how should I here's an error code. How do I debug it? So it's a huge productivity increase in combination with the other tools. So that's what we, you know, what we rely on internally. Yeah. And then I'll say the other thing is all if not every engineer, almost all of our engineers internally use it, I will early use it. I won't pretend to speak like I'm an engineer myself. But, it just augments every single engineer, every single product person. I think that's really the way to think about it is all those tools that you mentioned. You could change out the models. You can continue to play with it and see where the best use cases are, and then just use it to augment to help you write code and then edit it. Like, you know, imagine if every single one of your engineers had an intern. That's how we think about it internally. It's okay. I have a point answer. Benjie, use Amazon Q. Same system. So Amazon Q is the Amazon developer assistant. There's a free ID plug in, or you can get the enterprise where it you our approach has been to look at the entire life cycle of software development from planning, not just in line coding or coding assistance, all the way to ops. It's the only coding assistance is, like, live telemetry from your AWS account to know what's going on inside your Amazon instance in production operations. Nobody else has that, so you're relying on your users to go put that together. It is everything from inline assistance to helping you write unit tests, this fully agentic automated code, generation to basically generate an entire diff multi file diff. It uses the latest and greatest. We announced, Tuesday, we announced that it uses Claude, three five SONNET. The newest version is actually used inside that as along with other Bedrock models. So it's the best developer agent in the world. It's a it's a fast moving space. But we are investing in that in a way to make it useful for you to the entire life cycle of applications in a way that I don't think any other coding assistant is. And, we've been on the SuiteBench leaderboard, with the new CloudSonic. They built a really simple agent. They've done an amazing job, and we will take advantage of that technology and, again, challenge ourselves to push even higher on the autonomous. But, our engineers, tens of thousands of them use Amazon cube. We're writing Amazon and AWS and using AWS with Amazon cube. So I'd encourage everyone to check it out. Give it a give it a shake out if you haven't. That was a great Well, thank you to all of the panelists for that fantastic session. That was a recently recorded panel, But now I'm delighted to be joined by Alex Jones, head of EMEA Partnerships, go to market at Glean. Alex, I can see you in the backstage, and now you're on stage with me. That's also firstly, thank you for joining us. Live on your Thursday morning. How are you doing? Yeah. Good. Thank you. Very good. Hello, everybody, and greetings from London. Awesome. We've had a few questions come through the chat, and folks, do keep dropping them through. We've got nine minutes to to go through a few bits. So let me I was just gonna add, I know there was a question about the, about whether this was accessible. It is fully accessible. It's a, as you said, Tim, it's a recording from a recent session that we did, with AWS and Anthropic in San Francisco. It's on YouTube. It's also if you go to Glean, Glean site on YouTube, you can search and find it there. Awesome. Awesome. Well, Alex, if it's okay with you, let's start with Harry's question Yeah. Who was interested and actually a really interesting point I read about recently, around the use case of custom support. He asks, do oh, just moving. Do clean CS tools differentiate different cultural nuances in responses across the globe? He sees it as an obvious task in, I guess, a seemingly straightforward direct human employee replacement with these agents? How do they We're we're not a customer success platform, a customer support platform, but what we do is we plug into things like, Service Cloud or Zendesk or, ServiceNow, those sorts of platforms where we can then search across the knowledge base to actually help prepopulate answers to questions. So we're going across the enterprise data to help funnel that information into those kind of those kind of tools. But, actually, the nuance aspect of that is a lot of that is relying on kind of LLM technology of interpreting. Now within Glean, we've got ways that we can learn and we can progress and we can constantly kinda analyze what that's about so we can we can pick up the nuances within your organ the organization about, different nuances around the world. But it's a combination of using the customer support tool, using Glean and the data that it's accessing and indexing and the LLM to actually resolve those. So, yeah, definitely ways to do that. Awesome. Well, thank you, Harry, for the question. Jack has the next question. He's asking around agentic tools that self correct. How do these influence the way the enterprises should design their operational workflows? Yeah. It's a it's a really interesting one because I think there is there's apprehension about that ability to run an agentic workflow that is self learning, self correcting the whole time. So our view is, again, that we were looking at keeping the human in the loop at certain stages within that workflow, that business process. Where do you want the the humans to come in and interact with it? And I think it's about building that trust and building confidence that the data it's pulling on that is indexed in Glean, that you've got confidence in that index, that you've got confidence within that dataset. That's often where we see the hesitation, and that's one of the problems that Glean is trying to solve for companies. It's saying, actually, you do need to when you're using things like LLM technology, you do need to be anchoring it and rooting it in trustworthy, permissioned, governed enterprise data, and that's one of the things that that Glean really solves for within the enterprise. Great question there because I think there's a lot of apprehension about that. For sure. For sure. And then Albert had another question here at the end. He is asking, how does Glean ensure accuracy or relevance in its search results from working with domain specific jargon, specialized knowledge bases? Yeah. So there's a I mean, there's a a whole host of ways that we do that as a as, Tamar was talking on stage as well around. The way we integrate, and connect to data sources and the way we index those data sources, we're pulling multiple signals off that data source. It's about 50 or so, and that can be around, you know, when things will last access, popularity. But it's it's it's it's a really kinda complex algorithm. It is the secret sauce of Glean, I would argue. It's how we kind of index that data, profile that data, understand that data, and then rank it so we know the when the the relevance of that data to the query that's coming in. And then we also have within the platform a self learning capability that actually as you use it more, it learns the language and the jargon of an organization or a knowledge base. And so it it could say, you know, that that you might call it revenue, Tim. I might call it ARR, but we know we're talking about the same thing. So it's multiple different vectors that we're bringing together to really analyze that data in an incredibly detailed, comprehensive way so that when you're doing a query, when you're doing a search, when you're surfacing that information to an agent, we're giving you the most accurate information possible based on your knowledge index. Awesome. It's super impressive. We've got one more question from Harry again. How do you how does Glean ensure robust security? I think that's a huge Yeah. I I this is probably the number one question that gets asked, and I was wondering when somebody would ask. It's a great question, Harry. And, again, I think as Tamar was talking about on stage, security is paramount. And one of the first things we do when we engage with the customer is we work with their security teams. We work with their CISO organization. One thing to note about Glean is it's not a multi tenanted architecture solution as as, as Tamar was saying. It's a single tenanted solution. So if you're taking it from us as a managed service, we will create a tenant on a GCP instance that is unique to you and manage it just for you. Also, we have the ability to deploy into AWS, Microsoft Azure, or GCP. So you can deploy it into your tenant, and then you've got more guardrails, more, and more sort of hands around that. So that's one aspect of the security is the deployment architecture. The other thing is we have a product called Glean Protect, which is constantly looking at external threats, jailbreaking of of, of agents, you know, DLP type, capabilities, constantly scanning the permission capabilities within the underlying data sources so people are only seeing the data they're allowed to see. So security is absolutely paramount to everything we do at Glean because I think when you're when you're doing an Internet search, it's very different to when you're trying to apply that internally on enterprise data. You've gotta make sure that it's fully permissioned, fully governed, fully secured. So it's a huge aspect of what we're doing with, with organizations. And it is it's probably the number one thing because of the anxiety level of kind of chief security officers, etcetera. When you're when you're coming and talking about deploying an AI solution on their internal data, accessing their internal IP, that puts the blood pressure up on security at the offices. So sitting down, working through them, we've got a lot of detailed information on our website, Glean.com. If you look at our trust center, whole reams of information around how we manage security because it's the number one most important aspect of what we do. I'm not surprised at all, but it's interesting. Has that kind of, perception of rep decreased, Alex, recent years or or not so much still? No. I think it's, I think, actually, in some respects, it's it's kind of increased as as the models are getting more advanced and and and people are using them within their enterprise space, sort of the shadow AI kinda capability like we have shadow IT and shadow SaaS with applications. But it is increasing. So I think security is probably even more important now and is more up the agenda for board level as well as kind of CXO level as well as the CSO. I think it's actually more important than than actually decreasing. But it's, again, sitting down and explaining understanding the customer security posture, understanding how we define security and how we map to that posture, that then brings everybody's kind of anxiety. Once you so once you're in the detail in discussion, that's when the kind of anxiety levels come down, Tim. Awesome. Awesome. Okay. Well, I mean, a a super interesting probably could probably go on for a lot longer, but I know, Alex, you've got another commitment in a couple of minutes. Any other questions? Sharing my LinkedIn profile. If people have got questions, please, drop me a a a message on LinkedIn. Happy to to answer. And, you know, hopefully, it's been useful, giving you some thought provoking, ideas. And, yeah, hopefully, we'll hear from you at some point. Cool. Well, Alex, thanks so much for joining this morning. Thanks to the team that put the video together and the panelists on the session. And, yeah, thank you everyone for joining today's live webinar. We'll see you soon. Thank you again. Thanks, everybody. Bye.