Name: Stop building AI pilots that go nowhere
Uploaded: 2026-04-01T17:30:08.519Z
Duration: 59 min 40 s
Description: Stop building AI pilots that go nowhere

Transcript for "Stop building AI pilots that go nowhere": No. I was on mute. Hi, everybody. Good morning. Good afternoon. I'm super excited to be here to moderate this awesome discussion around building AI pot pilots that go nowhere, and we are joined with some awesome panelists. To kick it off, my name is Nicole. I'm director of RevOps and a RevOps Alliance Ambassador moderating the discussion today. And we also have Adam and Mason with us, who I will kick off to do a quick round of intros. Adam, why don't you give us a start here? Yeah. Sure. Hey. Yeah. Happy to be here. I'm VP product at ZoomInfo. I'm one of the VP products. I was a founder before ZoomInfo acquired my company, and a lot of AI initiatives roll up to my team, MCP, internal tooling, productivity. And I obviously work closely with a lot of our customers on this stuff too, so happy to be here. Awesome. And great to be here. Mason McMullen. I'm the head of rev ops and strategic partnerships at Alisio, which is the go to market AI workspace. I forged a lot of our tech partnerships and handle all the rev ops internally here, and I do a lot of other stuff as well. We're a start up. And prior to this, I was in the rev ops organization at Alteryx for about four years. So been in ops for a while and very passionate about it. A lot of hot takes to share today. Love it. Yes. Yes. I think in in ops, we're seeing this influx of use AI, use AI, where where can we use it, where can we do it better. But, like, for this discussion, like, a lot of our AI pilots don't make it to production. In fact, only 20% of them actually do. So when we take a step back, I think for the first topic, can you both walk us through, like, what's breaking down? When sales ask for AI tools and the data's not not ready, where does the conversation go wrong? We'll kick it off. Mason, why don't you start us off here? Yeah. Yeah. I mean, that's that's kinda where it all starts. And and, you know, organizations are are pushing for AI, and and they're wanting to roll out AI in some way. And the data is always gonna be a problem. Think what we're seeing right now is is more of a common in info set questions first and then in the data problem second. So, you know, IT is is where it starts. You know, can can you look at an AI tool and understand the security and make sure that you're not exposing internal systems of data where they shouldn't be exposed? And then the second question, once you're past that, is is the data good enough to run an AI pilot? And I think, you know, part of it is the visibility of AI tooling too. You know, it seems like analytics solutions in the past have failed kind of silently or quietly. Maybe they lack adoption, but when we're talking about AI, it's often a very public failure when it doesn't work. So, you know, just to start off, it's it's just getting buy in from the from the right folks and the understanding that AI is not going to solve all your problems. It's going to solve specific problems. And I think starting with those specific problems and we'll probably talk about this later in the discussion, but starting with the business value, starting with what you're trying to solve is really important for success when it comes to comes to AI regardless of of data quality. Absolutely. I and I think, like, in rev ops, we sometimes see, like, different definitions of data points across the business, which can lead to further down downstream issues. That's a really good point. Adam, any feed feedback or comments on this part? I don't know if I can say better. I I think that was a that was a great answer. I'll I'll yes and it, I guess, which is absolutely right. The data is where it all falls apart. Right? I I think what people don't ask when they wanna roll out an AI initiative, an agent, a project, whatever it is, specificity is right. I think Mason said that. They don't ask, like, how do you do it today? I think people often think of it as, this magic box, and we just ask it and it's gonna do it. And you say, well, how does your SDR do it today, or how does your AE do it? And they haven't really mapped the process out, and and that creates an issue. Right? Because you are it's just the same as automation. You know, ten years ago, everyone's automation automation. And, you you gotta understand the process first, and then the inputs and outputs. Garbage in, you get garbage out. I think with AI, we lost a lot of the basics. Everyone got super excited about it, and it's just same thing we've been doing for twenty years. Right? So Absolutely. Yeah. And so going going off of that, do you have any, like, real examples of, like, a project or something you put into place that show, like, how bad data can compound in a in AI? Oh, man. I have I have so many. I mean, ZoomInfo's whole business is this now. Right? And, we spend something like $250,000,000 a year on data cleaning, just like that part of it. Right? Something like that. And so I see the whole spectrum of it because ZoomInfo is pretty big. I think we have about 35,000 customers. I talked to a lot of them. Some of them are very mature. Fortune 500 companies may update science teams with evaluation frameworks, and they're doing all of the right stuff. And then the other side of it, you see, hey. We tried to do this. We put an LLM on top of our CRM, and it doesn't work. And it's always wrong. It's like, well, yeah, because the CRM's always wrong. Like, no one's using the CRM. Right? And so I I think that the basics that it always comes down to whether we're building stuff internally and or whether it's customers, the failure modes, I think, are always the same. People over optimize for prompts. Like, hey. If we have a magic prompt, it's gonna work. And actually, all you really need is good evaluations. Just 50 test cases that you can check it's working against, make sure you know your bounds. Like, we're okay if it fails in this way. We're not okay if it fails in this way. It doesn't have to be super complex. I'm not talking about big engineering projects, but that's the basic failure mode. Everybody's learning the evaluation matters more than prompting or everything else. Yeah. Very, very good good point there. Mason, from a rev ops perspective, like, previously, we've really been focused on data quality as it pertains to, like, duplicate accounts or making sure that the reps are filling filling in fields. But how do you think the data quality for AI is different than, like, what we've previously experienced with CRM data? Yeah. Yeah. That's that's the that's the million dollar question. It it you know, we always think about, you know, data as needing to be, you know, clean to transform outside of the source. And I think where we're going with AI is that your source data should be the source of truth in that, you know, it should be the the the cleanest the cleanest data you can make it and let AI kind of sit directly at the source. And so those are the conversations that we're having where where companies are saying, hey. We're not ready to implement something, you know, an AI solution because we haven't gotten the data in a place where we could use it. And I think when we're talking about AI tooling now, it it becomes a meta analysis type type approach where it's like, we can actually help you clean your CRM. We can help you keep source data clean with with some of these AI solutions. So, I mean, there's it's kind a double edged sword because people want to move quickly, but they also want to wait until their data is perfect. And I don't think you can you can do that. I think you need to to have solutions that can kinda equip your AI with the best data, also but be able to use the data to keep your data your a a sorry. Use AI to keep your data clean. And, you know, that's, like, part of part of the solution. So it's it's kind of a it's kind of go back go back and forth between, you know, is the data clean enough, and can we use AI to help keep the data clean or to make it cleaner? You know, we always look at the CRM as the example of of dirty data. But if you're not constantly cleaning it, you're always gonna have data problems. And then that exists without with or without AI in in my opinion. Yeah. Very, very true. Do you do you have any real examples that you can share about how bad data can count compound in AI? Yeah. I I mean, the the CRM is kind of a a perfect example. If you are using AI to, you know, pull account list and do direct data enrichment, if you're, you know, pulling duplicate contacts or you're pulling old data that's not up to date, then you're always gonna have a problem with with your AI solutions, especially when it comes to the really low hanging fruit use cases that we're seeing AI deployed in, where where it's contact to data enrichment. So it's like being able to make sure that your data is clean at the source when it so it can it can actually match to to good data. You know, that's kind of what we have been doing with ZoomInfo is, like, the the data quality is is the most important thing. And so being able to keep your CRM clean at the source is is the place to start. And so, you know, it it always becomes a oh, we're actually seeing duplicate contacts. We're seeing old contact data. So being able to identify those, clean them up at the source, and then pair them with good data like like the ZoomInfo data that that we've we've we've worked with, then that's that's a that's a good solution. So, you know, we we see that all the time when it's when it's, hey. Use this to get your data clean and then pair it with with good data to to make sure it stays clean. Yeah. Perfect. Yeah. Clean clean versus not clean data is mission critical. We actually have a question in the chat that I wanna throw out there based on this topic. So from Pascal, we have, when we want to remove the old data, how long should the retention time be for that data in your opinions? Adam? I mean, I I I don't yeah. Go ahead. I'm sorry. yeah. I was gonna say it depends on what what data you're talking about and in in what cases you wanna retain it. I think you just wanna think appropriately about, like, reversible decisions. Right? Like, are you gonna regret like, what's the what's the downside to removing this? I actually biased more to what for a data company, and I'm admittedly a data hoarder. I hoard all kinds of data in my personal and professional life. But I do try and bias, hey. If I don't have a use case for it and I can't, I'm I'm removing it at least from the main signal, put it in cold storage. Like, state storage is so cheap, just generally speaking. Just keep it out of your main pile of data because it's noise. Right? But at like, be ruthless in putting it in cold storage. Be very hesitant to permanently delete stuff. Yeah. I I agree with that with that take. There's this kind of rev ops need to have everything at your fingertips and to store everything forever. And when you're talking about, you know, daily snapshots or weekly snapshots or monthly snapshots, those things compound, you know, exponentially very quickly. I I would say, you know, when it when it comes to revenue data, at least have, you know, two full years of of good snapshot of revenue data because that's gonna be the the most relevant when it comes to predicting the future of your business. Past that, like Adam said, keep it in cold storage. Maybe don't maybe don't delete it. Yep. Absolutely. I'll also add in a point too, like, make sure to take your IT's policies into into consideration for security purposes. I think that's also an important point there. Alright. So we talked about, like, central governance and, like, why it's important to have our source of truth and data quality. But now, like so we're in this place where rev ops, we want to do AI. We wanna keep our CRMs clean. We want to build agents to help our our reps do things. But your IT team needs standards for how that's all built. So in your opinion, who do you think actually owns what with that process? And then how do you go about piloting that change both, like, organizationally, arc architecturally, and on the op side? Any volunteer for for first one on that one? Yeah. I I can I can maybe leave this one off? IT is mean, AI is kind of unique when it comes to to ownership. And I think that there's there's an opportunity for kind of a decentralized ownership structure when it comes to when it comes to AI. And that that couples up IT with with RevOps. RevOps owns the kind of use case definitions and the success metrics for the AI implementation. It owns kind of that that backwards work from here. What do wanna achieve with AI and then the steps to get there? And then IT owns the data the data itself and the security. So and I and I think that that could be the approach to to take. Like, if you're able to to figure out that kind of partnership where RevOps is not the the sole owner. And I'll probably say this a lot today. You don't want to fully burden your RevOps team with full ownership of AI solutions. It's it's not fair. I will say that. So being able to to own it from RedOffs perspective, like, hey. This is gonna be successful because we make it successful and let IT handle the data, the security, and be in those be in those conversations across the org, I think, is is really important when you're rolling out AI solutions. Adam, anything from your side? I I think that's absolutely the right answer. If you think how we how software played out, right, from, like, '99 through the February, you started off with IT as a software buyer and the implementer and, like, on prem to cloud. And then eventually, we got to the SaaS sprawl. Right? And we all have ops functions now. There's marketing ops and rev ops, and and that's the right paradigm because it means that you can solve local problems for that function really well. Things like governance security centralized in IT is is the right model, I think. And I think we'll see the same with AI. Everybody wants to try it. We have we know what we need in RevOps. We know what we need in MarketingOps. IT don't really know, but they do know security governance, that sort of stuff. So I think that's the right answer for sure. Yeah. Good. Good. Good point. With that in mind, though, how do you do that without actually rebuilding your entire tech stack, which I'm sure you both are seeing all over LinkedIn, like, rebuild this, rebuild that. But that also is a lot of work and maintenance. Very interesting. Yeah. I mean, this this might be a little bit of a biased answer because of what we're building in Alessio. But, you you know, to me, it's you know, we don't want anyone to replace any of their their GTM tools. I think the best AI solutions will unify data across different platforms. And it's an easier conversation to have when you're coming in and saying, hey. We can we can bring this data together in a meaningful way that will will improve your experience across the organization with with a myriad of applications versus trying to say, like, this is gonna replace replace something. So I I would say, you know, hot take is that the best AI solutions don't require you to replace anything. Like, you can if you if you use AI to figure out, hey. We're not actually using this tool. Great. Get rid of it. I mean, another another meta application and emergence value value add for IT for AI is just, hey. We're actually not using this tool. But, yeah, don't don't replace anything if you don't want to. You know? Keep keep the the stack that you're using and optimize it accordingly for the tools that you're using. But don't let an AI tool come in and try and actually, you know, replace several tools or or, you know, require you to redo your entire tech stack. That's a that's a dangerous road to get on with AI. Absolutely. I, yeah. I I kind of agree. I think this stuff goes in phases. Right? Because it's the same as buying tools. It's like, you don't wanna buy a tool just because you want the tool. Right? You you've gotta have a reason for doing it and a problem you're trying to solve. I think getting rid of tools is the same thing. It's like, we all love a spring clean. Right? Like, it feels so good to clean out the closet or whatever. And there's that temptation, but maybe it's not even serving a purpose. Of all the things that you could be doing, the highest leverage is probably not trying to migrate a bunch of tools right now. Like, there's so many other value adds you could add on. And I think I think, Mason, you're right. I think, like, you'll get your AI to a place of maturity, and one day, you're gonna wake up and realize, hey. My sellers, my frontline users never touch the application left for half of these tools anymore, and they're basically just the database. Let's migrate our data to a database. I think that will happen. I think that will that's the downward trend I think we've seen in the SaaS market, and we'll see that play out over the next few years. But I think if you're an operator today, I don't think you should wake up and try and vibe code your way out of these tools today. Like, there's just other things you can be doing. Like yep. How would you how would you recommend if leadership pushing, like, AI AI, replace whatever you can with AI? Because I think a lot of us are probably experiencing that too. Hence, why a lot of pilots don't don't make it out of pilot. Yeah. It's a good question. Go ahead. go? Go ahead. Okay. Yeah. I mean, that it does seem like that that's that's top of mind for leadership. Like, we're we're spending a lot of money on on tech tools. Like, we need to, you know, cut across across the cut cost across the organization. When you I would I would push back and ask, you know, where does the data that powers AI tools come from? And in a lot of aspects, it comes from the tools that you're using day to day. So if you get rid of your tools, you no longer have data that can that you can use AI to leverage. So that's that's the paradox of of telling everyone to cut costs using AI because something has to offer your AI tools and its data, and it comes from your systems. Yep. Yeah. Yeah. I think I think that's basically right. I think the I think the push for AI is right because, like, if you look at, say, coding, right, like, with AI, the curve of it is so far ahead because it works really well and, like, you can do so many things now. The difference between that and, say, revenue and other business functions is that a coding environment, a code base is homogenous. It's all in one place. The agent can see all of the data it needs, and so it works really well. The models work really well if they have the data and everything else. And I think if you start with the tool again and say, like, we want a tool. We wanna replace. You gotta go back to, like, what are we trying to achieve? What do we wanna do? Are we trying to get faster, cheaper, better? Are we are we like, what is it you wanna achieve? Don't start with the hammer. You know, if you have a hammer, everything looks like a nail. And I'm not I think that's the wrong framing. Right? You wanna start with, well, where are we trying to get to here and where is it appropriate? So yeah. Awesome. Love that analogy too. Yeah. Really great. Alright. We have a couple questions in the chat. So, Adam, I'll read this one to you first, and we'll kick off the conversation. So we have David asked, in examples like CRM, couldn't we use AI to help mitigate convoluted data and steer agents to keep up with what is current and value valuable? What what. are the yep. Yeah. That's that's a great use case. If you think about what when we say AI, we're really talking about LLMs. Right? Not all the other spectrum of AI that you can use. But if you think about what LLMs do really well, they take large amounts of information. They run it through a kind of stochastic compression process, and they output something at the end. So if you give an AI, like, 10,000 words of research and you say, write me a 800 word blog post, it'll do really well. If you give it, like, a 100 word prompt and you say, hey. Write me a blog post, it'll spit out, like, a fine but not great thing at the other side. So it takes data, compresses it. That's, like, the main function you can use it for. CRM is a great source of that because the data is structured. And so one use case we have at ZoomInfo is we, we obviously have Chorus in our product suite, but we integrate with Gong and Zoom as well. And we take all of those call transcripts. We process them through what we call the semantic data service, but I'll spare the technical stuff. It's an ontological knowledge graph builder processed through LLMs, and what you get out at the end of it is this really structured dataset instead of, like, a raw messy transcript, and you can query it. And, hey. Show me all the pain points for this account. Show me all the objections they've raised, and you can timeline it. And so you can then match that CRM stage and stuff. And so when you're talking about, like, enhancing CRM or let's let's take CRM out of it. Let's just call it, like, customer life cycle timeline or something. Right? AI is such a great use case for that. Like, writing emails and, like, putting pitch, like, so, yeah, it's cool and flashy. All of the value in AI is really in these kinds of things, I think. Mason, any feedback? Yeah. I actually kinda wanna dig into the second part of this question in the chat there. He asked, do you couple evaluation with direct intention as to what you're wanting to achieve? And and and I would say yes. And what we've been what what we've been doing with with customers is is they're evaluating kind of our AI solution is we've been really, really saying, let us help you identify the pain points that you're having day to day. And this is especially true with rev ops or salespeople or BDR teams. Like, it it really crosses the gamut of of GTM. Everyone has some sort of pain that they're facing day to day. So And being able to kinda dig in and identify one or two pain points that we can solve with AI is that is that intention as you and and that and couple that up with the evaluation of the tool's ability to solve those problems, I I think is is the right approach. And it and it it kinda goes along with thinking about these sorts of of data cleanliness initiatives, but but also, like like Adam was saying, kind of this customer life cycle, how do you apply AI at each each part of this customer life cycle? When you're acquiring customers, when you're getting them through your sales funnel, when you're retaining them and expanding them, really thinking about two or three things initially that you want to solve and getting those things solved and proving out the the value of the AI solution that you're evaluating, has been really valuable for us. Because what we're seeing is people will solve those two or three pain points within the period of a of a trial. And then within that same period, they'll also expand into five or six other use cases that are emerging that they weren't even thinking about when they started using the tool. So, yes, definitely couple up intention with evaluation of these tools. Can it actually solve the thing you're you're going with? And I and I always get a little nervous when when you get into a bake off and they don't actually wanna try the tool that you're that you're providing them. I think I think AI is a must try. Like, please try the tools before you buy them. Like, go go in go into it with intention. Like, what can I solve in the in the next month with these with these platforms? How would you recommend where to start with that? Because I think one thing that we're also seeing is the customer life cycle, for example, is very nuanced. There's a lot of stages from acquisition all the way through post sales and retention. But how would somebody go about, like, figuring out where to start when it all can see kind of overwhelming? Adam, we'll start with you on that one. Yeah. I I think that's, like, the perennial question. Right? Like, if you ask someone that ten years ago, it's the same problem. Like, what do we need to do this quarter to get us to where we need to be? And it's that hammer tool situation. Like, if you start with AI and you're like, where can I apply this? You see all you see is, like, a million places, and then a million blockers is why that won't work. And then you start with that, like, I have to redesign my whole stack to make this work. And, like, Mason's point is right. You you find the place that's on fire. Even just take a typical, like, funnel, right, a very simple one. There's a friction somewhere in there. And there's you probably know off the top of your head where the biggest friction is in your customer journey right now. Can you fix that? Maybe you can fix it with AI. That's fine too. But, like, that prioritization should always come from what's in the way of customers, what's blocking revenue, what's blocking frontline, what's blocking our view of reality. Are we getting good data up so we can make decisions? And then AI can be a tool there for sure, and it's a really powerful tool, but you have to start with the problems, I think. Yeah. Absolutely. Mason, anything from you? Yeah. That's that's a great answer. You. know, the we're we're the find the fire and and fight the fire with with a and a potential AI solution. And and I and I would say that, you know, lean into the rev ops rev ops team because rev ops team always has a point of view on where the business needs to lean in. And it's often it's often hard to get that get that buy in from the rest of the org as well. You know, robots tends to tends to yell loudly and oftentimes is yeah. It can be ignored or somewhat diminished. So, you know, shame plug for RevOps folks. The RevOps team usually knows what's going on with the business and where to apply AI, but I I agree with Adam completely. Find the fire and fight the fire. Awesome. Alright. We have another question in the chat. So, Mason, we'll have you kick off this one. The question from Brenda is how are you dealing with sprawled tool stack? Right now, I'm running a shadow IT project and have found out that we have at least 40 apps that are AI related that seem redundant. Yeah. That's a that's an ongoing problem. You know, we've we've, we've kinda seen it all as as we've been talking to talking to folks around the around the GTM sphere. You know, the the biggest biggest problem is just being the ease of spinning up a an LLM generative AI companion solution. And and then all of a sudden, you've got, you know, folks across the org that are copying and pasting proprietary data into public LLMs. So I I I don't know if there's a there's a perfect way to to solve that. I I I would say that, you know, proposing a a centralized solution is probably a a good approach where it's like, okay. We have four or five different generative AI platforms being deployed. We've got three trials running for GTM workflow builder platforms. Like, whatever it looks like, you know, it it it needs to be kind of an org wide initiative to to shrink the the footprint. And I and I think this is a great place to involve IT. Mhmm. They're the ones that can can actually, you know, gatekeep what what is connected to, you know, from a security standpoint, what we can and should be connecting to and what we shouldn't. So I would say, yeah, lean lean on IT to help help kind of curve the the shadow AI across the organization. But that's a it's a it's a tough problem to solve. You know? Centralized solutions are are a good approach, and I think, you know, the the do do more with with with less, I think, is probably the the name of the game when it comes to solving it. Yep. Absolutely. Adam, any input on this one? I think that's I think that's the right answer. I think there's two parts to that question. Right? One is how do I clean up what's already sprawling, and then how do I prevent this or enable in future? And I think we all know that, like, everybody all all of us in our work will take the path of least resistance to get somewhere. So if I wanna try a tool and I'm blocked by some central policy, like, I'm just gonna go the easiest way I possibly And so I think what you see I I see it across all of our customers. I see it within our own teams. The pull of wanting to use AI from frontline users is so strong. Everybody wants to try Cloud or ChatGeek. Like, we all wanna use it. Right? And so I don't think you wanna block that. I think you wanna find passwords, like, done within a framework that's easy and then you can at least have visibility into it. The worst is not even knowing about it. Right? Of those 48 tools, what about the 30 that you don't even know about yet? Like, I I I guess they're there unless you've been very, very thorough. And then the second part of that is, like, it goes back to the problem and the evaluation stuff, which is often we try stuff because we have an intention to see if it can do a thing for us, but we don't very clearly state, hey. Under what circumstances are we gonna say this failed and we wanna get rid of it? We all do that. Right? And I see so much sprawl just from the fact that the thing didn't work out, but there wasn't a clearly defined criteria, which we say, alright. Now we get rid of it. And so things just kind of linger on for a long time. And so before you start anything, I think just define, like, alright. This is success, and this is why we call it a day. And, again, that's framework. Right? Enable enable people to make decisions, but guide them through it. Yep. Absolutely. Great great answer there. And we, like we touched on this briefly about, like, the importance of picking, like, a narrow a agentic use case over, like, designing the whole system because that can be overwhelming. But in your opinions, like, what does agentic mean? And then how do you, like, measure success on your AI before it's even deployed? Mason, we'll kick that one over to you first. Yeah. I mean I mean, to me, an agent isn't isn't just a chatbot. An agent is is kind of an AI system that can take a multistep process and and repeat it autonomously. And the I the ideal agent is is smart enough to to self self to self fix if it if it runs into something that it that it doesn't recognize. So it's more than just a workflow. It's more than just a chatbot. It's actually a repeatable process that you can use that maybe takes multi steps to to do something. You know, the the common agentic approaches that we're seeing are are within GTM. You know, a AEs and and BDR teams are using AI to quickly find find contacts, enrich data, draft draft and send emails, and and be able to kind of meaningfully contact customers using using good data. And and that whole process can be done with with automation. And and the agentic approach has has guardrails. It has source specificity. You know, what what does it do with each source of data that it's touching? And then it has fallbacks. So if it runs into something unexpected, what does it do? How does it kind of approach that? And the the smarter, the better. So to so to me, a agentic means an intelligent process that is is repeatable across multiple source systems that can actually have guardrails and fallbacks if need be. Love that. That's. a really great answer there. Adam, any follow-up there? I I could talk about this forever, so I'll I'll try and do the quick one. Yes. Agree with that. I think the definition of an agent so if you think of automation as we decide the rules, and then it follows the rules. And if anything falls outside of that rule set, it doesn't work. Right? That's what automation does. Agents are different in the those are so automation has explicit rules. Agents have implicit rules. We give it instructions. We give it a goal. We feed it context, and then we trust it to reason over the tools and steps it needs to do to to hit that goal. And sometimes it fails and sometimes it succeeds. And sometime mostly it fails on data and evaluation, less so on the prompts in the middle. The models are very good now. Like, one of the best agents we built from testing has a two line prompt, but it has incredibly strict data and tool descriptions. Right? And so and so that's that's the agent part of it. The other thing I would say is that the mistake so many people make and that I've seen over and over again is we equate agents with people. Right? And so I've I've heard a million versions of I'm gonna build an SDR agent. And it's like, okay. Wait. Because an SDR is doing, like, 60 things. Like, an SDR is not just sending emails. Do you mean you're gonna do a cold email agent? And you say, well, yeah, that's what I mean. You say, okay. That's still, like, five different agents. Like, it's a team of agents to do that because you have to research who you're emailing. You have to look at the product and, like, all the the market and their IC. You have to make implication. And so this agent is really multiple steps. Like, when you write an email, Nicole, you reach into your memory of the context of the person you're talking to. You join that to the context of the conversation you may be already having. You then write the email. You then edit the email, and then you send the that's, like, five steps that we do as people. But because we do it all at once, we don't think about it as those broken down components. Right. With AI, you absolutely have to break it down that way because you need to test each step. Did it do research well? Did it do the drafting well? Did it do the editing well? Like, all of that is a different agent. And within that step, it's free to reason, right, and, like, make its own decisions and stuff. But if you just say, hey. I wanna do a cold email agent. You can package it and sell it that way if you're a product builder. Right? The customer doesn't care that it's five agents. But behind the scenes, it's it's not just this big homogenous lump, and that's that's a big mistake people make. Yeah. That it kinda reminds me of, like, that old thing where it's like, tell. me how to make a peanut butter and jelly sandwich, and you have to go through all the steps. Exactly. It's really, like, evening when talking to a to a LLM. And most most people miss a couple of the steps in there. Alright. We have another question in the chat from David. So how how do you think org should go about evaluating their AI provider? I think this goes back to a point, Mason, that you made that we should be testing and evaluating, but what's what's the best way to do that? Yeah. I will kinda circle back to something that I, you know, I said earlier. Like like you mentioned, you know, it's really about identifying the the pain points. Like Adam said, like, find the fire, fight the fire. You you when you're evaluating AI provider, you should start with the thing you actually want to solve with it. And then like Adam said, I think there's there's two pieces. There's a success criteria. Like, if it works, how do we roll this out? And then if it doesn't work, what what at what point do we just drop this and say this wasn't this wasn't for us? So I I think it really is those one to three solid pain points that you're like, this is this is causing me, you know, a time sync, a cost sync, a a person resource sync, and and figure out how to how to solve those problems. Like, start with one to three use cases that you wanna solve, evaluate the provider based on its ability to actually actually solve those those use cases, and then also kind of have a plan to to expand into other use cases as well. I mentioned this as well. Like, if you start with two or three, there's a good chance you'll find five or six that you can solve. And then you get three or four or five more users using it, and then all of a sudden you're solving all kinds of problems across the organization, and then you have a pretty good idea of whether the tool can do do what you want it to do. But, again, specific criteria for success, specific criteria for at what point do we do we call it a day? Yeah. And some Adam, anything on your side? I think that's a great answer. I'll. I'll I'm not gonna rehash that because that that's a really great answer. I am gonna say one other thing though, which is when you're evaluating and you're trying to say, hey. Did we succeed or not? Don't celebrate too early. There's this famous story from, I think, the late nineties. So when they introduced telephone banking, could call up and do banking things over the telephone. And they they launched it, and it it took off in, like, thousands of calls, and people would call back all the time and do telephone banking. And everyone's high fiving each other because they say, hey. Look. The thing that people used to have to go to a branch for, they'd now do on the telephone. Like, we've made it so much easier. And what they didn't realize is that, on average, the person had to call back, like, five to 10 times to solve their problem because it wasn't getting solved for them. And so the volume was like, oh my god. It's like, everybody loves this. And it's like, actually, no. It's a real pain in the ass, and it's not working. That's And a concept called failure demand, where demand for the solution goes up because it fails most of the time. Mhmm. And agents are so primed for that. You see that in so many ways. AI, generally. Hey. People love this and we're using it and, like, support chatbots are like that. It's a real risk for support chatbots. People keep coming back to our support chatbots, like because it's not resolving things. So you gotta watch for that stuff. I wouldn't, you know, celebrate too early, but, like, attack that onto what Mason just said, and I think you have a real recipe for success. Awesome. Awesome. Ed Ed advice there. So then we think about that, all the use cases, making sure we're being a little bit narrow with our, like, scope at first. But and we we touched on this a little bit too. When leadership is telling you AI, AI, use it, how do you actually measure the ROI? Like, what are the metrics that you can show your exact team that you're seeing value? And how do you actually, like, tell them, like, hey. This might take a little bit to stand up even though you want the results like yesterday? Alright. Now let's kick it off to Adam to start this one. Oh, man. I don't know if this is a controversial opinion. I think most things you can't measure ROI on. I think it's very difficult. There's too many factors that go into determining success or failure. I think for critical projects that you're really investing in, you really have to spend time thinking about how you're gonna measure that ROI. And there isn't a one size fits all in my view. You can't just say this is how we're gonna measure ROI, and it's just like you've gotta design the system that says, alright. Well, this is the actual thing we're doing, and this is how we're gonna see if we're getting what we want out of it. And then you need confidence intervals on that. Like, how sure are we that this is true? Because it's not just a 100%. Right? Like, what is it? And if it drops below a certain point, are you still okay with that? And I think most people it's not that you can't measure ROI. It's just that it's so much effort to do it. Most people would never bother for most projects. And so you really gotta think about, like, how much do they actually matter for this project or this initiative. Right? For. the big ones, you should do that, though. You should have a game plan, basically, and treat it as part of the not an afterthought. Right? Let's launch it and then check it later. Like, no. No. You need to build this in from day one. Yeah. Absolutely. Mason? Yeah. I I think also a great answer by Adam. I I I think I I'll I'll maybe just just add that I think there's a few ways to approach the ROI conversation. I think I think the first is to set reasonable expectations with with leadership. And I I say this with the same thing that I say about any analytics or or intelligence solution. It's like, have to be able to accept imperfection. And I think with AI, it's especially true. AI is not going to be perfect. Just like any analytics solution is not gonna be perfect. Just like any internal build solution is not gonna be perfect. So I think setting expectations early on, like, hey. 90% is is good for this solution, I think, is is pretty. important. And then when we talk about specific ROI, I I think, really, it it comes down to a couple of things. One is that take a snapshot of where you're at right now when it comes to revenue, productivity, quota attainment. And I'm talking from a GTM perspective because that's my world. But identify specific metrics that your your business is struggling with, whether it's, you know, customer acquisition costs, whether it's, you know, velocity through the sales cycles, whether it's, you know, falling off renewals or expansion. And and those metrics are usually tracked closely, and you can identify where there's pain. So snapshot that when you start using an AI solution and then measure it across the duration of the solution, especially when you're trying to apply it to those specific problems. So I I think being able to kind of take that snapshot and then and identify the the success criteria and the metrics and then see how those change as you're using the solution is is one way to do it. And I think the other is to to look at how much time AI AI can save you. And and really that's where we're seeing a lot of value with with organizations, especially in the GTM space where it's what process is taking your reps time to do every every week? Is it they're having trouble finding good contact data? They're having trouble, you know, moving deals forward? They're having trouble multithreading? You know, whatever whatever that looks like that takes time away from from doing the the key sales activities, I I think is is important. And so it becomes a question of this time is costing us, and can we save time by implementing this solution? And then and then identifying that kind of cost savings, again, as you go through the process. So I think it's really important to just identify those criteria early on, set specific metric goals, and see if you can reach up to those metrics as you're using the the solution. I think it's necessarily the dollar value conversation. Like, it's gonna be hard to tie specific dollars to AI AI tooling, but I think the time savings is identifiable, and I think the the metric changes are also attractable. Yes. Good point. The the metric changes over time, I think, are a really great foundation to set there. With that in with that in mind, say your VP of success is really focused on, like, retention rate, and then your VP of sales is wanting to know all about ACV. And they're trying to in to increase it. Now you have this competing priority with which both are important. What would you recommend an operator to do first to figure out where to start when you have those competing priorities come up? What do you what what do you think there? You there's a there's a real there's a real answer, and there's the answer that we probably is more practical. Which one does the CEO care about? That's probably the first one. Right? That's like the, you. know, pulling, I guess. I think I think the real answer, you've gotta get to the root cause of both of those things. By the way, I'll just say, Mason, the point you made about ROI being not just dollar value, time driven as well. Right? Like, if you think about all the time we're spending or sorry. Frontline seller anyone spends on non core value driving activities, like, they have four hundred and eighty minutes a day. How many do they spend with customers doing the like, it's not half even, right, for most frontline. And, anyway, you get to the root cause of that and you say, like, well, what is what's the drag on either of these? Right? And I actually think on a lot of those things that seem to compete, you probably get to a root where you could do both even if they seem a little in conflict to the surface level. You gotta go down. It's not easy, but but if you don't wanna do that work and you don't wanna do all the deep diving, just go ask the CEO, which I think is important, then do that. Perfect. Mason from your side. Yeah. I mean, that's that's a good answer at times. You know? What's what's the business priority based on what the guy at the top or the guy or gal at the top thinks is is the most important? You know, I I I I think, you know, again, like, here's a here's a shameless plug for rev ops. You know? If you're if you want to know about what's driving the business and what levers you need to pull to to maybe fix either of these issues, whether it's, let's say, an average deal size velocity problem or it's a retention and expansion problem, you can actually dig into the data and figure it out. I think that's that's goes along with what Adam was saying. It's like you gotta get one level deeper than just what an executive is asking for. You know? If if if if your business is primarily driven by existing customer expansion, not by growing average deal size on land business, then it's a pretty easy easy decision to make. At that point, you dig into the to the expansion, from existing customers. So, yeah, let RevOps do a little digging and maybe let AI augment that digging, to make it a little bit quicker. You know? So I. think I think RevOps RevOps run. Like, that's what we like to do. We do. And we see it all for sure. So with that in mind and all the a the a AI stuff, like, what do you think, like, what or what should RevOps and the data teams that we do work work with to, like, look at this data and dig in and build the infrastructure? What should be should we be able to do together in the future that we can't today? Open it up for for thoughts. Adam, we'll kick it back over to you for that one. Okay. Nice. I I'm gonna ask you a question that I would love to hear Mason's thoughts of when he answers this, which is do you think RevOps is becoming more of an engineering function unlike a data science function? That's, like, one aspect to this. I think in a lot of aspects, is. I see that with our customers. I see more and more engineers joining RevOps roles and building things from scratch and automating. And and that that division between it's like it's similar to that ops model where ops was a function in IT. Right? And everybody sort of relied on them, and engineering and data is kind of like that in a way now. And I think what we're seeing is people want these AI solutions is the pull on the central engineering and IT. It's not their core thing they should be doing, and so people are like, in I have the the product ops team at ZoomInfo rolls up to me. Within the product ops ops team, we have engineers who just build internal tools, and they're full stack engineers. Right? And I think we'll see and I see that in revenue teams too. And so that question of, like, where are we at two years from now? I think we'll see a lot more technical RevOps teams even if just it's even if it's just part of the RevOps team, they're gonna have their own engineering resource and data resources, I think, because it's such a core function to the business. Right? You get it right in RevOps. You get it right throughout your customer life cycle. It's going back to ROI. It's a huge ROI when you get that right. So Yeah. That's a great great question. I and this is one that I could probably talk about for a long time I I it it does seem that that engineering and I put that in quotes because I don't think a lot of of what RevOps is doing from an engineering perspective is is is truly engineering. You know? It might be, you know, workflow orchestration or architecture. And I think that's becoming a key part of RevOps skill set. And and I think it's it's important to couple that up with with the rest of the skills that RevOps tends to possess. You know, that's partnership across the business, aligning stakeholders, and getting consensus, being able to to understand the the key drivers of the business and and evangelize them across the organization to the different functions. I I think coupling up the ability to architect AI solutions with those other skills is important. And I think that's gonna be something that will be important forever going forward now in this world. What what I kind of worry about is is the the the required maintenance of of workflow automation. Yeah. I don't I don't ever want to see rev ops become the managers of of endless workflow processes. You know, I've I've seen in the pre AI world where we were architecting things with with workflow builders, you know, ten or fifteen years ago. You know? The these aren't these these solutions aren't new, and we've been doing this for a long time. And and it becomes detrimental to rev ops when they become just that. So I I would say that's that's the the danger. I think it's it's going to be important to be able to to work these solutions, but also to have a rounded a rounded rev ops skill set going forward. And and to really kind of be a a true operator, you know, and that means a partner to the business. And, you know, we talk a lot about democrat democratization of data, democratization of analytics, and soon it'll be democratization of AI. And I think it's important to to be focused on how does AI help people be better versus how does AI let us hire fewer people. And and I think that's the danger for robots too, it's like, oh, we can do what a 30% rev ops team can do with three people. No. I'm never going to support that. I don't think it's the right approach, and I think you're gonna be, you're gonna be underselling your rev ops team, and you're gonna be selling them up the river to some extent if if you try and do that. So around about way of saying, just be be very careful how you approach the engineering function of RevOps. It should be a key a key part of your skill set, but it should not be the only focus. And workflow maintenance is a problem. It's been a problem for a decade. Love that point. How would you recommend, like, smaller teams maybe deal with with that? Like, as a start up land where you have one, two, three folks only and, like, introducing, like, that that engineering concept into a smaller team. I I love it as a force multiplier. You know, like, AI should and like I said, like, AI should work alongside you to help you be better at your job. So for a for a small team, amazing solutions exist out there. They can let you really ramp ramp up things. And I think, like, if you're thinking about small organizations that have, you know, five to 10 sales sales folks, a couple sales engineers, it's like, that's a good time to start democratizing AI. Don't just keep it in the rev ops here. You know? Like, obviously, it can help you be better at your your rev ops job and you help you keep your CRM clean. It can help you really analyze the business quickly and and efficiently. But getting it in the hands of your sellers and the hands of your business development folks and letting it help them be better at their job, think, is is key for for small teams. So let it be a force multiplier, but also start democratizing it early because it's way easier to do it when you have five or 10 salespeople than it is when you have 30 to 50 salespeople. Yeah. Great great point there. Alright. Last question here, and then we'll open it up for any other q and a since we have about ten minutes left. How do you figure out how to push adoption? Like, I'm sure, like, a lot of the pilots and AIs things that we're building are great, but if they're not adopted, then it's kind of like, well, it doesn't really matter. So how can you, like, make sure that adoption is going well, and how do you tell if it's a data problem, change management, or we pick the wrong use case? Adam, we'll start with you on this one. I oh, this is such a good question. I there's a nice join from the last thing you were just saying there, Mason, around, like, the bad version of RevOps is where a business thinks they need a RevOps function when they hire one, and they treat RevOps as, like, administrators and rule enforcers for CRM hygiene and, like, just kind of runners for data fetching. Right? Like, that's a bad version of RevOps. The good version of RevOps is the strategic partner, the enablers. Right? And so with pilots, I think I think you get back to the same thing, which is especially with the the old world pre AI was you roll out software, all fails on adoption. You just done a three month sales cycle and planning, and day one, crickets, no one wants it. And we've all done those rollouts. Right? I've done them. They're very painful. And you never really recover from them. A year later, you churn that contract out nine times out of 10. So it's a roundabout way of saying, like, I think if you're not getting traction organically with your revenue team, it's not solving a problem for them. Not really. Right? And. CRM is kind of like that where it's like, it doesn't solve a problem for the frontline seller to update the CRM record. So, of course, they're always it's never the top of their list to do. Right? Now the opposite of that isn't we see this all the time is the AI pilots, agents, features, whatever, that really add value, frontline sellers or leadership over, they literally drag you across the table to get them. Like, they can't get enough of them. And so I think when it comes to AI, my maybe biased opinion is, like, it'll either take off like a rocket ship or you're never gonna get it there. Perfect. Like, you should just kill it. Yeah. Mason from your side. Yeah. I think there's a there's a couple things here just to kind of piggyback on what Adam was saying. I I I think part of it is is the input from the stakeholders they're gonna be using. The the tool that you're you're thinking about implementing, like, getting the the feedback from them as to, okay, here's what we're thinking about doing. Like, how is this gonna maybe solve some of the pain that you're feeling in your role? Whether that's, you know, AE or VP of sales. So kind of understanding the how how they're going to actually use it first and then maybe using that to inform some of the use cases that define your success criteria for kinda circling all the way back to the conversation from earlier. Right? It's it's it's really important to have those kind of value drivers defined. Like, what does success look like? What is what is the point where we drop this look like? And I think that comes if you're planning on rolling this out to the organization, that that comes from the stakeholders that are gonna be using it. So it it kinda becomes both a feedback and an enablement coupled up together to make sure that that this is successful. And then and then running it like a like a an implementation of of software, kinda like I think what Adam was was getting at too. Like, making sure that your stakeholders are bought in, that you enable them properly, that you you train them up on the tools that they're using, and and make sure that it doesn't just become a I don't know how to use this problem because it it does feel like that is what kills ninety percent of the the AI pilots, and their 10% is killed by this doesn't actually work. So getting getting folks kind of enabled on how to use it, see how they can drive value based on those success criteria that they've helped you define, and then making sure that you kind of handhold them throughout the entire process, I think, is a is a great way to kinda drive to to success. You know, don't let it become a a a problem of of, hey. This isn't this isn't working. You know, fix fix that on day one. And then also kinda set the expectations, like I was talking about earlier, that 90% for an AI pilot is is actually a success. You know, nothing nothing a 100%. Very, very true. Alright. Awesome. Well, with the last few minutes, we wanted to open it up to any last q and a from the audience. I do have one question here. When we're evaluating AI vendors, how do we know if the tool will actually work with our data, not just the demo environment we're shown? And is there anything we should be asking before we sign the agreement? And either of you, feel free to hop hop hop in in here for this one. Yeah. I mean, to me, I would I would never trust a demo when it comes to AI tooling. And I say that, again, as someone who is building and selling an AI AI platform, like, you actually need to use it. No. The way we've seen the most success is, hey. Let's start with InfoSec, get through that, connect your tools, and see how it works for you. Like, that's that's the bottom line. If someone is hesitant to let you connect it and use it before you buy it, that's a huge red flag. And don't don't buy the thing. Like, try it first, use it, and see if it works. And if it takes three weeks to get through InfoSec, fine. It's it'll be worth it. It'll be worth it in the end. And as as far as the second the second point of that that question, like, what should you be asking during during the pilot? Ask ask about about anything. Ask about, you know, pricing and and tokens or credits. Ask ask about specific use cases, other success, you know, customer stories that they've they've had with with similar similar use cases. Just just dig deep with with them and, yeah, don't don't trust the demo. Use it yourself. Yep. Love that. Adam, any feedback on that one? That's perfect answer. Yeah. I is AI so easy to get to a great demo? Like, the easiest thing in the world to get a good demo. Very difficult to get a good working production system, so absolutely try it. Like, it it's just the taste test. Right? You don't know if it's gonna work until and I think even the vendor doesn't know it's gonna work in a lot of cases until you connect your data unless they're very strict around, like, hey. Where is this coming from? How is it gonna work? I think the other thing I would just ask is I'm going through that qualification process. A customer asked me this, like, hey. We're evaluating a bunch of AI vendors in go to market. How should I think about it? I was like, just take the word AI out of it. Like, forget AI. Just replace it with the word data, and imagine you're comparing data vendors or dates like, what's been stored here? What's the operations on it? That's how you compare them. And then I would ask things like like, everybody's got customer success stories. Where does this fail for people? Why doesn't this work for people? When people do your pilot, what's the main reason it doesn't work out for them? And get a good idea if you fall into that category because you might. You probably will. Right? Not everything's for everyone. So, yeah. Great answer. That's amazing. Awesome. Alright. So no q and a. Anything else left to answer from the audience? So my last question here for both of you. Well, we have four four minutes left. Favorite AI tool or favorite agentic or agent that you've built and experimented with in your day to day? Whoever wants to go first in this one too. That's such a good question. I live inside Clawd Code all day long, so I do everything there. My favorite agent has nothing to do with work. I built an agent that runs continuously. Every Friday, It scrapes all the local events near my house, like, kid friendly events, and it sends me a list of things I can do with my kids that weekend. So it's my absolute favorite agent because there's nothing worse than Saturday rolling around, and the kids have been like, what are we doing today? And I'm like, other line. Love that. That's an awesome and fun use use case. Mason, from your side. Yeah. I mean, I I use I use most of the generative AI platforms day to day. You know, I I I use Quad, ChatGPT, and and Gemini kind of individually and and often. Also, you know, I I this is, again, shameless plug for what we're doing at Alessio, but I use Alessio a lot. It it connects to all of our GTM tools, and it gives you kind of that natural language interface, but also the agentic builders. So, you know, a few of the cool agents that we've built internally are are, you know, meeting prep agents that, you know, comb your calendar and and we'll send out a a meeting prep summary kind of half an hour before we meet that you have. And we do that for the for the sales team, for our for our founders, just just to kinda prep you on prep you on meetings. It it also you know, weekly, we'll we'll go through and kind of aggregate our sales activities together. You know, how much pipeline did we create, new opportunities, things closed, and it sends it directly to to to Slack messages to the account owners to kinda just pay remind them of of what's been what's been going on. So just just quick GTM stuff that that you can do easily in Alessio. You know? Again, shameless plug. It's what we're building, and and we use it internally all the time and myself as well. So I I will say that I rarely go into into our CRM these days. I just do it all do it all with the interface. So Yeah. I'd love it. We've it looks like we did have one last question come in with two minutes left here. Do you recommend using AI based on where the data is stored or centralizing all of your AI where you load business data into the data model? Last last question here. Adam, you wanna kick it off or make Mason, you wanna take it? Whichever. There's probably a few less that I if I'm reading it right, I don't know if you're talking about server location or storage location, but I don't think it makes a difference either way. Cool. Yeah. I mean, we we we connect directly to our to our source data and and try and click keep it clean at the source and let the kind of AI cross platform context do the do the heavy lifting when it comes to aggregating the data together. So but it really like like I said, like, I don't I don't think it really matters. If you have a good, you know, Snowflake database that is very clean and and you trust as the source of truth, then then great. You know, set your. AI tools on top of that. Yeah. Perfect. Perfect. Alright. Well, we have one minute left. So thank you everybody for joining. And Adam and Mason, really awesome conversation. I think we all learned a lot. I know I did, and I think a lot of good takeaways here to put into our day to day. So if you have any other questions for any of us, I'm sure you could find us on LinkedIn. We're all available, for any other follow-up questions. But with that, we'll leave you to it, and have an awesome rest of your Wednesday. And thank you, everybody. Thanks. Bye.