Recap: Scoping, Architecting, and Delivering AI Features

Hey everybody, we are live with what is technically an AI Thursday. Most of these have been AI Fridays, that has a nice ring to it, but this week, Thursday was what we needed to do for scheduling, so welcome to our first AI Thursday, but our fifth in this series of AI discussions. Up until now, we’ve been focusing more on how to leverage AI tools, so we’ve been less technical and thinking more about how do you use existing AI tools and make the most of them.

Today, we’re getting more technical and talking how do you build AI tools, and we were just clarifying before the call kind of two big use cases that we can talk about today. One of them is you’re a software company and you’re building AI into your product. The other is you’re a professional services company and you’re building pretty robust AI to help you operate more efficiently. Quick reminder on who we are and why we have these. I’m the founder of OneGuide.

We’re a portfolio operations platform that supports 30 plus PE and VC firms and hundreds of collective portfolio companies. If you’re a portfolio exec, you may have gotten this from your investor and you probably have access to a bunch more resources through their hub. If you need help finding anything, you can also feel free to hit the help link on your investor’s portfolio resource hub and we can help you find great resources.

We also may have some in the audience who are portfolio operations value creation leaders, and so today we can talk about things both from the lens of an individual portfolio company as well as how investors should be thinking portfolio- wide. So with that, I’ll pass it over to let Sangur introduce himself. Thank you, Kate.

Hey everyone, my name is Sangur Barabasi. I’m the founder and CEO of Bonsai Labs. My background is technical. I have been in the AI machine learning space for almost 10 years now and midway turned entrepreneur.

Founded and exited a couple of businesses and now the last one where I spend all my time is with Bonsai Labs where, especially in the last 18 months, we have been working with private equity funds, some of the top 10 funds around the world and their portfolio companies to help them deploy AI and deploy AI both at B2B enterprise software businesses and also at professional service companies. All of the use cases that we have been spending our time on is mainly directed to EBITDA impact, so tangible business results.

And the pain that we are really trying to solve for these businesses is that AI adoption is really difficult to get right. Most of the portfolio companies that we met, they are stuck and they also lack the right talent and expertise to move forward. So our aim is to help these feedback portfolio businesses to nail and figure out a solid business model based on which they can lay their AI fundamentals and also foster a more sticky customer base to new AI features and capabilities for their product.

And I know that there is a huge responsibility that we are undertaking with this because ultimately, on one end, you have all the AI startups and products being built. On the other hand, you have the feedback portfolio companies. So we have this large responsibility to help translate between an extremely fast- moving world down to a value creation and EBITDA impact use case for the portfolio companies. So very excited to be here.

Awesome.

All right, so we will start to get into the meat of this. Quick reminder, one, we will have a recording of this. It’ll be available through the portal where you registered. And two, if you have questions during the conversation, please put them in the chat. We’ll also be pulling in pre- submitted questions. Thanks to everybody who submitted questions in advance. All right, over to the presentation.

Amazing. So this presentation will be fairly technical. We will get into how do you actually scope, architect, and deliver AI features within your company. We’ll skip the introduction about, as already spoke enough about that. Now, the first step where we see companies starting is actually, of course, everyone starts thinking about potential AI use cases and what can they implement within their business.

But the reality is that once they get to a couple of AI use cases, and we get into the meaty part of the conversations, then the AI readiness question suddenly pops up. And that’s when most people start asking themselves like, oh, this all sounds great. We can see the ROI, but are we ready to really get started?

So over the last 18 months, we have identified basically five key pillars that are important to assess to understand if your private equity backed or the portfolio company itself is ready to take on an AI initiative, or you should spend your time somewhere else. Because ultimately, there is a simple rule in value creation. You either want to maintain value or create value, or you do not want to disrupt value, right? So why spend time on something which will disrupt the value of the company and would waste resources?

So, the very first step is around executive alignment. And there have been over the history of technology adoption, various philosophies, which then permeated sales as well, whether we have a bottom- up adoption, a top- down adoption, right? Everyone knows these two terminologies. With AI, it’s becoming very clear that it’s a top- down adoption. And if executive alignment is missing, there is no clear executive sponsorship.

And the leadership is not showing an active initiative and communicating with the company that they are behind becoming an AI- driven organization. Then we have seen that the bottom- up adoption is mostly being blocked by the middle managers. And there is for fair reasons, honestly, because they are worried that they will lose their job. So this also entails, how do you start within the executive alignment? By starting with your AI policies, having candid communication with your team. But the most important one is to start within your executive team.

Then the next question is about your data foundations and how easy it is to access that given data. Because we have a long, long time saying in AI and machine learning, it’s garbage in, garbage out. And this is not so different for generative AI as well. It’s actually even more important. Because we have, it’s all about in generative AI, it’s about context engineering. What is the right data sources that you pull into your AI system so you can generate a reliable and high quality? And that starts with your data foundations. Where does your data lie?

And how easy it is to retrieve that data. And we meet a lot of companies who went through a series of bolt- on acquisitions. They have 20 different data sources, 20 different ERPs. And at that stage, if for a given use case, you have to integrate the 20 different data sources just to get the use case up and running for a POC, that is already way overblowing the budgets available. And it just doesn’t justify the ROI. So we always recommend for teams to look at your data foundations first.

And can you get a little more specific here? Are you usually talking like clean ERP? Are we usually talking like you have a data lake? I know it depends a ton on what you’re trying to build, but what do you see most often?

So the fastest time to value is just to get the data lake, pull in all your data, have a flattened, have your ETL pipeline, flatten out the tables, and just have one single source of truth. You can keep your 20 different ERPs across your 20 different subsidiaries, but have one single source of truth. And then the AI systems can plug into this source of truth. This is what we have seen as the quickest time to value. Because otherwise, if you want to migrate everyone off of their ERP, that will be a way overblown.

There is one caveat, though, which I will mention with data foundations, is that nevertheless, the data is important to go into production. But data foundations is not critical to do a POC and to test your hypothesis, whether there is a business case behind what you want to achieve. Because all you could do is go and manually download data from your database, get a subset of your data, build a quick prototype, and then build a business case around it.

So that also incentivizes you then to invest more in your data foundations, because you know the potential of the ROI that comes afterwards. So the impact is not always linear. Sometimes there are a couple of leading metrics in between that you can work on before you get to direct EBITDA impact.

That makes sense.

The next one, the third point, is around tech stack and integration capability. So here, from our experience so far, building POCs, we can work incredibly fast and have a POC done within even a week. The biggest challenge is integrating into the mother core. And that is because of all the legacy technologies that is in place. And we often find ourselves in a situation where nobody knows how systems work or how we should integrate. Product roadmaps are also super crammed, and you have to wait until next quarter before you can get your AI feature in.

So I think that’s also an honest assessment and conversation with the CTO of the portfolio company, whether the tech stack is at a level of maturity where we can move fast with AI integrations. If you have a lot of tech debt, it’s better to hold off, wait another quarter, and then get to it. Otherwise, with whomever you will be working with as your AI partner, everything just will be delayed.

The fourth pillar is around use case clarity and being able to formulate very simply and clearly what is the problem that you are trying to solve and what is the measurable defined success metric that you want to impact. And somehow this success metric should tie back to your EBITDA, be that it’s a top line or bottom line impact. Sometimes it’s not a direct EBITDA impact, sometimes it can be indirect. For example, if you impact your MPS score, then your EBITDA will grow.

And this is especially MPS is a big focus for the entertainment industry or B2C type businesses where higher customer satisfaction can lead to more revenue and more repeat purchases. And the final one is psychological safety. So once your executive team is behind it, you are aligned on the tech stack, data foundation, and use cases, then make sure you get your team along as well on the journey of this transformation and set out this psychological safety across the organization.

That basically giving them an answer of what will happen with my job after this AI agent gets rolled out. When we get to evaluating use cases, then this is a little bit of overlap as well with the previous one, but basically we take the same filter and then we’ll just narrow it down to one use case. So we have identified the use case, we know the EBITDA impact, and then we ask ourselves what data do we need for this use case? Is it available or not? What is the technical complexity of the given use case?

And here what we are advising portfolio companies is that AI and especially generative AI is at a state where you do not have to do R& D to derive value. There is so much value to be derived from just doing the basics, if you know how to do them right, that you should just keep doing that and let open AI and entropic and the likes to do your R& D and they will be investing significantly more than you, tens of billions every year into the R& D initiative and the models will get better over time.

So your only role is to get quickly to value and leverage what is already working, especially because it’s for some of the companies where there is high compliance requirements for various regulations, you want to keep the quality bar high, so you want to make sure that you work on the domains where generative AI has already a high bar for accuracy, because otherwise hallucination comes in place and then that can be challenging.

For the use case evaluation, also around integration and infrastructure complexity, here if it’s possible to define a bounding box that can be as detailed as possible of your legacy systems and of all the complexity that is within your business, that is the best way to go for it and try to minimize the amount of integration and interaction that is required with the legacy system. Also often, AI use cases are a great new chance to set better engineering best practices, modernize your tech stack, move from ASP. NET to a React- driven front- end.

So this is a good moment to do this and justify also the costs given the ROI. And then finally, just having clarity on the definition of success, which always should tie back to EBITDA impact. No, sorry, we… then there’s an interesting question. Oh yes, the final one here, and this is the final point, which is just a little call- out at the bottom. How do you estimate technical complexity?

A very simple 10- minute test to test if generative AI is ready for your use case is just, as I said, download the data from your databases, put the data in chat GPT, do a little bit of prompt engineering for 10 minutes, and then see what is the quality of the output. If the quality of the output is not bad, I think this is workable and we can just work on it and make it better, you should continue. If the quality of the output is absolutely terrible, then you are likely hitting R& D territory.

A stupid question on this 10- minute test. Is the best way, so if I want to give it a bunch of data, I’m going to download data, upload it into ChatGPT. Should I just do that in a regular conversation or should I create a custom GPT? Just regular conversation. Okay, cool. Then here’s a question that I find, it’s an interesting one. What are the top three AI use cases that have generated the most ROI for your clients? Getting this a little bit more specific, can you share a couple of examples that passed the use case criteria?

Yeah, absolutely. I think some of the no- brainers right now in the market are embedding AI into your marketing and sales teams, so augmenting sales efficiency, or also improving content generation. These are no- brainer use cases. Save a ton of time for your teams. And also often you can save costs from reducing reliance on external agencies and partners, because now your team can get it all done internally.

And when you talk about that, are you talking, hey, if you’re doing content, go subscribe to Right, or if you’re doing sales, if you’re doing ABM, go subscribe to Sixth Sense? Is it buying stuff or is it building stuff?

It’s mainly in sales and marketing is off the shelf.

Yes.

And if you want to go a little bit more advanced, like you shouldn’t, let’s say, you can either get Writer for content writing, right? Or you could just use JGPT and then… But predominantly these tools are off the shelf.

Yeah. Okay, so I guess maybe, what are some other use cases that, you know, where you get engaged, where it’s worth the spend of some more custom development?

Totally.

So an area where we spend on custom development is around customer support. And this is surprising to some because there are so many off the shelf solutions. But the reality is that there is a large pool of companies out there who have their own bespoke customer support setup. And five, eight years ago, they moved off of Zendesk because they were very unhappy with that platform. And now they are stuck with their own internal build customer support platform.

So for them, building workflows and automating the response to tickets and also augmenting their internal team can lead to between six to seven figure ROI in terms of cost saving. Then we also have implemented voice AI, especially around automating hiring processes for companies. And the biggest success we had, this was in the blue collar industry. And this company had to interview about 10, 000 people every single month just to get to hire 1, 000 people at the end of the month. And in blue collar, the candidates are not tech savvy.

They are not on LinkedIn, not constantly on email. They are not used to Google meets and Zoom meets. So the best way to get a hold of them is to call them. So we work with this team where they had the 40 FTEs. And the 40 FTEs, they were each spending six hours a day cold calling candidates in order to get them on a phone interview before they can qualify them and move them on. And that was very inefficient. So we implemented voice AI. This was actually a mix between NA10 as a workflow automation.

And on top, we used off the shelf voice AI infrastructure. And the workflow was basically calling 10, 000 people every single month, asking them a series of qualifying questions. And if they were all passing through those questions, then scheduling in a virtual call as an interview. And the unlock was we lowered basically time to hire by 50% for the company. The HR team now wasn’t spending any time cold calling. They were all day in back- to- back meetings, their productivity 3x.

And if we nailed down the day rate for each employee compared to also they are hired 50% faster, at the end of the day, we were unlocking a seven- figure extra revenue for the company every single month. So that is one use case, which is, I would say, half bespoke and half off- the- shelf tooling, because you shouldn’t build your voice AI telephony infrastructure. And then the ultimate layer, these are the biggest impacts.

And this usually we recommend to companies who have already went through the initial off- the- shelf marketing and sales, and you have some confidence. Then you get some NA10 workflows plus voice AI. Then the next layer is for software companies, especially how can you build new products? How can you invent new services which you can deliver with AI? So here we have been working already with two B2B enterprise customers.

And for them, we have discovered the new customer pain point, which they haven’t been solving before, then came up with a product proposition, validated this product proposition to POCs. And then we launched the product within six months. And because this private equity- backed portfolio company already had distribution, and this is why we love working with POC companies, by that sixth month, we were at multi seven- figure ARR, which, if you do the multiple on enterprise software companies, then you get into eight- figure EV expansions.

And now they also have a more sticky customer base. The valuation of the company is much higher. They have new IP, and they have expanded their product suite and product ecosystem beyond their core product that they have initially.

Thank you. I appreciate those tactical examples. They really help bring sort of this, what could AI do to life?

Totally.

All right, we can get back to the back to the plan.

Yes, happy to move on. Well, as next one, how do you decide between different implementation approaches? And here you could have approaches as we have just mentioned, so you have fully off the shelf solution, you have a mix, and then you have fully bespoke. Off the shelf solution there, the biggest decision factor is especially, I keep referencing back to PE companies, there is so much legacy going on in these businesses.

So many processes and ways of working that are really hard to change unless you want to spend time on change management for the next six to 12 months. So the biggest question you should ask yourself that if there is an off the shelf tool, can this easily fit into my existing processes? And if the answer is yes, you should always go with that tool, because there is a company which will spend significantly more money on R& D and improving the technology than you could.

Then the second part is around having a mix between having an orchestration layer, which is a no code, no code solution, like you could take N8n and Zapier, and then building custom integrations on top. And here, the analogy that we use is that most companies start off with buying chat GPT subscription and giving it to all their employees, but that is like a single player mode, right? Like you ask a question, get an answer, but your job day to day is rarely single player.

You will go to various applications, fetch different data, put together an NDA, put together a sales proposal, a new document, so your work is much more complex.

And if you truly want to automate end to end processes, the first step is you need an underlying orchestration layer, which can plug into all your existing applications.

And once you plug into all your existing applications, then you can very easily build these workflows where you fetch data from five different sources, then you can implement an open AI integration in the middle, which is like an agentic behavior, which can take decisions, can assess data, can come up with potential next steps.

And that is how we have seen companies automate, especially back office processes.

And when you do custom AI innovation, that we mainly recommend for software companies, and they are the biggest factor is who will take over the custom software, who will maintain it.

And if you have your internal team, or you are ready to hire AI and machine learning engineers to maintain that, that is great, you should go for it. And if you also feel that this will increase the value of the company, and you will have more IP and ownership, that’s the path to go.

I really like this breakdown. These three levels make a ton of sense to me.

Now moving on, just a quick comparison between low code, no code.

We have this APRs, the N8Ns versus custom development.

With low code, no code, you have a very fast setup. So these tools already have thousands of integrations out of the box, they can integrate into your Google Drive, into your Notion, into your Gmail, Outlook, whatever tool you are using, most of it is already integrated.

So you are saving a lot of time. Like today, no one should spend time building a Google Drive integration, in my opinion, you should just really spend time in what’s high value for your company.

You also have a lot of flexibility.

It’s cost effective.

So in terms of economics, I mean, with N8N, you can automate over 10, 000 runs, 10, 000 task executions for about $ 100 a month. So it’s really, really cheap. And you can also very quickly prototype. So you could get to value even within one day, if you have full clarity, and you are integrated with your existing systems, within one day, you can drag and drop end- to- end workflow and see initial results. Then of course, taking it to production is a little bit of a different game. And you have a couple of requirements. But again, it’s doable.

And here one caveat, which I would say, we are working very closely with N8N. We are their certified partners.

And the reason why we went with them is because for companies in highly regulated industries, they want to maintain the data privacy and keep everything within their own environment. So with N8N, you can self- host.

and then you can basically abide by your compliance and regulatory requirements. On the other end, when you get into custom development, and here the most common tech stacks we have seen are of course Python, because it’s very favored in the data science machine learning ecosystem, but then also JavaScript and TypeScript is taking off massively, but here you have full control and you can also implement much more advanced observability and monitoring of your systems. You can detect issues way faster than your customers would.

And also scalability- wise, I mean, that’s all in your control and you can scale to the hundreds of millions of customers if you have those many at your company. One thing I would also mention, as I’m sure many from the audience are aware, that AI coding tools are taking off. So even if you go for a bespoke development, building a prototype, especially with generative AI, shouldn’t take months.

Even if you go for bespoke development, you can build prototypes in Python or in JavaScript and TypeScript very, very fast to prove the business case and only afterwards get into productionizing it.

I think that’s also very good advice. And I didn’t know that you guys were an end- to- end partner. I honestly thought that you were living almost exclusively in this custom development column. It’s interesting and makes sense to hear that, hey, you actually also do a lot of low- code, no- code, because why would you start with something more complex than you need?

What we have observed over time is that we have overestimated the AI maturity of companies. And over time, we have learned that some companies are way earlier in the journey. They would like to get the value faster. Also, it’s much easier to prioritize a back- office use case, which is not impacting customer relationship. So then we thought about what is the fastest way that we can get our customers to value.

And I think you guys share this belief as well, that the old way of consulting, where we have to sell you a bunch of people because they are all on our bench, and we have to do a six- month strategy consulting, that is over. So for us, it was always about how can we get to value in the quickest time possible? If we could get to value within one day, sure, that’s the best scenario both for us and both for our customers.

And that’s actually a really good segue into this question that I wanted to get to. Hey, everybody’s seen this MIT survey floating around that many AI implementations fail to generate ROI. One part based on this conversation we’re having now that I think is really important is what is the I? It’s easier to have a high ROI if the investment is low. I mean, if you do something lighter weight, is that part of the problem in your view that people are just investing too much without testing enough and validating?

Great question. I think there are at least two problems, maybe three. So the first one is that companies lack clarity on their AI use case. They just haven’t determined ahead of time what is the ROI they are expecting and how does it tie back again to EBITDA impact? We have met countless leadership teams who want to build an AI agent just because it sounds super cool and it can change the company narrative. But when we dig down to the details, there is the lacking ROI behind it. And that’s like a failure from the get- go.

You didn’t even start and you failed because ultimately everyone will be unhappy. So that is the first problem. The second problem, which is tying back to what I was mentioning around the incentive mechanisms that exist in the traditional consulting firms. And the incentive isn’t to get to value as quickly as possible. The incentive is around how can we charge for more headcount? How can we charge for more time? And how can we maximize the budget that we can earn from our clients? But the dynamics now are changing.

So you don’t need a six- month engagement to build a POC in regenerative AI. You can do this in a couple of weeks. So after a couple of weeks, you should know already if the use case is proceeding into production and is worth the investment, or you should just cut it short and move on to the next one. And based on that POC phase, you should be able to very accurately determine the ROI impact. So one example we have worked on was in the publishing industry, like audiobook publishing.

And there we were working in the POC phase and the ROI calculation was very clear. We look at how much time does it take today for your team members to execute, I will keep this high level, but to execute a given task within the preparation phase of audiobook recording.

Let’s say that takes one hour, great. Then our hypothesis is through the POC phase, can we decrease that one hour to 15 minutes or 20 minutes? And we kept this, let’s say, higher. So we weren’t ambitious to like one minute because we knew that we were processing PDFs with hundreds of pages. And that takes time with LLM. So just let’s get it down to 15, 20 minutes. And then you can do the simple math of how many books are you processing every single year? What is the cost per employee per hour? And then you just do the math.

If the ROI makes sense, then you will invest. Otherwise, just cut it short.

I think that is really good guidance. Okay, I’m waylaying us. We might not get to every slide. I’ll let you keep going and feel free to… Oh, no, we will. We will make it through.

We

I will be a bit quicker in a couple of minutes. So one of the use cases that we see very often, and this is becoming nowadays also table stakes where companies can start is around unlocking unstructured data. And over 90% of our enterprise data is in unstructured form. Is it like PDFs, meeting recordings, audio files, logs, videos. And so far, we haven’t really made use of it. We only were able to take our product analytics, our sales numbers, and have a very, let’s say, yeah, only stick to our structured data.

And now with LLMs, you can actually extract meaning and extract metadata from all this data lying around. And one of the most stable stakes we see is just establishing a very simple document ingestion pipeline where you can process all the documents, all the unstructured data across your organization, extract what are the relevant features for your use cases, and then store that in a structured format in your SQL database. And then let your data science team extract insights and do their magic.

And here it’s very critical collaborate with domain experts. So for example, we have done work in the legal tech industry. We have collaborated with this team who has millions of documents in their database, and we were legal professionals. So we had to work very closely with the legal team of the company with the product managers who have been ex- lawyers, and together with them to define the requirements. This was a collaborative back and forth process between engineers and the domain experts.

And some of the tools that you can use to process your data are Mistral OCR. For example, this is one of the leading OCR models out there. Here you could pass in either your digital PDFs or your scanned PDFs, and it extracts data with very high accuracy. It also maintains the structuring of your data. So for example, if you process invoices and you want to extract line items, it’s working really well. We have achieved over 90% accuracy with this model. Alternatively, you can have simple text extraction from digital PDFs.

And when it comes to audio content, then you have speech to text models where you can convert your audio content into text. So companies like DeepGram offer this kind of solutions, and they also support multiple languages. And I’ve realized that I didn’t put it on the slide, but for video processing as well, there are various techniques how you could make use of your video. The most simple one, if the video is very conversation heavy, it’s just take the audio of the video and pass it into a speech to text solution like DeepGram.

They are so advanced, they will also detect multiple speakers and label multiple speakers so you have access to all the data. And there are also solutions like 12Labs, which we have been using across a couple of projects, which do a mix between analyzing the transcript of the video, but then also looking every couple of frames, almost like one frame per second, and then passing that screenshot of a frame into an LLM, asking the LLM to describe what is in the frame and merge the description with the transcript at hand.

And that’s how it becomes ultimately like a summary of one second. And then you have the holistic summary of the whole video. So this way you can, for example, search across your video library, and you can also extract deeper insights that merge the transcript with the visual.

And if you don’t need to connect data from different sources, let’s say you just wanna have a really good way to search your video library. That could be kind of a standalone thing, and maybe you don’t have to build as much data infrastructure to make that happen.

You can just take 12Labs off the shelf for that. Yes. So if you have a bunch of videos, you can just take 12Labs off the shelf and upload all your videos, and they offer you an interface where you can- But if those videos are connected, if there are recordings of customer calls and you wanna connect them to your CRM, then we have to start thinking more about good data hygiene and data infrastructure.

Absolutely, absolutely. But these companies at this point, they, at least I positively tend to believe, they are responsible companies and there are policies in place about data handling.

I’m just worried about the garbage in, garbage out part. If you want to have rich insights where you’re tying together both something from the video and something from your CRM and something from another tool, then we get back to the problem of, cool, we got to get the data into a good, clean, centralized spot.

Understood what you refer to. Yes, you are totally right. And that gets into a little bit of a custom- built territory, because you are mixing multiple sources and multiple products into one. Yeah, awesome. How should you approach data architecture considerations? And this is especially a question…

I’m getting ahead to the next question, exactly, yeah. Totally.

So here the biggest question is if your data is in place and it lives in many different tools. There are already multiple data lake solutions out there or newly from Microsoft is promoting data lake house solutions like Fabric, which have both the data ingestion and the transformation within one tool, which is ultimately what we need. So that’s what we recommend to most of our clients. Just pick an off- the- shelf tool.

You have a lot of out- of- the- box connectors that can also pull in your Shopify data, your NetSuite data, your Google Analytics data, or your Google Drive data, whatever it would be. And then once you have it all in your lake house, then you can do your AI processing on top and process your data, vectorize… the vector database, and so on.

When it comes to how does a, let’s say, an AI workflow architecture look like, and this AI workflow architecture is especially relevant as we see it for professional service companies and back- office automation in general. So the first step is that you have to choose your cloud, and first your cloud provider and your workflow orchestration platform. So we have mentioned NA10 already, which can be a workflow orchestration platform. And then if you are tied to a cloud provider, like you could host on AWS, on Azure, on GCP, it’s up to you.

Then with regards to which AI model you commit to, the nice thing about lately the AI models is that all are fantastic. And if you write a great prompt in one and then you just want to migrate to another, most of the time they work equally well. So it’s more about committing to one which you see that has the best performance for your given use case. Then the third component is vector database. So vector database is the place where all the data that is relevant for your large language model and for answer generation will be living.

And then the fourth one, application framework, this is optional because if you do it all on NA10, then you have everything in NA10 and you do not have to worry about an application framework. Otherwise, if you decide to do custom build, then in application framework, you could use, of course, a Python and a fast API library as a backend. And then we use LangChain, for example, as an easy orchestration across multiple large language models, multiple vector databases.

Though here, one thing which I would mention is that these frameworks like LangChain, they evolve so fast and they change so quickly because they also try to keep up with a change in the market. That is, it’s recommended to not over commit and over rely on these systems. Really be thoughtful on where you are using them, at which I would call them like choke points in your application, like connection to the vector database, connection to the LLMs, but all the rest, it can be just pure Python code and it will be much more easier to maintain long term.

And a relevant pre- submitted question here, what architectures and orchestration tools are best if you want to support secure multi- tenant deployments where models interact with sensitive customer data?

Well, of course, like there, LengChain is mentioned. LengChain is just a Python library, so if you build your custom Python backend service, then you are independent and you can also host it in a secure multi- tenant setup. The other one, if let’s say, if you go with NA10, again, you can host NA10 for various clients. LengFuse as well was mentioned there. We use LengFuse a lot for evaluation and for monitoring and for traceability, which is very important to know how your AI models perform. Again, it’s a great tool because you can self- host it.

It has an open source version. So ultimately, as you can see, to this question, the answer is look for the open source, because the open source is equally as powerful today as the closed source. So we always look for the open source tools that are top of the leaderboards, and we are self- hosting them in our own infrastructure and then making sure that data is not being leaked.

And we also have some guardrails in place for, for example, PII data filtering and making sure that even though it’s self- hosted, sure, but we still don’t want to leak PII data into the tools. So we do a couple of filters and checks before sending the data to an LLM or to any of these tools. I think we have just covered Langfields, so I’m happy to jump over this.

When it comes to integrating AI into operational systems, and here we are talking more about you have an internal system that you are using for an existing workflow, and you want AI, the generative AI or the agents to use this application. So what are the ways that this AI agent could fetch data or put write back data into those systems? And currently we have three approaches that we have been seeing most common. The first one is an API integration approach. Of course, this is most efficient, most reliable, battle- tested.

So if the given tool at hand has an API interface and you can like read data from it and write back data to an API code, you should also do that. Then we also have another, I will just jump to the right, with MCP integration. The way to think about MCP integration is that every tool out there has a custom API, which is bespoke to their own tool. MCP is another layer on top of all APIs, so it creates a one single unified layer that is easy to interpret for AI agents or for large language models.

So there is not a large difference between API first and MCP integration approach. The only caveat is that with MCP integrations there are still some security concerns and issues that we have seen in the market. So often it doesn’t pass by the CIO or the CISO review. So yes, that is a consideration to keep in mind. And finally, if these external tools do not have any way to connect to them, I think the times where you maintain your market share as a tool because you have a closed ecosystem, I think it’s over.

Especially because of the OpenAI agent mode type tools. And these are basically browser automations and agentic systems that can go on the browser or even on your Microsoft desktop environment and navigate through the application, figure out what’s the next best button to click, and download and extract information. And basically they use the software the same way as you would. And I think that is a big next unlock towards companies easily integrating with their own existing systems if there is no API integration.

Or sometimes it will also open up the opportunity of switching from existing tools to new tools, because you could use a browser automation as a migration from one system to another. When it comes to integrating AI into customer- facing products, here there are a couple considerations to have. Of course, in terms of technical architecture, having a microservice architecture, being ready to scale together with your customer, because sometimes customer behavior can be very spiky. That is key.

It depends whether you are a cloud- first company or you also have on- premise requirements and on- premise deployments. So cloud- first is easier to pursue because the vendor takes care of scalability, but it can also serve as a choke point. And there are some clients that we have met who want the solution to be on- premise, basically. And in those cases you can self- host local LLMs.

I realize there is one more thing which I haven’t added to this slide, which is around when you integrate AI into customer- facing products, companies expose themselves to a lot of potential scrutiny and risk. So the methodology to develop these AI products, we call them evaluation- driven development, where what you want to do is, ahead of time, set up an evaluation dataset where you have basically the potential customer input and the expected output, and then you can measure the accuracy of your AI system against that dataset.

And you should always have a threshold of when you are ready to launch, or…

and what’s the minimum accuracy threshold that has to be passed. Another approach as well that we see to integrating AI into customer- facing products is around starting first with a co- pilot approach and then moving to a fully agentic approach. So what I mean here is that you won’t just build an AI agent and launch it out into the wild and just let it do its thing and interact with the customers.

Let’s say if you have a customer support team, in the beginning you will just integrate it behind the scenes to augment your customer support operations, like have pre- written emails or pre- written replies. And over time you can monitor how often your customer support agents are accepting those pre- written emails and replies. And when the acceptance rate passes a desired threshold, let’s say over 90%, that gives you a good sense that, okay, these agent recommendations are now good enough to go out into production.

And then you move from co- pilot into agentic mode. And at that point, the goal is that all customer support tickets would be sold by the agent. Whatever cannot be sold is deflected back to the human agent. Awesome. A common question that we ask is, what is the right team that we need to deliver these AI products? I would start with a high- level theme that delivering AI products is not only about having AI and machine learning engineers. There is a huge race and everyone wants to hire them and the headlines are all out there.

And they’re expensive. I mean, all of the headlines are out there.

They are very expensive, yes. But the reality is that you can pay an AI engineer a lot of money. They build the best, the shiniest product. And it could still, all on flat ears, have zero impact on your company and just be lost in the POC abyss. So you really have to think about how you are merging and keeping a synergy between your business model evolution, which especially focuses on which customer pain points are you solving, how you are aligning your go- to- market motion. What is the pricing strategy that you want to have for your product?

And having all this together with the AI engineering and the product engineering and development. And only this cross- functional team, I believe, can deliver successful outcomes. So here we have outlined these two responsibilities. So for us, we have product managers who are responsible for establishing the product vision, the product roadmap. And the product managers usually are from senior towards director level.

And they set the high- level vision, get the stakeholder alignment, get the board and exec buy- in, and set out what is the roadmap that we have to achieve. And then you have, on the other end, the engineers whose responsibility is to deliver very quickly and scale the solution into production. And this cross- functional setup can help.

Where does prompt engineering typically lie? Because I could imagine both sides of this table working a lot on that.

Yes, I would say mainly it lies with the AI engineer at this point. And for us, it’s also very clear that AI engineers work closely with domain experts or product managers. But it’s mainly owned by the engineers and with input from the domain experts. And the domain expert comes usually from the company side. So it’s not even the product manager.

We had a question on – I’m going to extend this question. So should you focus on upskilling internal talent or hiring AI- savvy external talent? And in that bucket of external talent, how should you think about hiring full- time expensive AI engineers versus bringing in contractors or agencies that can help on a temporary basis? So I guess you’ve got three levers you can pull. You can train your existing team. You can hire full- time new people, or you can bring in contractors. How should you think about those three levers?

Yeah, great question. I’m focusing on not giving a biased answer here. And I know that you do provide services. Yes, yeah. I think what we see the most common, and we realize that it actually also makes sense, like the reality is that you can build these POCs also if you are an exceptional backend engineer. But with the POC, there is a big difference between POC and production.

So what we see companies do is that there will be a couple of champions within the company who will step up and they will want to work on this exciting project and their internal team will be able to build this POC. And that is great. They will hit challenges taking the POC to production. And they will be lacking skills like how do you set up the right evaluation metrics and evaluation methodologies because maybe they are not coming from a data science and machine learning background.

That is why also I believe that people say that JAI engineers are just like backend engineers and now doing some open integration.

I think that is wrong and that is where POCs fail.

The best engineers have a machine learning background and they apply their data science and machine learning head to these new problems to solve.

So that’s how we see that once POC is done internal team, then they get stuck.

They get an external team to help take it into production. And then the external team also helps them to train up the internal team how to better maintain and how to better continue developing the product. So ultimately, yes, it’s in the best interest of any company to end up retraining their external teams because there are just not enough AI engineers in the world at this moment. But in between, it’s great to rely on experts who can help shorten the learning curve.

You don’t want to build a chatbot for six quarters. This is one of the lines I just heard from a potential client a couple of days ago. We have a roadmap to build a chatbot in six quarters.

I don’t think you should be doing that.

We can build a chatbot in six weeks, not six quarters. So that’s how I would balance this.

Awesome.

So how does discovery to deployment look like? And this especially applies to automating back office or also for professional service companies.

So this is not custom bespoke software development. This is where you have existing business processes, and then you want to automate those business processes, like, for example, the audio processing.

And the way to do that, you can literally, in one week’s print model, get to an automated process by the end of the week. And that looks like first two days, product manager, together with domain experts, department heads, or whoever is owning the process, scope out the given process, understand the requirements, have a clear process map, and make sure you are all aligned, and also pressure test if that process is really efficient or there should be some changes.

Day three and four, the engineer, AI, machine learning engineer, or backend engineer can focus on automating the process. If you use NA10, you have out of the box a lot of integrations already, which you can plug in.

And then by day five, you should be showing a demo to the main stakeholders and showing how this process can be automated and be much more efficient.

But this doesn’t mean you are in production by day five.

This means by day five, you have proven the business case. If you do this with a custom build, that is a bit longer. That is about two to four weeks.

And the way it looks like there is that the first week, you spend it on scoping and validations.

Week two to four, you spend it on development.

Week two and three, you spend it on development. Week four, you spend on iterations.

So by the end of week four, you show the demo to the main stakeholders.

And if that is approved, then you go into beta and then to production.

A couple final points.

I think we are coming to the end. Perfect timing too.

So success drivers is first and foremost, understand the problem that you are solving and have a super, super clear understanding of what is the KPI that you are working towards. And some common pitfalls that we see.

The biggest pitfall is that really the biggest risk that you can have as a company right now is inaction.

There is a lot of noise.

I totally recognize that. There are a lot of opportunities to be chasing.

There are a lot of voices and experts around you who claim to be experts.

And you can end up in this endless discussion and strategy. The best strategy is just very quick execution and learning. It’s not about having a consulting firm who comes in for three months and tells you what you should be doing. By the end of those three months, OpenAI will release three new versions. And your whole plan will be thrown out and you will have to rethink it. So just get going, start learning and iterating. This is how you can get ahead of competition.

And also making sure that your resources are smartly allocated. So don’t throw around with large budgets. You still should be frugal, cost effective.

You don’t need gigantic engineering teams to deliver those AI POCs.

Just one great engineer and a product manager can do wonders for this company.

Yeah, so we’ll finish up with a couple of pre- submitted questions we didn’t get to, and then if anybody wants to sneak in a question or two in the chat, we can weave those in as well. So one that we flagged but didn’t get to is, when you embed product features that have a consumption cost, how do you control cost while still delivering value at scale?

Yeah, that’s a great question, and that is always coming up. I think the biggest mistake that we see is that executives just never get down to the bone of, like, really, how much does it cost? And working together with the engineers to tell them, tell me exactly how many tokens are you consuming for one call, and tell me exactly, down to the dollar, how much does this cost per one month? So that’s the first one, just understand your basics. For every single AI interaction, every single click within your product, what is the cost?

And then the second part is around, you will not get it right from the beginning. AI pricing is an ongoing refinement. So when you are in a beta phase and you launch for the first time for your first couple of customers, you should be prepared to lose money, but then you should also be prepared to tell your customers up front that this is an initial pricing, this might change as we learn. And then observe your customers’ usage, and then you can keep refining your pricing tiers based on the usage patterns within those cohorts.

We actually did a whole session on pricing AI products a couple weeks ago. You can look that up in the portal. One nugget from that session that I found really interesting is that AI and consumption costs that go along with AI have a lot of people thinking in software about cost- based pricing. That’s a trap. In most software companies, if the gross margin is staying above 50%, you should not be thinking about cost- based pricing. You should stay entirely in the world of value- based willingness to pay pricing.

And so another answer to this might be, it might not matter. You should charge them as much as they’re willing to pay, irrespective of how much it’s costing you. That’s a very good point. Another interesting thought to go check out.

Ultimately, it’s about positioning, right? If you can position it against headcount costs, that is your best positioning to have, because the base is just so high.

Here’s another interesting one, cybersecurity concerns. When assessing AI readiness in a cybersecurity SaaS environment, how do you balance innovation speed with data privacy, compliance, and model transparency?

That’s a great question. I would say data privacy and compliance requirements, they just become table stakes. So what we have seen, and we have worked closely with a bank who started with establishing an AI policy. So even before you get to development, as you establish an AI policy, you establish an AI committee.

And the role of this AI committee is to review any, for example, review every model that the engineering team wants to introduce, review their data privacy and terms and conditions and all the documents and make sure that the board of the AI committee approves those use cases. This is a very simple way to start. Afterwards, if you get to the more practical, like hands- on development, then it’s about evaluation- driven development. We have mentioned this earlier.

And that is the easiest way to know exactly the performance of your system and where you stand on the scale of this accuracy. When it comes to model transparency, I think that is a field which is still in research and development. So we do not have full transparency around why these models are making certain decisions. And they also sometimes hallucinate.

One small trick, but of course, this is more of a trick like, I don’t think this is the solution for highly compliant and regulated industries, but at least what you could do is in your prompt for every single generation. Also mention that before you output your answer, output your reasoning behind the answer. So at least that is the model transparency that you can get into of why the AI made certain choices or gave certain answers.

Awesome. And we are at time. We will share the recap with everybody who registered, including your contact information. So if anybody is looking for help with AI projects, Bonsai Labs is one of the options that you can put in the mix. Thanks so much for your time.

Thank you as well, Kate.