AI Software Testing vs Hiring More QA Testers for Web Apps

**[00:00:00]** Hello everyone, my name is Armando Perez-Carreno and welcome to the Web Talk Show. Today with us is Pramin from BotGauge. How are you Pramin? I'm good. So thanks for inviting me for this podcast. Pleasure to be here. Yeah, it's a pleasure to have you. I think what you guys are doing in BotGauge is very interesting and I think people will very happy to hear what's going on a little bit of behind the scenes of of this space. So before we get into it, why don't you tell people a little bit about BotGauge and then also how you got into this yourself. Right. Right. Uh so I did uh my masters from IIT Kpur. Uh it's a premium college in India. Uh so after that I was looking for okay how should I start my career. So fortunately I end up in an uh startup which is a new test automation. I was the sixth employee there founding PM. So there I've seen the journey from 0 to 3 million. Later it got acquired by another company called S labs. So in S labs uh I was leading the low code division there. So from a PM side of it. uh so that gives me an exposure to what the problem statement right of how difficult the test automation is or how difficult to is to achieve more than 90% of test coverage. So during that time uh the LLA started coming in that's how it's like me and my colleagues decided to come together and decide to start B the vision of how can we have an increased test increased test coverage in a shorter period of time so that the companies can release faster with confidence. So that's how we came into this particular area of test automation. So more than a decade I spent in this particular field from a product management side also. **[00:02:00]** Nice. Nice. Well and it's it's a an under talked area right where where all the automation all the LMS yeah we can do things very fast but right for pilots and demos it's all very easy. The thing is when you get into the real work and that you have to make sure that things actually give the results that are valid and accurate and verifiable, that's where it gets tricky. So automated tests are super important and they're not talked about enough. Yeah. Right. That's fine. See, in this era, right, development has become much faster, right? As you know the coding or there is lot of coding enabled agents where any developer can develop the components much faster. How when this component get integrated into the product ecosystem or the production uh production ecosystem right then we have to test it end to end not only for that particular component. So BotGauge is hyperfocused on that end to end UI functional API side of it plus the agentic testing as well. So we test the agents too. So that's what we are hyperfocused on. How can we make sure the coverage is more than 90%age and it the speed is in par with the developer because QA is a current bottleneck for any uh SAS companies to release functionality into production. Development has become seamless. However, without releasing into production, they cannot establish the product feedback loop also. So, so we come in a very critical ecosystem u but it's a boring ecosystem as well because it's a lot of repeated testing have to be done. Yeah. Yes. Yeah. Especially the the UI part like end to end UI since it's visual like you said all the coding becomes **[00:04:00]** trivial right to a point but but then the UI part and making sure like the click journeys and where it takes you and that you don't just suddenly get dropped off where you shouldn't and that you don't accidentally trigger a thing that breaks everything like people don't do enough testing about that. So it's it's very interesting that you got into this part. Uh so how we do things right uh in bot cage we deploy the agents into the uh customer ecosystem like identification generation agents then execution as well as the maintenance set of agents and plus and forward deploy engineer to monitor these agents whether they're doing it right or not. So as a campaign package we deliver this to customer in an outcome oriented manner. Outcome is nothing but uh the coverage right. So coverage in a shorter period of time and maintaining the test cases throughout their life cycle with all the release incorporations from a weekly stand point of view or maybe it depends on the release cycle assess. So what the customer gets out of it it's like they would have tried for 6 or 8 months and they would have reached only 45%age coverage here in 3 weeks they are able to reach more than 80% coverage and there is also a cost reduction of more than 40%age. So for a lesser cost and increased productivity is what BotGauge offer because of this campaign package right which we provide to them. Yeah. Interesting. And so is a lot of this managed like the QA part of it it's is it managed by agents right it's agent first of all our deployed engineer right that's the model which you are following so it gives them the right productivity access so it's a highly value based outcome oriented model which you're following you would have heard about the term service access software which **[00:06:00]** is a little bit different from that SAS ecosystem where right now the LLMs are able to provide output but there should be a human in the loop right uh so the 80% of heavy lifting is done by the agents but there's a 20%age of uh human involvement when we package together then it becomes like a service as a software model okay that's a good way to do it I there is a continued rise of hype for a lot of these agents and tools and and uh AI and all this and once you start actually deploying it in production in companies you start seeing all the little edge cases and issues that appear and there's not enough testing going on and manual testing traditionally does not cut it as as you explained and and as you know by because you you need a lot of people and they all have to be on board and then there's miscommunications and there's time zone differences and there's all sorts of things that come into traditional QA. Right. Right. Where if that gets compounded with if now the AIS are building and deploying then there's a lot more things to test and now you have even less resources. So if your agent is building it, you need agents to actually do the test otherwise you can't catch up. Yeah. Yeah. So it's like a full-fledged QA layer, right? For the I would say that AI era at this point. So that has become more demanding at this point. So that's the hyperfocus area which we are into. Uh unit testing if taken care by the development side itself right now. They usually developers take care of it. Right now it has become very seamless. It's like component testing also has become seamless. **[00:08:00]** But now what has become more difficult is a end to end behavioral user mimic testing right that has taken more prominent right from well the example you gave about the the cannot complete the form. It's it's so easy for it to happen. It happens all the time because we interact with forms. Uh the other day, I don't know, we were I don't know if we were purchasing like a studio session or something something that we were doing where the form wouldn't let you choose what you needed to choose and then sometimes like the option doesn't exist the right one and then you have to choose another one and that becomes a mess downstream or sometimes you just can't continue but it is literally a mistake because the slug was different and somebody changed it and nobody noticed and if nobody's testing it becomes a problem. But with you can have these tests just continuously running and making sure that everything still works. The same way you have a heartbeat on a website just to see that it's up, you can have the heartbeat on every thing and and make sure. Okay, this is this is very interesting. How does it apply to any type of software or business or is it are you focused on specific niches where's your see we are we are sector agnostic uh and so but however we are hyperfocused on the web based applications so any application which can be open in a browser that's what we are more focused as well so we We are not into native mobile app testing at this point which we'll eventually go into but right now it's more all the web based application. Okay, that makes sense. When you talk about the forward deploy engineer, a lot of our listeners might not **[00:10:00]** know what that is. Could you explain a little bit about what that is in the terms of in general because we've been hearing it a lot more especially with anthropic and open AI creating um the de the deployment company and then these huge initiatives of doing forward deployed engineers into companies but still I think many of us don't really know what what that entails so can can you explain a little bit more about that that position that role how it differentiates itself from a consultant, right? Uh what's what's what is uh forward deploy engineer and how does it fit into what you are doing? Right? The forward deployed engineer is an uh like different companies use different contextual into it, right? But think about it like an uh AI skilled, right? who has more knowledge about the agents who comes into the picture with the customer success role about understanding what the customers uh is expecting and what kind of finetuning has to be done in that particular ecosystem. So it's a mixture of uh development skill, AI skill set plus the customer acument of resolving their problem statement. Right? So this is what is it a forward deployed engineer is called but people use it for different contextualization. However, in our context, forward deployed engineer is nothing but a domain or knowledge specialist with the AI skill set and who has customer equipment is deployed with the agents so that they can monitor agent whether they are doing it right, they have the right skill set of the domain. For example, if it is healthcare tech or if it is e-commerce. So there are different acronym played. So **[00:12:00]** that expertise is incorporated with the three skills set which I mentioned right that is the definition of forward deployed engineer in BotGauge to now it it seems logical but but still uh it's at least before it it wasn't necessarily the the most common approach. So it But if you think about it from a business standpoint, it's super helpful because you have someone that you can talk to that's close to the action, And they can see what's happening. They can make adjustments. They can consider if a change is valid or or not. Sometimes the somebody wants to make a change and and you ask the right questions and you figure out it makes no sense. It was very a very obscure edge case that never happens again. you don't want to modify the whole thing and that yeah that is that is super and I and I think it aligns with also the AI champions right there's there's this whole concept now as well of you when you can and and you need like for this QA part it's super important you have that that forward deploy engineer um who who sort of becomes part of of your ecosystem and in some companies you need to have that sort of same structure but maybe it's a larger company and you want it for all different initiatives. Then you have an AI champion in your own company that is trained to to like you were saying have that domain knowledge but also have that AI side and be able to assist people with their questions directly and and and actually help with the workflows and all that. Right. Right. Right. There right yeah that's right that's right. So it can **[00:14:00]** be from an internal perspective as well. It can be from an entire uh go to market strategy too, right? It depends on what the company's necessary is. What we have seen from a large company, yes, there are internal teams trying to structure in this particular manner. However, from a startup perspective, there is a lot of startup who focus and they build a model around. Yeah. It's a disruptive model by the way. Yeah. Yeah. It's it's a maybe I I should contextualize this. Is your main audience more in the corporate enterprise space or is it in the startup to mid company Yeah. If you see ideally we have customers from uh series A+ to midsize in the price, right? So that's the sector we are focused on. That's our sweet spot. But however, as we get matured, we'll be going into the enterprise ecosystem as well. Right now, we consciously taken the decision not to go into enterprise. Uh because the bar is very high there, right? Uh so we wanted to make sure our benchmark is set accordingly. But right now we can happily cater to midsize enterprises. Interesting. Yeah. And that's a smart move. Some people just want to jump ahead and say I want to land an enterprise client, but you have to know what you're getting into. Becomes a mess. Really? You have to stay with me. You have to Yes. Okay. No, I like this. I I we develop a lot and and now we've recently been using agents a lot for for internal right? Our own products and client products. And just yesterday we were we were even laughing about with a colleague how it can find little things like you were saying where **[00:16:00]** as you're going through and if you have the code you have a clean code base and everything and then you can tell it to do something and then it'll suddenly point out things that you're like oh let me recheck that and then it tells you oh by the way I noticed these shouldn't exist because xyz and it figures some things out that you might have not thought of and you're like, "Wow, that was an interesting find." And if you had have it well built like you guys are doing where it's it's already prompted in the right way to find these sort of things and and find edge cases and uh understand how a typical user would actually go through the flow. It's it's it's so useful and fast and there's so much money being left on the table by even large e-commerce companies. I just remember the example that it's astounding. So there's this clothes store. I won't mention name, but they sell clothes mainly for little kids and we've tried to shop there through the years. Faced issues many times. It's a very well-known brand. And the other day we were we were going through and it just there's a bug where if you're signed in it doesn't let you check out with a a certain set of parameters. If you sign in it just doesn't work and like it just the site doesn't work. And so many times we have not purchased from them. And last time we said let's just try it as a guest or something like that. anyone and I was like how many millions of dollars are they losing because nobody has taken the time to check. That is true. That is true. Maybe offline you can tell which company we'll try to reach out to them. Oh yeah. Yeah. For sure. I'll tell you. It's it's **[00:18:00]** funny. I don't like it's funny but sad because if you're a startup you're on it like you if someone tells you something's not working, you immediately know to fix it. But sometimes you get larger and it's such a big company, so many moving parts that nobody's really doing the checking. Right. Right. So some like this. Yeah. Because that is badly impacted business. Nobody's realizing that even though it's a straightforward thing, right? They will be more occupied to release a major feature to the product, right? Or to the customers. So because of that feature, it would have affected the checkout process. So yeah, they would have released that feature but end of the day uh they would have lost a bunch of customer because of that just because of the particular feature as such. Mhm. And this is a question that is not directly related to what you do but let's um it's close, right? I'd like to ask your your thoughts on this. Many sites don't have support or they have a very limited type of support because they grew and they they just can't manage so much support. They don't have the possibility of having too many uh people on the phone or in chat etc. And so it becomes an issue of let's say you find that if that happens but there's no real way to let them know not only hey I'm trying having trouble with myself but I'm letting you know of something that's breaking your checkout, right? So have you seen or or how how do you guys manage um having a way for the companies to accept feedback from people but actionable feedback that if someone says something is actually broken that that there's a flag or a trigger or something that tells the dev team to look or the QA team to look into it because otherwise again this keeps happening but there's not even a way to let them know **[00:20:00]** and that someone needs it and and knows that it's Yeah, usually for majority of the companies, right, uh there will be an internal tool which is monitoring the customer flow because for an e-commerce is pretty straightforward like end of the day if they click on the check out button they should end up buying it right and the order ID should be created. So if there are lot of drop down drop off at that point either something is not right uh from the maybe the interface perspective or there is a bug there right so the system should automatically highlight that even without any feedback as such however as you clearly mentioned the customer should be able to express that this is not working usually so through at least support ticket but majority of the customer as you mentioned doesn't have that but how we do it right if we figure out something on that uh because we are in that particular space we directly go to LinkedIn and message the product manager or the dev team right so we send them the video so that it will help them out uh because it we can automate it faster right we can automate and show this is what is happening please fix it uh but for a customer as such yes there should be a flagging system but instead of that majority of them go for Because sometimes what happen they get false posities right of festival. Uh maybe because of that they avoid that kind of uh functional flow from a support level but they should definitely have an internal monitoring system when this thing happen especially for e-commerce otherwise it directly impact their business. Yeah. And and now with LLMs and agents, I think it's even more feasible for business to do so because they could just have a layer even if it's semmanual if you want just checking checking the stats every day, right? like you were saying, check out drop offs or **[00:22:00]** something or even just requests like you could even have it receive the requests and even if you don't have someone to read them all, at least have an AI agent just reading them seeing oh 55 people told us basically the same thing. Right. Right. Then let's let's get on it and see what is actually happening. It's uh what I I'd like to mention a tool that I recently learned of Microsoft Clarity. Have you used it? Yeah. Yeah. Yeah. It uh it's like an analytics, right? For those who are listening, we typically everyone just thinks of putting Google Analytics on their site, Privacy aside, that's like that's what people want. Put it because it's free. Uh but then cloud released some analytics. Uh and then there's some other providers as well. But Microsoft has their version that that has analytics and we were testing it the other day because we have a friend that that it's very Microsoft, right? And uh it's amazing what it gives you out of the box. It's still free, right? Right. You can have it with your with your tooling, but it has a lot of little things like it detects rage out of you enable it. And so if someone's just clicking a bunch, it it has an algorithm that detects that they're having trouble and it'll flag it for you to go and check and and you don't have to do any any advanced things. You just go and you can see the replay and all that. So those types of tools I think are are very useful for for companies that might not necessarily have a an implementation or development team on handle that right. Yeah. There's bunch of tools which work in this particular ecosystem. uh that is realtime monitoring observability tools where it will highlight right even if an API fails it suddenly highlight through slack all those things can be incorporated but definitely depends on how the company's threshold is towards this particular um I would say the tools that are all about **[00:24:00]** they have to implement in good with BotGauge But if a company uses something like this, then they are better prepared to handle bigger loads, spikes, stuff like that because they're they're constantly checking their systems, right? That's a big problem that happens. Also, everyone is nervous when there's going to be a launch or a release or something because they they're like fingers crossed nothing breaks. But you shouldn't. If are if you're constantly checking things, then things should go as they should, right? So, do you find red stress in your clients when it comes to these sort of bigger events when they have some sort of tool like this like yours uh and and when they're developing new initiatives versus what they used to Yeah, that that's the entire goal, right? So the way we come in is to make sure they can release with confidence, right? How can we build a trust layer QA across across their development ecosystem? So once we come in, yes, it is going to be stress free for them for sure. Uh however they also should understand that the coverage is an ongoing improvement right the first there's a step-by-step improvement they need to cater to the first is the critical test coverage that should definitely be about 95%age right if that is not there the stress level is going to be very high no matter what the after that the second level is pos negative scenarios right after that is going to be edge faces. So once they cross **[00:26:00]** all these three stages then they can become absolutely right to achieve that is what is more important and all the analytics they should carry out because coverage is going to be a subjective term no matter what however it can be defined in multiple ways. How we define is like we analyze the usage pattern user pattern also right and I said okay these are the top 120 flows user goes through it so that becomes in a critical coverage portion then a derivation of negative scenarios so it goes it goes in that manner so you should go in a data analytical test coverage pattern it's not random so that's what we encourage customers to go with to have a stressf H. Yeah, that's good. How do you figure out what those typical user flows should be? Does this only work for companies that are already established and have a lot of data to back that or is there a way for you and your team to figure out what it will be? I'm talking about mostly like if someone's launching a new thing is there a way to do this see uh when it comes to CDC plus startups right uh yes we cannot expect everyone to have the analytical tools in place so that we can derive the data however they do derive from a segment perspective consider you consider e-commerce applications coming we know we already know what is the most important flow which is going to be Right? That user should be able to select the product. They should be able to check out and make the payment. Right? That goes without saying. So every business moment they explain their businesses. So the critical flows are defined right by you take enemies **[00:28:00]** right let's take slack the most critical thing that has is communication like they should be able to have a direct channel direct communication that goes without saying so every business criticality is there in the business itself right so we'll be able to easily now comes the point which is the most utilized flow Right? If they have data yes otherwise we need to have an observability lab build and then build the data but it's okay like we understand if we take a particular module what can be the critical flows that's why we do it end to end scoping before we even start the Yes. And that's something very interesting for people who are not in the software space or in the AI space or have not even live coded or experimented with with these tools. It's much different today than it was before to get sample data of of what you should test against. Right. Right. I've re I've found that it becomes a lot easier when we're onboarding new clients even sometimes when they don't have the data we can go and do research with these agents and obtain industry or segment data that then can be put in so that we can at least show them the demos or the prototypes with what the industry shows and so it's not just dummy text and and dummy because it makes sense. It's actual KPIs and things that make sense in the industry and then they're like, "Oh, okay. Yes, this this way doesn't work for us. We have to adjust this." But it it sparks ideas, right? Right. With with that new possibility of having these tools assist, which is super helpful. That that's the end. What did QA **[00:30:00]** look like before Budg Cage? Like for someone who's listening who is not at all in the space, what would a company and for many what does the status quo look like today with with just regular QA with people and what are the the downsides to it? Right? You see I will talk from a different lens right let let me take an example of an e-commerce customer who recently come to us they having almost 25 QA testers both from manual and automation engineering now the company is at the position that they have only 56% of test coverage right that they think it's 56%age that's another benchmark now they want to increase it above 80% because otherwise there is a lot of drop off uh from their current because it's e-commerce right they're seeing a lot of drop off they know that their bugs are getting into production uh so they the management in the division okay can we hire 10 more people right 10 more automation engineers now the question comes when you hire 10 people what is the increase you're going to get are you going to get just 10% of coverage right then you have to not hire 10 people you have to hire 20 30 40 right so he'll keep on hiring but what happens when a man power changes right then like we have recently even encountered a company where they were having three selenium testers or three left the company and they enter automation stop because even when they hire people their learning curves increases they need to understand their scripts which they written the format every users are different right so that is also going to happen for this e-commerce company. Then we came into picture. We are like okay we can reach 80% of coverage. It was for them too **[00:32:00]** good to be true. Like they were trying for 10 months right? This is what is happening. Now we are seeing okay we'll do it in 45 days. So it's too good to be true like initially they were like does it really happen? But it's just 45 days right? Why why not? So they did the the paid PC with us and then they understood okay the enhancement is much more than what they can expect and right now they reduced the or they have reallocated their team member and right now three are in the particular project which we are doing right three are just for the monitoring purpose and understanding how we do it so that they can guide us in which level of enhancement they are looking forward for and they were able to allocate other people for other projects and other department has such so that's a kind of change so it's like before and after for a particular company it's a state how the QA environment is changing and if you ask me what's going to happen in future yes the human dependency is going to reduce to the minimum for sure and there's lot of relocation is going to happen from all the skill set including deva dev you already start see also you start seeing it is going to spread to other departments as and it's a it's a it's amazing and at the same time it's sad that people then follow the narrative of oh but people are losing their jobs because of it when when in reality you you use the right terminology they're relocated that doesn't mean necessarily that they lose their job that just means that they're changed to something else and I like to see it from the point of view of okay you had 30 people testing or 10 people whatever you have let's say you have 30 people testing you bring in a tool like BotGauge that means now you just need three people on the team at least for a while and then those other 27 **[00:34:00]** can go to another initiative that also put something like BotGauge in now those three can handle that big project and then you have other so because these are big companies and so you can now have the same amount of people now doing more rewarding work because They're not clicking on things all day. They're actually monitoring and adjusting and providing their input as an expert. Right. You turn people from clickers into knowledge domain experts. Right. It's an upskill and it's rewarding for the person. The company can scale more, which means they're going to need to hire more people. It's just a it's a virtual cycle if you if you think about it the right way. Right. Right. Right. It's all about the mindset of a company. Right. how they want to evolve otherwise they're going to lose the race no matter what. So it's all about how they otherwise you know like for the past 20 years there have been multiple startups which have become unicorn. It's just because they have evolved and the large companies failed to do that. So it's all depends upon how the evolution happens within and from an external point. Yes. I I think you gave an excellent example. The the the cost of hiring a lot more people to try to increase that That expands into so much more because now you need managers and then managers for the managers many managers just it becomes a messy thing to try to increase a percentage that may or may not increase just a little bit and it has diminishing returns. wearing using the right tooling can actually get you further as you with your clients which is super interesting. I I think people are gonna are gonna learn about uh this and start maybe in their companies thinking about okay so should I use something like this do I need to use something like this at what point right so so my question to you **[00:36:00]** as we start to wrap up here is what would you say to someone who has a company and they're developing something or they have a software web based software and and they're they're not happy necessarily with with where they coverage wise or maybe they don't even have tests happens right with with some of the fast growing startups what would you say to them if if they asked when am I at the right place to start looking into something like this right so we always conduct discovery calls right in the discovery call we do reject customers We tell them you don't require us or you don't require automation. The first point is your release cycle. If you plan to release once in M1, you don't you don't require speed. You can do it with the current traditional tools or manually itself will hold. Now second, so that's like an internal push, right? Release is an internal, right? Second parameter, how many customers you and what kind of customers are they? Is it like a design partner or is it like a friendly customer or are you in the race of like competing with other major players right where you have n number of customer and losing customer is not not that ideal way of doing it. So when you have more number of customers and there are bugs getting released in the production then it becomes like an external push to adopt an AI enabled tooling into your ecosystem. So these are the two major push right. So when we see any release which is happening more than once in a month they're not the right fit or who doesn't have a push for an external customer being utilized on more number of users then it's not a right customer for us or **[00:38:00]** ideally it's like no bugs are getting released into production then that means not making much changes or they have figured out their testing right then yeah we won't be the right vehicle topic. Nice. Yeah. And I appreciate that because it's uh it's good for for service providers to be able to tell people, hey, hey, just it's not use this other thing or do it this way and and you'll be fine. When you get to that point, call us and we'll we'll get into it. But that's excellent. I' I've learned a lot today, Pramin. I really thank you for sharing your expertise with us today and and opening our eyes as well into what exists today because I think more and more this year we're going to see so many changes in the way a lot of different industries work with regards to agents and and all that. So uh I think this was a great introduction into the topic and we we might have a few more similar that uh go into into the ways that these tools are going to help us more be more efficient uh in our processes. I mean if if people want to reach out maybe think about a discovery call or anything like that where's the best place to find more information about BotGauge? Yeah, it's in the website itself, right? We do provide a free tree free bug analysis, right? Free bug report I would call it. So they can utilize that or they can directly reach out to me in LinkedIn. We can have a discovery call with my team. Uh I can do that. Yeah. Mostly through website or through LinkedIn. Anything works well. Excellent. Excellent. Well, thanks again. It's been a real pleasure to have you on your show. And uh for anyone who's listening, if you haven't liked and subscribed to the show, please do so and make sure to follow Bermine as well there on LinkedIn and I'll put the links in the show notes so that the people can know where to find you. Thanks so much forine. Yeah, thank you. Thank you Armando for inviting me. It was a pleasure talking **[00:40:00]** to you. Likewise. Thank you. Thanks everyone and we'll see you in the next one.