How to Choose an AI Automation Agency

Search for an AI automation agency and look closely at who shows up. About half the results are not written for a business owner at all. They are videos and guides teaching people how to start an AI automation agency this year, what to sell as a beginner and how to land a first client. That is the market you are hiring from. Some of the companies in it have run real systems for years. A lot of them finished a course last quarter, and on a sales call the two sound exactly alike.

I build AI applications at Killerspots, and I end up looking inside a fair number of systems that were set up somewhere else before the owner called us. The nine questions below are the ones I would put to any AI automation agency before signing, ours included. None of them takes a technical background. Each one needs a straight answer, and the way the answer comes back tells you most of what you need to know.

What is an AI automation agency, and what does one actually do?

Three different businesses use the name, and it helps to know which one is on the other end of the call.

The first resells a tool. You get a subscription to somebody else’s software with a login and a setup call, and the agency’s job ends when the invoice goes out. The second builds something custom, demonstrates it and leaves. It works on the day of the demo. The third builds the system, wires it into the way your business already takes leads, and stays to run it. Only the third one is really selling AI automation services. The first two are selling software and a project.

The work itself is narrower and more boring than the pitches suggest. The automations that reliably pay for themselves answer the calls you miss, reply to a website chat at midnight, ask a lead the qualifying questions before a person gets involved, book the appointment, ask for the review after the job and move information between systems nobody has time to keep in sync. That is most of it. The ambitious ideas are where the money usually disappears.

So when you compare companies, compare them on that work. Our own AI automation agency page lists what we deploy, and it is a short list on purpose.

1. What would you automate first for a business like mine, and why?

This is the question that separates people who run these systems from people who sell them. Someone who has done the work will not answer it right away. They will ask how many calls go to voicemail in a normal week, how long a web form sits before anyone replies, and what happens to an estimate after it is sent. They need those numbers because the first automation should go wherever the most money is leaking, and they cannot know where that is from the outside.

We start every engagement with what we call a bottleneck map: where the team’s hours go and where leads leak. Every automation we build has to trace back to something on that map. For most service businesses the first item is the unanswered phone, which is why missed-call text-back is the automation we install most often, and why speed to lead gets fixed before anything clever does.

The warning sign is a proposal that arrives before anyone has asked about your phones. That agency is selling the same package to everybody. The other warning sign is an agency eager to automate a process that does not work yet. If your follow-up is broken when a person does it, automation gives you the broken version faster and at volume.

2. Can you show me a system you built that is still running?

A demo is a conversation the builder already knows the ending to. A production system gets the caller who mumbles, the customer who asks two things at once, and the person who types the name of a service you stopped offering in March. The distance between those two is where most of the real work sits, and you cannot see it in a sales meeting.

So ask about the bad week. What did the agent get wrong in its first month? How did they find out? What did they change? I can answer that for our own systems because we were on the wrong side of it ourselves. We run AI automation inside our own agency, across intake, follow-up and reporting, and the first versions did not survive a busy Monday without corrections. That experience is what a client is actually paying for.

There is also a test you can run without asking anyone’s permission. Fill in the agency’s own contact form at nine at night, or call its main number after hours. A company that sells instant response should have it. If the reply to your form shows up two days later from a salesperson, you have learned how much of its own product it uses.

3. What happens when the AI does not know the answer?

Nobody loses a customer because a chatbot said, “Let me get someone who can answer that.” Customers are lost when the agent makes up a price, promises a time slot that does not exist, or tells someone a repair is covered when it is not. An agent that admits it does not know is doing its job.

That is why we build the escape hatch first, before the agent learns a single answer. The order matters. If the handoff is an afterthought, it gets tested last and trusted least.

Ask to see two things. One is the list of subjects the agent is told to stay out of: exceptions to pricing, anything that sounds like a diagnosis, complaints, legal questions, whatever is sensitive in your trade. If the agency has no such list, the agent has no boundaries. The other is the handoff itself, live. Who gets the alert, on what device, and does that person see the whole conversation or start cold? I went through how the guardrails get built in my guide to AI chatbots for service businesses, and the same rules apply to phone agents.

4. Where does every conversation go when it ends?

A chatbot that collects a name and a problem and then leaves both in a log is worse than no chatbot, because the customer believes they have been helped. The conversation is the easy part. What happens in the ten minutes afterward decides whether it turns into a job.

In a system built properly, the conversation creates or updates a contact, tags what the person wanted, drops them into a pipeline stage and starts the right follow-up, whether that is a confirmation text, a reminder or an automated follow-up sequence for the ones who did not book. The phone, the forms, the chat and the calendar all write to the same record, so a person picking it up can see everything that came before. We build this on KillerConnector, our CRM, which we set up and brand for the client.

You do not have to take anyone’s word for it. Ask the agency to run a test conversation and then show you the contact record it created. If that takes them more than a minute, the connection is not really there.

5. Who writes what the AI says, and how does it learn my business?

When owners ask why a build takes two to four weeks, they expect me to talk about technology. Most of that time is the interview. Somebody has to sit down with the people who answer your phone and write down what they actually say: what you do and do not service, where you will and will not drive, how you explain a fee, which questions come up twenty times a day. Your website has perhaps a third of it. The rest lives in your office manager’s head.

An agency that skips this and points the software at your website will launch faster and answer worse. Ask how they collect the information and who on your side they need time with. If the answer is “nobody, we handle it,” be careful.

Then there is the writing. What a voice agent says to a frustrated caller at seven in the morning is a script, and it is closer to scriptwriting than to engineering. This is one place our history helps. Killerspots has written call handling scripts, auto-attendants and on-hold messaging for businesses since 1999, long before any of it was automated. Ask to read the greeting and the first few replies out loud before launch. If they sound like a terms-of-service page, callers will hang up on them just as fast.

6. Will my customers know they are talking to AI?

People are comfortable with an AI that solves their problem at eleven at night. They get angry at one that pretends to be a person and is found out. So we disclose. The agent says what it is, does the routine part well, and passes anything unusual to a human. In my experience callers care far more about getting an answer than about who is giving it, as long as nobody tried to fool them. I covered how callers react in more detail in my guide to AI voice agents.

There is a second half to this question that almost nobody asks. Automated texts to customers generally require their permission, and the rules on recording phone calls differ from state to state. I am not a lawyer, and your automation agency is not your lawyer either. But it should know these questions exist, build the consent wording into your forms, and tell you plainly which parts need your attorney’s eye. An agency that looks surprised by the question has not deployed many of these.

One more line worth knowing about, because it tells you how an agency thinks. We use AI to answer the phone. We do not use it to voice a commercial. The voices in the ads we produce are human, and that is a decision, not a limitation. An agency should be able to tell you where it will not use AI, and why.

7. What do I own if we part ways?

Ask this before the deposit, while the answer costs nothing.

Go through the pieces one at a time. The tracking and texting numbers: are they registered to you or to the agency? The CRM: is it your account, or a sub-account you lose access to on the day you cancel? The knowledge base, meaning everything written down about your services and policies during the build: does a copy come with you? The conversation history and the contact list: can they be exported, and in what format?

Then ask where the data goes. Who at the agency can read your customers’ conversations? Are those conversations used for anything besides running your system? You are handing over what your customers say when they think they are talking to you, so the answer should be short and specific.

A fair agency will keep its own internal tools and templates, and that is reasonable. What it should never keep is your customers, your numbers or the record of your business. A simple way to settle it is to ask them to export your contacts and conversations to a file once the system is live. If that request turns into a negotiation, you have found the lock.

8. Who keeps it working after launch, and what number will you show me?

These systems drift. You add a service, change your Saturday hours, raise a trip fee, stop covering a ZIP code. A person on your team absorbs that in a morning. The agent keeps telling callers what was true in the spring until someone updates it, and an agent quoting last year’s policy is a bigger liability than a phone that rings out.

So the question is who reads the conversations. Not a dashboard of totals. Somebody opening real transcripts every month, finding where the agent hesitated or answered wrong, and fixing it. That review is also where the tuning happens. A chatbot can launch a week or two after kickoff, and it gets noticeably better over the following weeks only because someone is correcting it against what customers actually ask.

Then ask what number you will see in ninety days. Messages handled and minutes saved are activity. Appointments booked, leads recovered after hours and revenue you can trace back to a conversation are results. We give every automation its own number and review it monthly, and the ones that are not earning get turned off. Ask any agency what it has switched off for a client. A company that has never killed one of its own automations is not measuring them.

9. Do you need a specialist, a consultant or a full-service agency?

The common advice is to always pick the specialist. I do not agree, and I should say why, since we are a full-service agency and you should weigh my answer with that in mind.

For a marketing business problem, the automation sits in the middle of a longer path. Somebody sees an ad or finds you in search, lands on a page, calls or fills in a form, gets answered, gets followed up and books. A company that only builds the middle will tune the middle. It will not see that the form asks six questions too many, or that the ads are sending the wrong callers. When one team runs the ads, the site, the CRM and the automation, the system gets judged on booked jobs, because that is the only number the whole team shares.

A specialist is the better choice when the work has nothing to do with marketing or leads. Invoice processing, inventory, internal approvals, anything deep inside your operations software is its own discipline, and we would point you elsewhere for it. A consultant makes sense when you have people who can build and need someone to decide what to build. A freelancer can be a fine choice for a single workflow, with one question added: who picks up when that person is busy or gone?

And plenty of owners should do the first steps themselves. I laid out which ones in my guide to AI for small business marketing.

When should you not hire an AI automation agency?

I would rather lose the sale than build something that should not exist, so here are the cases where I tell people to wait.

The process is broken. If quotes go out late, if nobody owns follow-up, if two people give customers two different answers, fix that first. Automation copies whatever you already do.

The volume is small. A business taking a handful of calls a week does not need an agent. It needs to pick up. The return on these systems comes from the gap between how many people reach out and how many get a fast answer, and when that gap is small, so is the return.

The calls are the relationship. Some businesses sell trust, one long conversation at a time. A family in a hard moment, a large commercial account, a customer with a serious complaint. Put a person on those. We do not recommend AI for them even when a client asks.

One tool already does it. If your scheduling software can send reminders and your phone system can text back a missed call, switch those on before paying anyone. A good agency will tell you that on the first call. It costs them a project and earns them the next one.

Start with the leak

You do not need a strategy to begin evaluating an agency. You need two numbers: roughly how many calls go to voicemail in a week, and how long a web form waits for a reply. Bring those to every agency you talk to and see what each one does with them. The ones worth hiring will ask more questions. The rest will open a slide deck.

If you want to run us through the same nine questions, that is the right way to start. Tell us where you think you are losing leads and we will show you what we would automate first, what we would leave alone, and the systems we run on our own business. You can reach our team here, read more about our AI chatbots, AI voice agents and AI content and automation, or see how we handle AI automation in Cincinnati, where our studio and team are based. We work with businesses nationwide, and the first conversation is about your phones, not our software.