At LeadTruffle, we serve Home Service companies in the USA & Canada and help them book jobs through various lead channels (especially paid ones). We basically help them optimize ad spend to try to quickly book jobs. We focus a lot on booking metrics and ROAS. This is different from a typical AI-secretary AI voice product that just tries to do everything, including supporting existing customers.

Anyway, LeadTruffle has grown to be quite a complex piece of software in the one-and-a-half years we’ve been live and had paying customers.

A lot of this is due to the market segment: SMBs (actually, they are VSMBs - very small businesses) tend to naturally want a combination of things:

1. All-in-one solutions
2. Full or almost-full service, or coordination with a service provider
3. Integrations with other big platforms they are stuck on

I think a lot of this comes from the users being non-technical or, more succinctly even, “anti-technical”. They don’t care about your software; they in fact want less software; they ideally don’t even want to log in. Many of them either want an employee to handle it all or they want to tell their agency guy they talk to on the phone for any technical issue to handle.

The agency guy they talk to, by the way, might do anything from running ads for them to managing their website. The agencies are often small local people, and they don’t want to deal with your new software either. The employee is very likely disinterested in following any instructions for setup you send them and learning something new.

They also don’t want to change anything because it requires extra work. This is very understandable: 90% of the pain with IT management in a business is changes, not new things.

This means we need to plug into every existing system they are using in the way they use it (not necessarily the way we know would be optimal). And we need to do it for them. Our staff, or else the customer churns.

I hope I’m setting the picture up for why LeadTruffle is in a difficult market.

Evolving as a platform

When we first launched, we were a pretty simple system: AI could have a conversation with your customers who came in from a landing page on a website, usually after running ads.

The AI would seal the deal by sending a booking link for the customer to book the job. This was usually a Calendly link or a Housecall Pro or Jobber public booking link, something similar.

Since then, we have grown to manage more of the lifecycle of a customer contact and do conversational booking.

We handle a lot of scenarios

We now handle conversational booking on multiple mediums and platforms, booking into multiple CRMs.

• We can book jobs from Voice, SMS, Email and app platforms.
• We integrate with Facebook Messenger, Google LSA, Thumbtack, Angi and more (including custom forms).
• We book jobs live conversationally into Jobber, Housecall Pro, Service Titan, Job Nimbus, Builder Prime, LEAP.

This is a combinatorial explosion of scenarios that we handle with simulations (which I will get back to later).

Booking is hard and full of edge cases

Booking a job via human conversation is hard because, although the happy path is pretty simple, the reality is often full of edge cases. Some examples:

1. The customer may be outside of your service area.

2. The customer may be in a service area that only one specific technician can handle, and that tech can’t do some specific types of jobs.

3. The customer might request a time slot, get booked, then immediately request a different one.

4. The customer might be confused and change their mind multiple times on a single phone call.

5. The booking might fail because the CRM could not be reached.

6. The booking might fail because of a weird data validation issue with a CRM.

7. The customer may immediately call back and ask to change a slot.

8. The customer is actually not a customer but a stalker sending dick pics to an employee.

You get the point. At this point, after hundreds of thousands of conversations we’ve handled, I’m not sure you can ever see it all, but it does span the gamut.

The improvement loop

The improvement loop I focus on is:

flowchart TD
  A["System Design Deployed"] --> B["Observed Real-World Data"]
  B --> C["Test case failure"]
  C --> D(["Simulation"])
  D --> E["Improved system design"]
  E --> A

  classDef step fill:#ffffff,stroke:#c5cad3,color:#12161d
  classDef sim fill:#e8f1fb,stroke:#0b66cc,color:#12161d
  class A,B,C,E step
  class D sim

The simulation step is testing the live AI model against a specific staged set of data + inputs and testing the expectation that it makes a combination of tool calls.

I heavily monitor our production events and feed them into AI tools.

If we see a booking that failed or encountered a scenario that was unexpected, we create an incident from this. I then use the best AI tools (e.g., GPT-6 Astra) to run a detailed diagnosis of the event.

Then I do the following:

1. Build the failing simulation using anonymized test data.
2. Verify that the simulation, as the system currently is built, fails.
3. Introduce a change.
4. Run the simulation again and see if it passes.
5. Keep going until we get a successful simulation that seems reliable and is a small change.
6. Run other simulations in my test suite and make sure there is not a regression.

After doing this a while, you end up with a suite of simulations that cover all kinds of scenarios and features.

Because we have a combinatorial explosion of lead sources, communication mediums, and booking providers, we end up with thousands of these scenarios. It’s the only way that is possible to ensure the software works, and running all these simulations is an expense, but it’s very important.

AI in production

On xitter, people just care about benchmarks and coding performance. However, we do not.

There are several things that matter A LOT more than intelligence:

1. Token throughput (tokens per second - ideally we like above 70; 100 is great).
2. Instruction following: do they reliably return structured JSON and follow complex prompts?
3. Tool calling: this is critical for running agentic flows.
4. Time to first token: latency is a huge issue for us in some workloads.

All of this causes us to make some interesting model choices. It’s pushed me toward trying Gemini quite a lot because it’s good at instruction following & it has excellent token throughput.

Another thing I will mention is fallbacks: you need to always have multiple fallbacks for every workload, and you need to test them. AI providers are highly unreliable, and you can expect multiple of them to be down at a time. Routers are great for this but come with their own challenges.

Some stuff I have learned about AI in product + AI simulations

AI models are non-deterministic. Yes, you know that, but you really need to feel this intuitively.

For example, I tend to use Luna for simulations because it’s a small model that behaves worse than Terra in production, which is similar but larger and smarter. If my simulations consistently pass all the time with Luna, I can be reasonably confident they will be even more reliable with Terra.

Btw, you still need to run the full suite with every model you deploy. And you need to run it for the fallbacks too, since they will fire frequently.

Even a tiny bit of reasoning tokens is better than none. For example, Luna low passes a lot more test cases in my suite of evals than when I use reasoning-none.

A final thought about automating loops

The real productivity gains from current-generation AI tools come from verification loops. Can the AI verify the outputs of a system after it changes something? Can it make an oracle value or a North Star metric it can improve?

For us, it’s jobs booked (correctly).

flowchart TD
  A["AI changes the system"] --> B["Verify the outputs"]
  B --> C(["Jobs booked (correctly)"])
  C --> D["Hill-climb the metric"]
  D --> A

  classDef step fill:#ffffff,stroke:#c5cad3,color:#12161d
  classDef sim fill:#e8f1fb,stroke:#0b66cc,color:#12161d
  class A,B,D step
  class C sim

If you can put an improvement loop around some kind of system that an AI can use to hill-climb metrics, you can use that to improve a system automatically at scale.

This is real, and once you can do it for your business, you can make magical things happen.

I think we are going to see all kinds of improvements in ordinary things that can be verified using digital systems. It’s going to be slower than people, but this is the actual engineering work that needs to be done.

Contrary to popular opinion, software developer jobs are not “over” - it’s now your job to put verification loops around everything you possibly can and let the AI go to town.

The same loop scales past booking software:

flowchart TD
  A["AI changes the system"] --> B["Verify in the real world"]
  B --> C(["Healthy years of life"])
  B --> D(["Output per robot-hour"])
  C --> E["Hill-climb"]
  D --> E
  E --> A

  classDef step fill:#ffffff,stroke:#c5cad3,color:#12161d
  classDef sim fill:#e8f1fb,stroke:#0b66cc,color:#12161d
  class A,B,E step
  class C,D sim