How we actually test AI software, step by step
This page opens up the process — how a tool is selected, how we set up an account, what devices and scenarios we test on, how we check AI output for accuracy and hallucinations, and how a score actually gets calculated. Our broader editorial policies live on Editorial Standards; this page is about the mechanics of testing itself.
Minimum active testing window
1–2 weeks per tool
Re-test cadence
Every 90 days
Account type used
Real, paid, small-business tier
Selection, accounts, environments, and devices
Before any hands-on testing starts, four things are locked in: which tool, which account tier, which environment, and which devices. Skipping any of these makes every later result harder to trust.
Product selection
A tool is shortlisted against real reader demand and relevance to small business workflows — the full criteria live on Editorial Standards.
Account creation
A real, paid account is created at the pricing tier a small business would actually choose — never a vendor-provided demo with unlocked enterprise features.
Testing environment
Testing happens under normal business conditions — typical home or office internet, no synthetic lab network optimized to flatter response times.
Testing devices
Desktop browser, mobile browser, and the dedicated app where one exists — since a business owner may set up on a laptop and manage day-to-day from a phone.
Prompt, workflow, and automation testing
Once the account is live, testing moves through three connected stages — each one builds on the last rather than testing the tool in isolation.
Prompt testing
Individual prompts and inputs are tested in isolation first — a single customer question, a single scheduling request — to see how the AI responds before anything is chained together.
Workflow testing
Prompts are chained into a full realistic sequence — a caller books, reschedules, then asks a follow-up question — to see whether context carries through correctly.
Automation testing
Any triggers, integrations, or scheduled actions the tool sets up on its own are checked for reliability over repeated runs, not just a single successful demo pass.
Accuracy, hallucination checks, response quality, and speed
These scorecards describe the rubric bands we score against — not a specific tool’s actual result — since every AI platform gets its own individual scorecard on its review page.
AI accuracy
RubricMeasured against real business content we already know the correct answer for — a menu, a service list, a pricing sheet — not a synthetic benchmark set.
Hallucination rate
RubricA hallucination is a confidently stated claim that’s verifiably wrong when checked against the input it was given.
Response quality
RubricJudged on whether a real customer receiving the response would find it clear, appropriate, and usable without editing.
Speed
RubricTimed under normal business-hours conditions, not off-peak, since that’s when a real customer is actually waiting on the other end.
Ease of use, feature validation, pricing, and support
Four practical categories, each with its own check method and pass bar.
| Category | What we check | Method | Pass bar |
|---|---|---|---|
| Ease of use | Core setup task completion | A non-expert completes onboarding unaided, timed | Completed without external help |
| Feature validation | Every advertised feature | Used directly, not read from a spec sheet | Works as described at the tested tier |
| Pricing verification | Every listed price | Checked on the vendor’s live pricing page | Matches exactly, or discrepancy is disclosed |
| Customer support | Response time and answer quality | A real support ticket is filed during testing | Resolved accurately within a reasonable window |
Security review and privacy review
Security review
We check what a vendor’s own documentation discloses about encryption in transit, account access controls, and data storage — we are not a penetration-testing firm, and we say so rather than implying a security audit we didn’t perform.
Privacy review
We note what customer data the tool collects, whether it’s used to train the vendor’s models, and how easily a business owner can export or delete it. Compliance-relevant context for regulated industries is flagged, not assumed.
How performance is weighted into a final score
Every category above feeds into one of five weighted groups. The exact per-criterion math lives on our Review Methodology page — this is the weighting at a glance.
Comparison methodology: When several AI tools are placed in a single comparison table, each one is scored independently against this same weighting before the table is assembled — the table reflects pre-existing scores, it doesn’t get built first and scored backward to fit a preferred outcome.
How often we retest, and what actually changes a score
Every review is fully retested at least every 90 days. Below is an illustrative, generic example of the kind of change that triggers an update sooner than that — not a real tool’s actual history.
Before
After vendor change
Illustrative example only — pricing changes, feature removals, and repeated reader-reported errors are the three most common triggers for an off-cycle re-test.
Testing limitations, transparency, and where we’re still improving
Testing reflects a snapshot in time, a specific pricing tier, and the specific real-world scenarios we chose to run. It cannot capture every configuration, every integration a reader might use, or guarantee an identical experience on every reader’s own setup. Where a limitation is specific to how we tested, it’s stated in the review itself.
Every review states the pricing tier tested, the date of the most recent verification, and at least one genuine limitation of the tool. If a reader’s own experience differs from ours, we want to hear about it — see Contact.
As AI software itself evolves — new AI agent capabilities, new generative AI models, new automation features — our testing process is expected to evolve with it. Material changes to this methodology will be reflected on this page directly, not published elsewhere and left unlinked.
Quick answers
Five questions we hear most about our testing process specifically.
We test the tier a small business would realistically choose. Higher enterprise tiers are noted in the review but are not the basis for scoring unless that tier is the only one with a usable feature set.
We run the tool against real business content we already know the correct answer for, then check the output against that known-correct baseline rather than relying on a vendor’s self-reported benchmark scores.
A confidently stated factual claim in the AI’s output that is verifiably wrong when checked against the real business content it was given or against a primary source.
At least once every 90 days, and immediately if we become aware of a pricing, feature, or policy change significant enough to affect the published score.
Testing reflects a snapshot in time, a specific pricing tier, and specific real-world scenarios we chose. It cannot capture every configuration, every industry use case, or guarantee identical results for every reader’s setup.
Want the short version, every Friday?
One practical AI idea for your business, tested by us first. No hype, no spam.