Exactly how an AI software score gets calculated

What a score is trying to measure. A high score means real, measurable usefulness to a small business, not polish, not marketing copy, and not a long feature list that goes mostly unused. A tool with a beautiful interface and a thin feature set scores lower than a plain-looking tool that reliably does the job it’s bought for. Scores are built from what testing actually found, not from what a vendor’s homepage claims.

A concrete example. Take an AI scheduling assistant. If it books appointments accurately but its interface looks dated, that’s a minor Ease of Use deduction on a category worth 20% of the total. If it has a polished interface but double-books appointments during testing, that’s a serious AI Output Quality and Feature Validation problem on categories worth 25% each. The weighting reflects which failure actually costs a business money and customers.

Why weighting stays fixed across reviews. The same five weights (25/25/20/15/15) apply to every AI software review under this methodology version, regardless of category or vendor. A weight doesn’t shift because a particular tool would score better under a different formula. If the weighting itself changes, that’s a new methodology version, logged with a date, not a silent adjustment applied to make one review’s math work out.

Normalization and fairness. Every category is scored on the same 0 to 10 scale using the same rubric bands before any weighting is applied, so a 7 in Pricing Value and a 7 in AI Output Quality represent the same relative performance before the weights pull them apart. That’s what keeps a “Very Good” tool in one comparison table comparable to a “Very Good” tool in a different one.

How the weighted average works. Each category score is multiplied by its weight, then the five results are added together: (AI Output Quality x 0.25) + (Feature Validation x 0.25) + (Ease of Use x 0.20) + (Pricing Value x 0.15) + (Customer Support x 0.15). Try setting every slider to the same number, say 8.0, and the total comes out to 8.0 too, since the weights sum to 1. That’s the check we use to confirm the math is right.

Why the total isn’t a simple average. A plain average would divide by 5 and treat every category as equally important, which is the exact problem the “Why Weighted Scoring” section above walks through. Two tools with identical category scores but different weighting priorities would never land on the same simple average, but they will land on the same weighted total under this formula, because the formula is what’s fixed, not the raw numbers.

Rounding. The weighted total is calculated to two decimal places internally, then rounded to one decimal place for display, using standard rounding (a result of 7.85 displays as 7.9, not 7.8). Bonuses and penalties, covered next, are applied to this rounded weighted total before the final published number is set.

What bonuses can’t do. Bonus points reward something genuinely above the baseline, not a marketing checkbox. They’re capped at +0.5 total per review, and they can’t lift a tool with a poor weighted score into a strong-looking published number. A tool with a 4.2 weighted average and every bonus available still publishes around 4.7, not a score that hides the underlying problem.

How penalties are applied. Penalties are evidence-based: each one requires something we observed directly during testing (a hallucination we can point to, a feature we tried to use and couldn’t, a cancellation flow we walked through ourselves), not a suspicion or a vendor’s competitor complaining. Where a security concern is based on publicly documented evidence, such as a vendor’s own disclosed data-handling practices rather than a claim we can’t verify, it’s treated the same way: cited to its source and applied consistently to any tool with the same issue.

How the final score is built. The published score is the weighted total, plus any bonuses, minus any penalties, capped at 10.0 and floored at 0.0. That combined number, not the pre-adjustment weighted total, is what displays as the review’s headline score and what’s used for badges, rankings, and comparison tables. It’s rounded to one decimal place for display, the same rounding rule used in the calculator above.

Why a tie can stay a tie. Two tools landing on the same number after all three tie-breakers is a real, honest outcome, not a formula failure to cover up. When it happens, a comparison table lists both tools together at that rank rather than assigning one an arbitrary edge it didn’t earn. Editorially, we’ll still describe what actually differs between them in the write-up (one might suit a solo operator better, the other a small team) even when the numbers alone can’t separate them.

Industry-specific recommendations. A tool can also be recommended for a single vertical, such as dental or health, when it scores Very Good or higher specifically on criteria relevant to that industry, like compliance-aware data handling, even if its general-purpose score is more middling. This recommendation is separate from the four badges above and is always labeled by industry so it isn’t confused with a general endorsement.

What “exceptional execution in one standout area” means for Editors’ Choice. This isn’t a vague catch-all. It requires a specific, named reason: a tool that solved a real testing problem no comparable product handled well, for example, or one whose AI output quality was consistently ahead of its category during testing. That reason is stated in the review itself, not left implicit.

Why a brand-new review starts at Limited or Medium confidence. A single testing pass tells us how a tool performed once. It doesn’t yet tell us whether that performance holds up after a vendor update, a pricing change, or a second look three months later. Confidence rises as a tool goes through repeat verification and the score holds steady, and it’s not a mark against a new review; it’s an honest signal that less history exists yet.

Data verification process. Every score is checked against the underlying testing notes before publication, cross-referencing that the written verdict actually matches the numeric result rather than the two drifting apart during editing.

Score quality control before publication. Beyond the reviewer cross-check above, every score goes through a separate quality pass: the underlying facts get verified (does the review cite what testing actually found), the arithmetic gets checked (does the weighted formula produce the displayed number), and a final editorial read confirms the written verdict matches the score rather than reading more positive or negative than the number suggests.

How scores stay comparable across different products. A score for a customer support chatbot and a score for a marketing content generator use the same five categories, the same weights, and the same 10-point rubric bands, calibrated through the reviewer cross-check process above. That’s what makes an 8.2 mean roughly the same thing in two completely different software categories: the categories being scored may look different in substance, but the scale, the weighting, and the review process behind them don’t change.

How updates affect scores. A re-test can move a score up or down. The previous score isn’t preserved as a hidden “original”; the current published number is always the most accurate one we have.

What triggers a rescore outside the routine cycle. A major product update, a pricing change, an update to the underlying AI model that affects output quality, a vendor acquisition, a disclosed security incident, a removed feature, or a vendor reaching out with a correction all trigger a look before the standard 90-day cycle would otherwise catch it. Reader feedback that surfaces a verifiable factual error works the same way.

Historical score tracking. Where a score changes meaningfully between refresh cycles, the review notes what changed and when, so the shift is explained rather than silently overwritten.

Reader feedback adjustments. Feedback identifying a verifiable factual error is treated as a correction under our Corrections Policy. Feedback reflecting a different subjective experience doesn’t change a score without new testing evidence, but repeated similar feedback can move a tool up the re-test queue.