SAG / ARCHITECTURE NOTE

Why Can Averaging Ratios by Engine Be Wrong?

The principle for calculating an aggregate metric while preserving the numerator and denominator of each ratio. A simple average of ratios from engines with different answer counts gives the same weight to small and large samples. This can misrepresent the actual set of answers.

SAG editorial · 2026-10-08 · Updated 2026-10-08

What Is a Weighted Ratio?

It is the principle of calculating an aggregate metric while preserving the numerator and denominator of each ratio. This note treats weighted ratios in terms of the responsibilities of their inputs, transformations, and outputs, rather than as a feature name. For an analysis result to be trustworthy, there must be a clear chain showing what data went in, what was checked, and how far the conclusions can go.

Why Is This Technique Needed?

A simple average of ratios from engines with different answer counts gives the same weight to small and large samples. This can misrepresent the actual set of answers.

Design Principles and Data Flow

The aggregate is the sum of the numerators ÷ the sum of the valid denominators. If engine conditions differ, first assess whether summing them is meaningful, and retain the individual tables as well.

Counts by engine → Sum of valid denominators → Aggregate and individual rates

Each stage should not relabel the success of the preceding stage as an achievement of the next. Recording data identifiers, periods, and validation status together makes it possible to locate where omissions and errors occurred and determine what needs to be checked again.

Connection to the SAG Architecture

SAG citation reports provide both the number of valid observations by engine and the overall rate. The calculation starts from the original counts, not rounded percentages.

SAG’s operational value lies in connecting this relationship to pages and questions, comparison results, and improvement work. Instead of reading only the numbers, customers can review both what needs reinforcement and the basis for the assessment. Patterns that require further application should be interpreted based on the scope of the relevant paragraph.

Illustrative Example and Criteria for Assessment

For illustration, if Engine A has 1/2 = 50% and Engine B has 1/8 = 12.5%, the aggregate is 2/10 = 20%. The simple average, 31.25%, is not the rate per answer.

The example above is provided to explain the structure and calculation; it is not measured performance for any particular customer. In an actual report, the selected period, target, observation conditions, and original records must be linked so that the same assessment can be checked again.

Practical Validation Checklist

Flow stage	Item to check
Counts by engine	Store numerator and denominator
Sum of valid denominators	Calculate before rounding
Aggregate and individual rates	Separate differing conditions

Check that the same meaning is maintained not only for valid inputs, but also for empty data, duplicate data, and data with differing conditions. Linking validation items to completion criteria can reduce the gap between the feature description and actual operations.

Limitations and Points to Consider in Application

Weighted aggregation does not eliminate differences in question difficulty. Review groups by market and model alongside their sample sizes.

Research and Official Documentation

PostgreSQL JSON Types — Official documentation describing the characteristics and limitations of JSON storage types.

The external reference provides background on the design topic above; it does not certify every SAG implementation or customer outcome. The application guidance and illustrative example in this note are based on SAG’s operational structure. Source checked: 2026-10-06.

How to Continue Reading About This Technique

Distinguish monthly samples, the denominator for citation rates, competitive benchmarks, and Goal attainment rates.