What you'll learn (objectives)
By the end of this tutorial you will be able to:
- Design a reproducible citation source tracker that ties vendor claims to verifiable evidence. Extract, standardize, and timestamp numerical claims so you can compare apples to apples. Produce a short, audit-ready case study with raw data links, screenshots, and an evidence trail. Spot common vendor tactics (cherry-picking, shifting baselines) and quantify the effect. Automate part of the workflow so future pitches are assessed faster and more reliably.
Prerequisites and preparation
This approach is designed for budget owners who want numbers, not fluff. Prepare the following before you begin:
- Tools: Google Sheets or Excel, a citation manager or structured CSV for source items, a screenshot tool that timestamps files (e.g., native OS screenshot + file naming convention), and optional: a simple database or Git repo for versioning. Access: URLs and PDFs of vendor materials, any A/B test reports, analytics exports (CSV) that correspond to the timeframes the vendor cites. Skills: Basic spreadsheet formulas (SUM, AVERAGE, %, VLOOKUP/INDEX-MATCH), the ability to copy raw numbers from source docs into a tracking sheet, and familiarity with the metrics you actually pay attention to (e.g., conversion rate, ARPU, CAC). Template: Create a "Citation Source Tracker" spreadsheet with the column headers shown below. Duplicate this as a new sheet for each vendor pitch.
Recommended tracker columns (create these exactly so you can filter/sort):
- Claim ID (unique, e.g., VEND1-C1) Claim Text (verbatim) Metric (e.g., conversion rate) Claimed Value (numeric) Baseline Value (numeric, if provided) Source Type (PDF, slide, web page, analytics export) Source URL / File Screenshot Filename (timestamped) Extraction Method (manual / API / spreadsheet) Retrieval Date Confidence Rating (1–5) Notes (assumptions, discrepancies)
Step-by-step instructions
Step 1 — Define the claim and the testable metric
Start by writing the vendor's claim in a single sentence you could measure. Example: "Vendor X increased conversion rate by 30% in 90 days." Convert that to a testable expression: "Conversion_rate_post / Conversion_rate_pre = 1.30 in the 90-day window." Put that sentence into Claim Text and set Metric = conversion rate.
Step 2 — Create a unique Claim ID and capture source metadata
Give each claim a short alphanumeric ID (VENDX-C1). Record the exact source URL or attach the PDF. Save a timestamped screenshot and put the filename in the tracker. Include where on the slide/report the number appears (slide 7, bottom right). Add Retrieval Date. This makes later verification trivial.
[Screenshot placeholder: capture the slide showing the 30% claim; filename: VENDX-C1_2025-10-13_1032.png]
Step 3 — Extract raw values and baseline data
Don't rely only on the "percent improvement" line. Get both the numerator and denominator when possible. If a vendor says "30% lift," ask for the pre and post conversion rates, sample sizes, and timeframe. Record the Baseline Value and Claimed Value separately.
Example extraction rows:
- Baseline conversion rate: 2.0% (obtained from internal analytics export dated 2025-06-01 to 2025-08-31) Post conversion rate: 2.6% (vendor-provided analytics export dated 2025-09-01 to 2025-11-29)
Compute the claim in-sheet: Claimed Ratio = Post / Baseline. Add a column for "Calculated Ratio" so you can see whether Claimed Value matches your derived ratio.
Step 4 — Standardize timeframes and definitions
Vendors often compare non-equivalent periods. Standardize timeframes and metric definitions before you accept numbers. In your tracker add columns: Period Start, Period End, Metric Definition (e.g., "unique visitors with completed checkout / total unique visitors"). If definitions mismatch, mark Confidence Rating lower and note the discrepancy.
Step 5 — Validate source credibility
Use a short checklist for each source:
- Is the source primary (analytics export) or secondary (slide citing another study)? Prefer primary. Is there a timestamp or version? Prefer sources with retrieval dates and raw CSV attachments. Are sample sizes provided? If not, request them before you accept a percentage lift.
Flag any claim that lacks a primary source. Add a Confidence Rating of 1 and escalate for follow-up.
Step 6 — Reproduce the calculation and run sensitivity checks
Recompute the vendor's math in your own sheet. Add a small sensitivity table next to the claim with +/- 10–20% adjustments for baseline and post metrics to show how fragile the percentage is to small data differences.
Example calculations in the sheet:
- Calculated Lift (%) = (Post - Baseline) / Baseline * 100 Sensitivity rows: Baseline ±5%, Post ±5% to produce a 2x2 table of possible lifts
Include the sensitivity table in the case study so readers can see how claims move when numbers shift slightly.
Step 7 — Build the short case study document with an evidence trail
Structure: executive summary, claim + measured values, methodology, data table, evidence links, and appendix with screenshots and raw CSV links. For each claim include the Claim ID and a link to the corresponding tracker row (or sheet tab).
Example mini-report snippet:
- Claim ID: VENDX-C1 Claim: 30% increase in conversion rate Our reconstruction: Baseline 2.0% → Post 2.6% → Calculated lift 30% (sample size Npre=120k, Npost=110k) Confidence: 4/5 (primary analytics exports provided; timeframe mismatch adjusted) Evidence: link to VENDX-C1_analytics_export.csv and VENDX-C1_2025-10-13_1032.png
Publish the case study with the tracker CSV or Google Sheet shared in read-only mode. The goal is reproducibility—not secrecy.
Step 8 — Ask targeted follow-up questions and predefine required deliverables
If the vendor cannot provide the primary exports, use a checklist to request:
- Raw CSV for baseline and post windows, with timestamps and event definitions. Full A/B test specification and randomization method, if applicable. Attribution windows and filters used (bots, internal test traffic).
Record their response and update Confidence Rating accordingly.
Common pitfalls to avoid
- Accepting percentages without denominators. A 50% lift sounds big until you see baseline = 0.2% and N=200. Always seek raw numerators and denominators. Comparing different timeframes. Week-over-week and quarter-over-quarter are not interchangeable. Convert everything to the same window or annotate the difference. Ignoring sample size and statistical significance. Vendors may present a point estimate without a confidence interval. Compute a simple z-test or at least show how lift changes with margin of error. Relying on secondary sources. Slides summarizing someone else’s data are one step away from hearsay. Track the chain: slide → report → raw data. Missing auditability. If you don’t save screenshots and timestamps, the vendor can change the slide and claim the original is gone. Save everything defensively.
Advanced tips and variations (expert-level insights)
These tips speed up evaluation and strengthen proof.
- Automate source capture for repeat evaluations. Use a small script or a browser extension to pull the text of a slide/web page and append it to your tracker CSV with timestamp and DOI-like unique ID. This reduces manual errors. Use persistent identifiers. If the vendor cites an industry report, capture DOI or archive the page in the Wayback Machine and include the archive link in Source URL. Version control your trackers. Store the tracker CSV in a Git repo or cloud versioning system. That way you can see when a value changed and who changed it. Pre-register the evaluation plan. Before accepting the vendor's numbers, propose a short verification plan (what you will compare and acceptable data formats). This shifts the burden of proof to the vendor. Thought experiment — The Invisible Baseline. Imagine a vendor claiming a 40% lift from a baseline that conveniently excludes a known traffic spike. Ask: how would the lift change if the spike is included? Run the sensitivity table with that spike reintroduced and show the revised lift. Thought experiment — The Cherry-Picker Vendor. Assume the vendor picked the single best-performing client among 20. Estimate the expected distribution of best-case lifts under random sampling and show how selection bias inflates the headline number. Even a simple simulation in a spreadsheet (random draws from a distribution) makes the point.
Troubleshooting guide
Problems you will encounter and how to fix them.
Problem: Vendor refuses to provide raw exports
Fix: Ask for masked or aggregated exports that keep PII safe. Offer an NDA or on-site verification. If refused, downgrade confidence and treat the claim as unverified. Document refusal in the tracker.
Problem: Numbers don't match between slide and export
Fix: Check definitions and filters first (e.g., conversion definition changed). If still mismatched, take a screenshot that highlights the discrepancy and ask the vendor to reconcile. Record both values with notes and date-stamped screenshots.
Problem: Small sample sizes with large claimed lifts
Fix: Compute margin of error and present it. If the confidence interval includes zero, treat the claim as inconclusive. Use the sensitivity table to show plausible lifts under reasonable variance.
Problem: Vendor cites third-party report but provides no link
Fix: Search for the report using the authors' names and keywords, archive the found report, and add the archive link to the tracker. If you cannot find it, flag the claim as unsupported.
Problem: Multiple claims across vendors that look identical
Fix: Compare Claim Text and Source URLs. If multiple vendors cite the same report, reference the original and treat all derivative claims as dependent on that single source. This prevents double-counting evidence.
Final checklist before you present to stakeholders
- Each headline claim has a Claim ID and at least one primary source (or a documented refusal). Screenshots are saved with timestamped filenames and linked in the tracker. Calculations that produce the headline numbers are visible in the sheet and can be copied into a reproducible cell range. Sensitivity analysis is included for every major claim to show fragility. The case study contains links to raw CSVs or is published with a read-only tracker so stakeholders can audit later.
Budget owners who have heard every pitch before need a simple truth: claims without a reproducible trail are opinions. Use a citation source tracker to convert vendor narrative into verifiable data. It takes a little upfront work, but once you have a template https://score.faii.ai/visibility/quick-score and a short automation layer, evaluating future pitches becomes a 15–30 minute evidence check instead of a leap of faith. When you present the numbers, you'll be offering a clear audit trail—not a promise.