How funds evaluate alternative data vendors

Reviewed 2026-09-08

Evaluating alternative-data providers: research fit, provenance, historical timing, permitted use and the evidence needed for an informed purchasing decision.

Start from the hypothesis, not the catalog

Define the decision the data is intended to support, the relevant population and the information that must be available at that decision's date. For example, a study of hiring activity needs a definition of a posting, employer matching and an observation history before it can test a relationship with company performance.

Record the benchmark, evaluation period and acceptance criteria before comparing products. This makes the decision reviewable without assuming that a compelling demonstration establishes research value. Research fit and legal permission are separate questions; a useful-looking dataset can still be unsuitable for the intended use.

The five-point technical screen

Coverage: identify the observed population, recruitment or collection channels, missing groups and changes over time. Methodology: distinguish direct observations from estimates, weights, imputations and derived indicators. CFA Institute's research framework emphasizes input validity, sampling, representativeness and appropriate evaluation rather than treating a source label as a quality guarantee.^[3]

History: separate the event date, first observation, delivery date and later revision. Frequency: distinguish the cadence of collection from the delay before records become usable. A long historical file may have been reconstructed later and need not be point-in-time evidence for a backtest.

Validation: compare an appropriately permitted sample or other evidence against the population and period that matter to the project. A restricted sample may reflect privacy, licensing or commercial constraints, not necessarily poor quality. If the available evidence is insufficient to validate the intended use, record that limitation rather than treating the absence of a sample as proof of wrongdoing.

Back-testing with honest costs

Reconstruct the information and eligible universe available at each historical decision. Include relevant publication delays, delistings, corporate actions and implementation costs. For trading strategies, borrowing constraints, capacity and execution assumptions can matter as much as an improvement in a statistical forecast.

Use a held-out evaluation period and keep a record of the transformations and model choices tried. A prospective pilot can provide additional evidence, but simulated trades still rely on execution assumptions and can be influenced by model selection. Neither a backtest nor a pilot guarantees future returns.

Common failure modes

Check whether historical entities have disappeared from the delivered universe, whether past observations have been restated and whether a change in collection coverage could explain a measured trend. Compare performance across relevant populations and periods instead of assuming that validation on a small group generalizes.

Claims about the data-production process also need evidence. In September 2021, the SEC announced a settlement with App Annie and its co-founder concerning misleading statements about how app-performance estimates were derived. The company and founder settled without admitting or denying the findings.^[2] The case illustrates a specific provenance and representation failure, not a conclusion about every provider or dataset.

Compliance and legal red lines

The SEC's April 2022 examination-staff alert identifies weaknesses in documented alternative-data diligence, assessment of collection terms and legal obligations, and consistent application of controls. It also notes missing processes for reassessment after changes in collection practices or the passage of time.^[1]

Assess the actual collection chain, permissions, privacy obligations and potential receipt or misuse of material non-public information. Exclusivity is a contract characteristic, not a quality or compliance certification. The alert is a staff statement, not a new rule or approval of any source; legal conclusions depend on the relevant facts and jurisdiction.^[1]

Commercial levers beyond price

Compare the product and permitted use, not just a headline price. Specify the entities, geography, history, delivery, users, exports, retention, redistribution and derived-product rights included in an offer. Clarify how corrections and methodology changes are communicated, and what happens to historical copies when access ends.

Do not assume that a multi-year term, quarter-end timing or a successful pilot guarantees a discount or a particular contract. Document the agreed conditions and the evidence needed for a purchasing decision. A provider's commercial promise should be distinguished from the buyer's own validation results.

Operational fit

Test delivery formats, schema changes, entitlements and failure handling using the tools the team will actually operate. Measure the effort needed for entity mapping, revisions, monitoring and support. There is no universally best format or delivery platform independent of those requirements.

A warehouse-native listing can simplify access without eliminating validation, transformations, monitoring or licensing work. Confirm the division of responsibilities between publisher, marketplace and buyer, including any compute or export costs.

From pilot to production

Choose a pilot period appropriate to the event frequency, seasonality and evaluation target. One quarter is not a universal standard, and rare events may require other evidence or a longer observation period. State the decision criteria and the limits of what the pilot can establish.

Assign responsibility for freshness, schema changes and source reassessment after adoption. Preserve documentation showing that the agreed checks were carried out; the SEC alert specifically describes deficiencies where advisers could not demonstrate consistent diligence or identify when it needed to be repeated.^[1]