Rating divergence: two raters, two answers

Imagine two examiners grading the same student. One tests mathematics and science, the other tests history and languages, and each gives the subjects different weight. Their grades will differ, and both may be fair.

ESG raters are in a similar position. They choose different issues, measure them in different ways and weight them differently. So the same company can be a leader for one rater and an average performer for another.

This matters because ratings drive real decisions: which companies enter ESG funds and indices, which bonds get bought, and how companies are judged by boards and the media. If ratings disagree, those decisions depend partly on which rater an investor happens to use.

The research: "Aggregate Confusion"

The best-known study is by Florian Berg, Julian Kölbel and Roberto Rigobon, published in the Review of Finance in 2022. They compared ratings from six providers: KLD, Sustainalytics, Vigeo Eiris, RobecoSAM, Asset4 and MSCI.

• The average correlation between pairs of overall ESG ratings was 0.54. The highest pair was 0.71 and the lowest 0.38.
• Agreement was weakest on governance: the average correlation was 0.53 for environment, 0.42 for social and only 0.30 for governance.
• They found a "rater effect": a rater's overall view of a company tends to colour how it scores the company's individual categories.

Three sources of disagreement

The study split the divergence into three parts:

The key finding is that disagreement is mostly about measurement, not about values or weights. Better and more comparable data would narrow it.

Other causes

• Data timing: raters update at different times and may use different years of data.
• Missing data: raters fill gaps with different estimates.
• Controversies: raters treat the same event differently in severity and duration.
• Industry classification: a conglomerate may sit in different peer groups.
• Relative versus absolute scales: a relative rating and a risk rating answer different questions.

Rewritten history

A second study by Florian Berg, Kornelia Fabisik and Zacharias Sautner compared two downloads of Refinitiv ESG (formerly Asset4) scores, from September 2018 and September 2020, for the same firm-years. After a methodology change applied to past data, not a single score was the same across the two versions. About 13% of observations were upgraded and 87% downgraded. In one test, the rewritten scores showed that high-scoring firms did much better during Covid-19, while the original scores showed no such effect. The lesson: backtests built on today's version of historical scores can overstate how well ESG strategies would have worked.

SEBI's attempt to narrow the gap

• Core ratings on assured data. SEBI requires registered providers to offer Core ratings built only on assured data, mainly BRSR Core. Using the same assured inputs should reduce measurement divergence.
• A common scale. Scores are given on a 0 to 100 scale.
• Disclosure of methods. Providers must disclose their methodology at summary level.
• Freedom of method. SEBI does not prescribe methods or weights, so scope and weight differences remain by design.

Using ratings that disagree

1. Read the drivers, not the headline. Find the few issues behind each rating.
2. Go back to the data. Check the BRSR figures that each rater relies on.
3. Form your own materiality view. Decide which issues matter most for the business and weigh the ratings against that view.
4. Use point-in-time data for backtests. Use scores as they stood at the time, not today's revised history.
5. Treat disagreement as information. Large divergence often signals poor disclosure or a contested issue worth investigating.