The Fed’s “Intelligent” Use of Data

By admin_45 in Blog

Simon Potter, EVP of the New York Fed and head of the Markets Group there, gave a talk last week with the provocative title “Models Only Get You So Far”.It seems to have been meant to be as a reality check, because the conference at which he spoke was billed as the “First Annual Joint Research Day on Quantitative Tools for Monitoring Macroeconomic and Financial Conditions”.

We read through the speech several times and found Potter’s comments instructive for 2 reasons:

• They give some useful insight into how the Fed thinks about recession probabilities and, therefore, monetary policy. Recall that the NY Fed is first among equals when it comes to Fed branches; their opinion carries significant weight at the home office in DC.
• He spends time considering how to effectively use model/data-driven analysis when making macro economic forecasts, and these observations are useful to all forms of financial analysis as well.

On the issue of recession forecasting:

• Potter starts by outlining the NY Fed’s current survey of market participants regarding the odds of a US recession both right now and in coming years. Only 2% of respondents think the US economy is in a recession currently and 12% believe a recession will occur in the next 6 months. 
  
  Respondents also believe the highest probability of a US recession lies in either 2020 or 2021, at 25% odds apiece.
• The cautionary tale: using survey data from 2007, Potter shows that even when the US was already in a recession (and a deep one, at that) respondents back then put the odds that the US economy was in a downturn at less than 50%. 
  
  The lesson here: either take survey-based recession odds with a big grain of salt, or (and Potter is sympathetic to this approach) simply assume that odds of +40% really mean something closer to 100%.
• Using the shape of the US Treasury yield curve as a recession indicator also benefits from “extremizing”, just like the market participant survey example. A Fed model developed in the mid 1990s and still in use today says the yield curve gives 23.6% odds of a US recession in the next 12 months. Potter makes an offhand comment that he likes to use 30% as the cutoff; that puts the 23.6% in a different light, to say the least…

On the issue of how to use data and modeling effectively:

• Potter is a big fan of the book “Superforecasting”, a work that grew out of a study done by the Intelligence Advanced Research Projects Activity (IARPA), a think tank within the US Office of the Director of National Intelligence. 
  
  After the US/allied intelligence communities’ flubbed call on WMDs in Iraq in the early 2000s, IARPA set up a crowd-sourcing system to collect predictions on a range of topics. They invited participants from both academia and the private sector to weigh in on topics like political outcomes and other world events. With some straightforward algorithmic tweaks IARPA was able to generate predictions more accurate than the “experts” within the US intelligence community.
• The formula that IARPA used to develop these superior insights: 1) find the individuals who had the best track record and 2) aggregate their forecasts based on how “independent” these “superforecasters” were. The second bit is the special sauce, of course, so there’s not much in the way of details.
  
  The key takeaway: it isn’t enough to have good forecasters in a crowd-sourced model. To leverage them for the best possible results, you need to make sure they come at the question at hand from a variety of disciplines and mental frameworks.

The upshot for how to use this observation in an investment setting:

• We all have our go-to sources who have made us money in the past.
• If 2 or 3 agree on an investment idea, we’re naturally inclined to go along.
• The better approach: consider how independent those sources are, both in background and investment discipline. The greater the degrees of separation between them, the better the chance their collective prediction will prove accurate.

Summing up: Potter closes his comments with a worthwhile warning: “… models and quantitative tools, like the ones we are discussing, can take us a long way. But they are designed to answer specific questions that are at best strong building blocks to the broader questions that policymakers grapple with. Being rigorous in our evaluation of these building blocks is critical, as is openness to other forms of information and approaches.”

Sources:

Potter speech: https://www.newyorkfed.org/newsevents/speeches/2019/pot190222

IARPA program details: https://www.iarpa.gov/index.php/research-programs/ace

A book by the lead researcher on the IARPA project, Philip Tetlock: https://www.amazon.com/Superforecasting-Science-Prediction-Philip-Tetlock/dp/0804136718/ref=sr_1_1?crid=1WVKHNEBBK192&keywords=superforecasting+the+art+and+science+of+prediction&qid=1551135998&s=books&sprefix=superfor%2Cstripbooks%2C136&sr=1-1