(putting_workflow)=

Model building and expansion for golf putting

:::{post} Apr 2, 2022 :tags: Bayesian workflow, model expansion, sports :category: intermediate, how-to :author: Colin Carroll, Marco Gorelli, Oriol Abril-Pla :::

This uses and closely follows the case study from Andrew Gelman, written in Stan. There are some new visualizations and we steered away from using improper priors, but much credit to him and to the Stan group for the wonderful case study and software.

We use a data set from "Statistics: A Bayesian Perspective" {cite:p}berry1996statistics. The dataset describes the outcome of professional golfers putting from a number of distances, and is small enough that we can just print and load it inline, instead of doing any special csv reading.

:::{include} ../extra_installs.md :::

We start plotting the data to get a better idea of how it looks. The hidden cell contains the plotting code

After plotting, we see that generally golfers are less accurate from further away. Note that this data is pre-aggregated: we may be able to do more interesting work with granular putt-by-putt data. This data set appears to have been binned to the nearest foot.

We might think about doing prediction with this data: fitting a curve to this data would allow us to make reasonable guesses at intermediate distances, as well as perhaps to extrapolate to longer distances.

Logit model

First we will fit a traditional logit-binomial model. We model the number of successes directly, with

Here is how to write that model in PyMC. We use underscore appendices in our model variables to avoid polluting the namespace. We also use {class}pymc.MutableData to let us swap out the data later, when we will work with a newer data set.

We have some intuition that $a$ should be negative, and also that $b$ should be positive (since when $\text{distance} = 0$, we expect to make nearly 100% of putts). We are not putting that into the model, though. We are using this as a baseline, and we may as well wait and see if we need to add stronger priors.

We see $a$ and $b$ have the signs we expected. There were no bad warnings emitted from the sampler. Looking at the summary, the number of effective samples is reasonable, and the rhat is close to 1. This is a small model, so we are not being too careful about inspecting the fit.

We plot 50 posterior draws of $p(\text{success})$ along with the expected value. Also, we draw 500 points from the posterior predictive to plot:

The fit is ok, but not great! It is a good start for a baseline, and lets us answer curve-fitting type questions. We may not trust much extrapolation beyond the end of the data, especially given how the curve does not fit the last four values very well. For example, putts from 50 feet are expected to be made with probability:

The lesson from this is that

this appeared here in using

rather than

to calculate our expectation at 50 feet.

Geometry-based model

As a second pass at modelling this data, both to improve fit and to increase confidence in extrapolation, we think about the geometry of the situation. We suppose professional golfers can hit the ball in a certain direction, with some small(?) error. Specifically, the angle the ball actually travels is normally distributed around 0, with some variance that we will try to learn.

Then the ball goes in whenever the error in angle is small enough that the ball still hits the cup. This is intuitively nice! A longer putt will admit a smaller error in angle, and so a lower success rate than for shorter putts.

I am skipping a derivation of the probability of making a putt given the accuracy variance and distance to the hole, but it is a fun exercise in geometry, and turns out to be

where $\Phi$ is the normal cumulative density function, $R$ is the radius of the cup (turns out 2.125 inches), and $r$ is the radius of the golf ball (around 0.84 inches).

To get a feeling for this model, let's look at a few manually plotted values for $\sigma_{\text{angle}}$.

This looks like a promising approach! A variance of 0.02 radians looks like it will be close to the right answer. The model also predicted that putts from 0 feet all go in, which is a nice side effect. We might think about whether a golfer misses putts symmetrically. It is plausible that a right handed putter and a left handed putter might have a different bias to their shots.

Fitting the model

PyMC has $\Phi$ implemented, but it is pretty hidden (pm.distributions.dist_math.normal_lcdf), and it is worthwhile to implement it ourselves anyways, using an identity with the error function.

Prior Predictive Checks

We often wish to sample from the prior, especially if we have some idea of what the observations would look like, but not a lot of intuition for the prior parameters. We have an angle-based model here, but it might not be intuitive if the variance of the angle is given, how that effects the accuracy of a shot. Let's check!

Sometimes a custom visualization or dashboard is useful for a prior predictive check. Here, we plot our prior distribution of putts from 20 feet away.

This is a little funny! Most obviously, it should probably be not this common to putt the ball backwards. This also leads us to worry that we are using a normal distribution to model an angle. The von Mises distribution may be appropriate here. Also, the golfer needs to stand somewhere, so perhaps adding some bounds to the von Mises would be appropriate. We will find that this model learns from the data quite well, though, and these additions are not necessary.

This new model appears to fit much better, and by modelling the geometry of the situation, we may have a bit more confidence in extrapolating the data. This model suggests that a 50 foot putt has much higher chance of going in:

We can also recreate our prior predictive plot, giving us some confidence that the prior was not leading to unreasonable situations in the posterior distribution: the variance in angle is quite small!

New Data!

Mark Broadie used new summary data on putting to fit a new model. We will use this new data to refine our model:

This new data set represents ~200 times the number of putt attempts as the old data, and includes putts up to 75ft, compared to 20ft for the old data set. It also seems that the new data represents a different population from the old data: while the two have different bins, the new data suggests higher success for most data. This may be from a different method of collecting the data, or golfers improving in the intervening years.

Fitting the model on the new data

Since we think these may be two different populations, the easiest solution would be to refit our model. This goes worse than earlier: there are divergences, and it takes much longer to run. This may indicate a problem with the model: Andrew Gelman calls this the "folk theorem of statistical computing".

:::{note} As you will see in the plot below, this model fits the new data quite badly. In this case, all the divergences and convergence warnings have no other solution than using a different model that can actually explain the data. :::

A model incorporating distance to hole

We might assume that, in addition to putting in the right direction, a golfer may need to hit the ball the right distance. Specifically, we assume:

1. If a put goes short or more than 3 feet past the hole, it will not go in.
2. Golfers aim for 1 foot past the hole
3. The distance the ball goes, $u$, is distributed according to

where we will learn $\sigma_{\text{distance}}$.

Again, this is a geometry and algebra problem to work the probability that the ball goes in from any given distance:

it uses phi, the cumulative normal density function we implemented earlier.

This model still has only 2 dimensions to fit. We might think about checking on OVERSHOT and DISTANCE_TOLERANCE. Checking the first might involve a call to a local golf course, and the second might require a trip to a green and some time experimenting. We might also think about adding some explicit correlations: it is plausible that less control over angle would correspond to less control over distance, or that longer putts lead to more variance in the angle.

Fitting the distance angle model

This new model looks better, and fit much more quickly with fewer sampling problems compared to the old model.There is some mismatch between 10 and 40 feet, but it seems generally good. We can come to this same conclusion by taking posterior predictive samples, and looking at the residuals. Here, we see that the fit is being driven by the first 4 bins, which contain ~40% of the data.

A new model

It is reasonable to stop at this point, but if we want to improve the fit everywhere, we may want to choose a different likelihood from the Binomial, which cares deeply about those points with many observations. One thing we could do is add some independent extra error to each data point. We could do this in a few ways:

1. The Binomial distribution in usually parametrized by $n$, the number of observations, and $p$, the probability of an individual success. We could instead parametrize it by mean ($np$) and variance ($np(1-p)$), and add error independent of $n$ to the likelihood.
2. Use a BetaBinomial distribution, though the error there would still be (roughly) proportional to the number observations
3. Approximate the Binomial with a Normal distribution of the probability of success. This is actually equivalent to the first approach, but does not require a custom distribution. Note that we will use $p$ as the mean, and $p(1-p) / n$ as the variance. Once we add some dispersion $\epsilon$, the variance becomes $p(1-p)/n + \epsilon$.

We follow approach 3, as in the Stan case study, and leave 1 as an exercise.

This new model does better between 10 and 30 feet, as we can also see using the residuals plot - note that this model does marginally worse for very short putts:

Beyond prediction

We want to use Bayesian analysis because we care about quantifying uncertainty in our parameters. We have a beautiful geometric model that not only gives us predictions, but gives us posterior distributions over our parameters. We can use this to back out how where our putts may end up, if not in the hole!

First, we can try to visualize how 20,000 putts from a professional golfer might look. We:

1. Set the number of trials to 5
2. For each joint posterior sample of variance_of_shot and variance_of_distance, draw an angle and a distance from normal distribution 5 times.
3. Plot the point, unless it would have gone in the hole

We can then use this to work out how many putts a player may need to take from a given distance. This can influence strategic decisions like trying to reach the green in fewer shots, which may lead to a longer first putt, vs. a more conservative approach. We do this by simulating putts until they have all gone in.

Note that this is again something we might check experimentally. In particular, a highly unscientific search around the internet finds claims that professionals only 3-putt from 20-25ft around 3% of the time. Our model puts the chance of 3 or more putts from 22.5 feet at 2.8%, which seems suspiciously good.

Authors

• Adapted by Colin Carroll from the [Model building and expansion for golf putting] case study in the Stan documentation (pymc#3666)
• Updated by Marco Gorelli (pymc-examples#39)
• Updated by Oriol Abril-Pla to use PyMC v4 and xarray-einstats
• Updated by Benjamin T. Vincent to use az.extract in February 2023 (pymc-examples#522)

References

:::{bibliography} :filter: docname in docnames :::

Watermark

:::{include} ../page_footer.md :::