The Z Files: Flaws and Limitations of Valuation Theory

It's no secret: projections are flawed. Seventy percent accuracy is bandied about, though it's not exactly clear what that means. Regardless, projection theory is not an exact science.

On the other hand, the efficacy of valuation theory isn't challenged, at least nowhere even close to the extent of projection theory. Here's the dirty little secret: it should be.

Valuation theory is very much flawed, yet so many treat projected auction prices, the resultant draft rankings and even end-of-season earnings as gospel, a static number.

The flaws go way beyond the inaccuracy of the projections fueling the algorithm-driven system. Even if projections were magically known, the resultant "value" isn't correct. Allow me to explain…

Value is the most misused term in fantasy baseball, well, all fantasy sports. Value is past tense. We don't know a player's value until after the season. Yet, many will contend a good draft or auction is all about getting value. A perceived good pick or buy is deemed a great value. After an MLB trade, everyone wants to know how a player's value will change.

I get it. I know what everyone means. Once a year I blow off a little steam and make this challenge. When you see or hear "value" in fantasy analysis, substitute "potential". Around 95 percent of the time, the statement makes more sense. The rest of the time replace it with "price". At the end of day, it doesn't matter, people know what you mean when you say "value". My point is, using the other terms hammers home that value isn't actually known, that we're talking about potential stats, potential points, etc. It's about the mindset. Value connotes rigidity. Potential serves to remind us that we don't know what will happen, which could lead to stronger roster construction. Sorry for the digression, but at least you're safe from the rant for another year, unless you listen to the RotoWire Friday podcast or the shows I co-host on Sirius XM Fantasy or MLB Network Radio, as I'm sure at some point I'll vent on those platforms.

Anyway, back to the flaws of potentialation theory. (OK, here it doesn't work). The flaws with valuation theory run deep, irrespective of the method. In fact, each of the three conventional methods (SGP, PVM and Z-scores) all have intrinsic foibles which will be discussed later. There are several shortcomings common to all approaches. Some of these are theoretical, some more technical and mathematically driven.

Something common to all valuation methods is the presumption the draft-worthy pool is composed of exactly the number of players needed to fill each team's active roster. For example, in a 12-team league with 14 active hitters and nine active pitchers, 168 batters and 108 hurlers are priced at $1 minimum. The problem is, you're not just drafting or buying the player; you're filling the roster spot. The player you choose will likely occupy the spot for the majority (if not the entire) season, but often the spot is populated with multiple players, emanating from reserve or waivers/free agency. Yet, the dollar value derived from the player's projection presumes he's the sole occupant of that roster spot.

Related to this is players eligible at multiple positions will reside in multiple roster spots. Technically, in leagues with a utility slot, everyone fits this bill.

It's common advice to take both of the above into consideration and pay a little more than book value in a draft or auction, but it's rarely quantified. I took a shot last spring, but in general it's a feel thing for most drafters.

The rebuttal to the 'buying the roster spot and not the player' argument is the price based on the projection isn't expected earnings, but rather what it takes to roster the player in context to his performance relative to the rest of the players drafted or purchased. This is a cogent argument, but it isn't usually applied in this manner. Mike Trout isn't "worth" $49; that's the proportion of the overall budget his stats are worth in proportion to the rest of the draft-worthy pool, and there's a difference.

The counterpoint to this is that for the above price to be representative, the draft-worthy pool must consist of the players legal to be drafted in each specific league. Some leagues deem only players on the 25-man roster to be eligible. Some use the 40-man as the pool while others have more liberal guidelines. Of course, individuals can determine bid prices and rankings on a customized player pool, correcting this deficiency, but most take the output of their favorite valuation engine and use it, without consideration of the pool composition. Chances are, these engines include minor-league players expected to be called up and contribute draft-worthy stats, throwing off the prices in leagues where they're not legal to draft. Obviously, this doesn't even account for keeper leagues, which will be a topic for a future column.

Another valuation blemish is more technical. In order to be "valued", ratio stats need to be converted into counting stats. There are a few common processes, but none of them are perfect. The same BA, OBP, ERA, WHIP, etc. does not affect each team equally. The influence depends on the player's denominator (at-bats, plate appearances, innings) and the team total. The fewer the team AB or IP, the more the player's contribution helps or hurts. Yet, when bid prices or rankings are generated, the "value" of a player's ratio stats is assigned a static number.

Those are a couple of valuation shortcomings inherent to all methods. Now let's focus on the three common procedures and the flaw(s) with each.

Standings Gained Points

Abbreviated as SGP, this process assigns value based on how many points in the standings each player gains you in each category. For example, as will be explained in a moment, it took 7.1 homers to gain a point in the standings for last season's NFBC Main Event. Ergo, Manny Machado earned 4.5 SGP by swatting 32 homers (32/7.1).

In brief, this calculation is done across all the categories for each player with the total summed, so everyone earned "X" SGP. These are all lumped into the same pool and value is distributed in proportion to the percentage of SGP contribution. This is an oversimplification, as replacement level and marginal pricing are involved, but it sets up the explanation of the chief pitfall of SGP.

Below are the average standings from the 2019 NFBC Main Event. Target drafters can use it as a guide. For this purpose, they're necessary to compute the categorical SGPs.

Ratios stats will be excluded since the point can be made without them. The proper means of determining SGP is leaving out the highest and lowest category total, then determining the least squares fit for the remaining 13 totals. Doing so yields the following SGP:

From here, value is assigned according to percent contribution of player SGP to the aggregate draft-worthy pool. The theoretical flaw, however, is that you don't start gaining standing points until you eclipse the last place team in a category, and those totals aren't proportional. The result is, not all of a player's SGP actually help gain points in the standings.

Focus on homers and steals. The last place team in stolen bases amasses about half that of the top team, whereas the last place team in home runs accrues well over half, cheating stolen base contributors of value.

The best way to explain this is introducing the concept of barrier SGP. This is the number of SGP needed before beginning to move up in the standings. The math is the last place total divided by the category SGP. Here are the barrier SGP for the above standings.

Let's use a slugger and a speedster as examples, showing last season's stats and resultant SGP:

Carlos Santana

Mallex Smith

Smith and Santana accrued the same number of counting stats SGP, so the same number go towards overcoming the cumulative barrier SGP. The majority of Smith's SGP are derived from steals; however, they disproportionately help reach the home run barrier. The repercussion is some of the expected points gained from Smith's steals don't contribute to his value as they're helping reach the barrier SGP in the other categories. While it's true Santana's SGP help reach the stolen base barrier, fewer are needed compared to the stolen base barrier.

Because SGPs are lumped together, Smith's steals help overcome the home run barrier, which doesn't make sense. The mathematical effect is the value of stolen base specialists is lower than it should be using SGP methodology.

The same process can be done to pitchers, illustrating closers are unfairly penalized.

While this is a theoretical flaw, it's not felt by the market as conventionally, drafters prefer not to pay for speed, so the error turns out to be serendipitous. Still, it's a defect of SGP pricing.

For those assuming the accuracy of categorical SGP is a limitation, it's not one that has much impact. Each SPG can be off by 10 percent and the resulting value is usually only +/- $1, so virtually the same.

Percent Value Method

Named by our colleagues at BaseballHQ, PVM assigns value in proportion to each player's categorical contribution. Again, there are allowances for replacement level, but the idea is if the player contributes X percent of the homers in the draft-worthy pool, he's assigned X percent of the budget allocated to the home run category.

Doing such assumes spending the same amount in each category yields the same place in the standings, but it doesn't. The way to demonstrate this is converting the average standings above to the corresponding dollar amounts earned for each standings place. The following chart uses a $260 budget, with a 69 percent hitting split, distributed equally across all the categories. Showing my work:

($260 x 15 x .69)/5 = $538.20 per category.

 The red text shows the points accumulated with $38 or more aggregate earnings for each category, while the green displays the same for $35. For PVM pricing to be accurate, the same earnings should produce the same number of points, hence PVM methodology has limitations.

As an aside, it's fascinating to see the colored patterns are mirror images. As you likely intuit, there are game theory implications, another topic to be addressed at a later date.

Z-Scores

Z-scores are similar to SGP, with standard deviations the currency. Each player is compared to the average replacement player in each category. The notion being, the more standard deviations they are away from the baseline, the greater the contribution in each category. Just like SGP, the number of standard deviations is summed across all the categories and pooled together with value assigned proportionally to the player's contribution to the total pool.

While this method is mathematically elegant, there isn't a theoretical basis. That said, the theoretical basis of SGP and PVM were both shown to be fraught with errors. Still, there's nothing to show the relative distribution of standard deviations correlates to points. Though, of course, the better the stats, the more standard deviations and higher the associated price/ranking.

There are several flaws permeating valuation theory as a whole, along with specific weaknesses innate to each method. What good is it? Is assigning prices/values/potential/rankings a complete waste of time? No, of course not.

The key is understanding the limitations and not getting hung up on a static number carried out to two decimal places. Even the following statement isn't entirely accurate, but rankings or a price list should be viewed through a relative, not absolute, lens. Player A might not be worth $25, or expected to earn $25, but he's probably more of a help, in a big-picture sense, than a player with a $24 tag. Risk and roster construction come into play, but we're working in the proverbial vacuum.

The best use of rankings or a cheat sheet with prices isn't to take one from the top, or to bow out if bidding goes over this amount. It's to get a general feel for how similar players compare to each other, then get a sense of the market, then draft/bid accordingly. Unfortunately, there isn't going to be a follow-up column detailing the magic formula to do so. Some of this comes from experience, some is common sense, some is logic, but mostly it's knowing the player pool inside and out, not drafting a rank or clinging to a somewhat arbitrary dollar value.

The natural question manifesting from this discussion is, "If values aren't accurate, how much over the price is it OK to spend? How much can I 'overdraft' a player if I know he won't make it back to me?"

Again, there's no secret potion. If there were a means of putting an error bar around valuation, it wouldn't be flawed, it would just have a wide range of prices. The answer is more subjective than objective, but the bottom line is pay as much as you want while still leaving ample budget to surround the player with the support necessary to win. Bidding $58 on Ronald Acuna Jr. doesn't mean you expect him to return at least $58. It means you want his stats and feel you can add what's needed with the leftover $202 to win. In a way, if you think someone paid too much, it really means they didn't leave enough.

Having gone through this, I'll no doubt venture into some topics using valuation, with the implication the prices are accurate. I'll do my best to keep things in perspective, but the onus is on you, not only while digesting the Z Files, but when taking in all advice. Look at valuation through a prism cognizant of its limitations. Valuation isn't worthless, but it does require knowing the failings to better apply the outputs.