I recently stumbled across a study suggesting that FF.net is going to die in 2019. Furthermore, the person responsible for said stumbling was claiming that all fanfiction follows the same trend - that fanfiction as a whole is dying out. This concerned me, so I gave the study a close reading.
Suffice to say, I was not impressed. The number of statistical errors and logical fallacies in the work was shocking. I re-ran the numbers, and double-checked their claims. To summarize the results, the original study is horribly wrong. Here's a blow-by-blow:
The Arbitrary 3000!
My first concern with the paper is encountered early on.
"We add an arbitrary 3000 to that number because accounts from 1 to 3000 are unavailable, and the account number generator did not account for it. What do we get? Since September 1998 fanfiction deleted over 20,500 users for infringement."
The addition of an arbitrary 3,000 users (14% of the total!) to the number of accounts deleted for infringement is a very serious issue. We have no way of knowing why those accounts were rendered unavailable, or why - it is entirely possible that they were reserved for authors, admins, or any one of a thousand other reasons. The fact that it is an entire block makes it vanishingly unlikely that they all were removed for infringement. (The exact statistics are left as an exercise for the reader - given 3,000 .85% chances of deletion, what are the odds that they'd all be removed?) This block cannot be considered removed-for-infringement, which changes a lot of the numbers concerning account deletions.
Deletions by administrators
A question not addressed by the survey is where the account deletions are happening. Are the accounts being deleted new or old, alive or dead, writers or readers? The paper mentions deletions due to policy changes, such as the bannings of MST and script-format fics, but that statistic would also change the numbers concerning the health of the site. A site removing new authors for being obvious trolls or advertisers would be a very different place from one picking on established, popular authors.
This also affects the statistics used in the study. Because we do not know why people were removed, that percentage could be (best case for the study) entirely added to the dead user population, or (worst case for the study) entirely added to the active-writers-from-2001 category. Either way, 20,000 is a very large number to casually throw out. For the sake of discussion, we will assume that all deleted users were in their own category (users-to-be-removed), however this is in no case a concession of the point.
The 2010 prediction
Some commenters have pointed out that the relative accuracy of the study's predicitons for 2010 are an indicator of the validity of the entire study. This is not a valid point. The predictions for 2010 were made when more than half of 2010 was over, and came from precisely two data points - the observed number of joiners in 2010 before the date of the study, and the observed date of the middle joiner over previous years. These statistics both come directly from the source study and are a very conservative estimate - to extrapolate from there to say that the entirety of the survey is entirely correct is bad statistics.
Is exponential growth the only sustainable growth?
My next main statistical concern comes from the following claim:
"The explanation is as follows: as the site grows, it needs a larger number of new accounts to sustain itself."
The study goes on into a confusing explanation of exponential growth. What it boils down to, though, is a claim that any website that does not realize exponential growth is doomed.
This is wrong.
Fanfiction.net makes its revenue from advertisements, on a per-pageview basis. Its expenses are bandwidth, storage, and salaries. Bandwidth, obviously, is also a per-pageview basis. Thus, it can be entirely offset with a fraction of the advertising revenue. This leaves the remaining fraction of revenue to cover salaries and storage. For a moment, let's assume that the two are comparable - as the site grows, the number of people it has on staff increase at a comparable rate.
Starting from the study's assumptions, total storage space needed is easy to calculate - it's the integral of the stories-posted-per-month curve. Per numbers earlier in the study, 32% of registered users have stories posted, and the population grows in what appears to be an exponential manner, ergo the total storage space (and thus expense) must also expand exponentially. Therefore, If the user base doesn't also increase exponentially, FF.net is inevitably doomed.
This is also wrong. It relies on several key assumptions:
First, that the stories posted rate will expand exponentially when the user base doesn't. Read those last two sentences carefully - it's classic cause/effect confusion. The rate at which stories are posted (and thus the rate at which storage needs expand) are directly chained to the rate at which new users join - and thus to the rate at which pageview revenue comes in.
The second assumption is that the cost of storage remains constant over time. This is also false - Moore's Law clearly states that it should drop off exponentially. In other words, storing an exponentially-expanding dataset will cost at worst a linearly expanding amount of money.
For a real-world example, consider Google. They dominate the search market to the point where they only grow as quickly as the internet does - which is decidedly non-exponential. According to the earlier assumption, they should be dying off rapidly... but they're not. Other examples could also be given, but the bottom line is that the status quo (no net gain of users) is entirely sustainable, thanks to the ever-decreasing storage cost bottom line.
Regression Models
Now, on to the regressions. This is where the most damning statistics errors occur.
First, the study treats its predictions for 2010 as facts. You do not do this. EVER.
Doing such is incredibly bad statistics. That error alone would be enough to make any of my statistics professors fail the entirety of it without further reading. The study also relies on this predicted 2010 value as an inversion point, where things start to trend downwards. This is bad and wrong. This is the single most damning failure of statistics in the entire study.
Secondly, they are using nine years of data (and a not-point of data) to try to predict what's going to happen ten years in the future. This is extrapolation so wild that it's entirely invalid. A decade is forever on the internet - sites have boomed and crashed in that much time. Trying to treat that wild a guess as fact demonstrates a damning lack of knowledge of what a regression really is.
Thirdly, the study uses a quadratic regression - a parabola - to model population change. This is extremely irregular. A parabola forces the population to go infinitely below zero as time approaches infinity - this is not a valid mapping to the real world. There is a reason population mathematicians work with exponential and logistic functions - a hundred years of population-change statistics have shown them to be the best fitting models.
Why did they use a parabolic model rather than an exponential or logistic one? The study never discusses it. My conversation with one of the authors also does not reveal why. The study ignores a hundred years of statistical practice for no discernible reason. This is also a damning failure of statistics. The only models they consider are a linear regression and a quadratic one.
I happen to have a graphing calculator with a statistics package. Let's crunch some numbers.
The statistic they give for CChange - Chained Change - is a key component of exponential growth. As the study explains, in perfect exponential growth, CChange would equal 1 for every year. Obviously, it doesn't. At the other end of the critical axis, a CChange of 0 means precisely that; 0 new members have joined the site, period. Because this is all in exponential space, the point at which the same number of members join this year as last year (linear growth) will appear to be an exponentially decreasing function.
Anyways, here's a bunch of functions, and how well they fit CChange:
Linear: The first function considered in the study, a linear regression produces a surprisingly good fit to the data. When the predicted value for 2010 is removed from the equation, the r2 value on the linear regression jumps up to .970 - a very close fit to the data. (r2 of 1.0 is a perfect fit, r2 of 0 is no fit.)
Quadratic: The quadratic regression with the predictions removed also has a very high r2 value. (r2 of .972) Here's where it gets interesting. The predicted value in the study is a parabola that opens downwards- if you recall algebra, as long as the X2 term is negative, it's a downwards parabola. However, a quadratic regression without the predicted value has a positive X2 term! (0.002 X2, specifically). In other words, this predicts quadratically increasing growth for all time!
Exponential: A straight exponential-increase model does not fit the CChange data. r2 is .887, well below the fit for the linear or quadratic functions.
Power function: The best-fitting power function for the data has an r2 of .899, which is similarly poor.
It's important to keep in mind that CChange is in exponential space - a linear function for CChange correlates to an exponential growth in the real-world population size. As you can see, once the estimate for 2010's growth is removed, the outlook for FF.net is very good - both the linear and quadratic functions, both of which fit the data very well, predict continuing exponential growth for all time.
Just for fun, let's look at something more real - the population of the site itself, per year. CChange is a value derived from nothing but the population figures, ergo the same conclusion should be supported by statistical analysis of the original population numbers.
Sure enough, it is! The best fit for the population data, with an r2 value of .997, is the power function y = 139098 x1.219. An exponential function also fits the data well, with an r2 value of .93. Both of these are unlimited growth functions, with the power function actually increasing faster than the exponential growth model which was claimed earlier to be required for the site to sustain itself.
To conclude this interlude on regressions, the study made serious errors of statistics, leading to irrational and invalid conclusions. Their prediction that FF.net will die by 2019 is based on massive statistical errors and wild extrapolations, and cannot be defended.
Member Permanency (How long will you last?)
And then the errors keep happening. The paper's final segment is concerning the permanence of members on the site. They look at the statistics concerning how long members will stay on FF.net, based on past results.
Their starting data is quite simple - a table of, per year (from 2002), the odds that your account remains active, sorted by whether or not you have published fics. This number is, presumably, calculated by comparing the number of active accounts to the number of inactive accounts for each year.
So far, so good. However, the paper once again dives into the minefield of bad statistics. The paper, again without any discussion of the alternatives, tries to fit a quadratic regression to the data, with only the footnote "(Works for values up to 10 years)".
This is, once again, bad statistics.
People with too much time on their hands (I.E, me) have noticed that their parabolic model bottoms out at the six-year mark and then starts to increase. This is not possible. Why? Calculus time.
Each percentage given is an absolute probability that you'll still be around by that many years in the future. It's not relative to the year before. The rate of people leaving the site per year of seniority is the derivative of this function. At first, it's negative - as people get more and more seniority, they tend to leave. Once you get past the six-year mark, though, the derivative becomes positive. In other words, people are joining the site with more than six years of seniority at the time they join.
This does not work.
Why did the paper overlook exponential regressions for the data? They do not say. Exponential regressions are a commonly used function to fit this sort of data, along with power functions, and my favorite, the Poisson distribution. Bottom line, they're using an entirely invalid function to fit the data.
There's an even more fundamental problem, though. The total study was based on 1100 samples. Per their numbers for ff.net membership, that's a 0.005% sample size. This means that their 6.4% of remaining people from 2002 is based on a population of... 16 people. Their number for 2003 comes from a population of 25. 2004 comes from a whole 37 people. At those sizes, statistical analysis is all but meaningless. Individual variations between people (a whole one person still active from 2002!) overwhelm any trends that could be observed.
Conclusions
In conclusion, I reject the claim that FF.net is going to reach a standstill in 2019. I reject the claim that a user's likelyhood of remaining active on the site can be modeled with a polynomial, and finally, I absolutely reject any claim that fanfiction itself is dying based on this study. Even if every claim made here were true, it does not equate to the death or even slowing down of fanfiction at large. The internet didn't die when Lycos went under, aviation didn't end when Voisin threw in the towel, and fanfiction will survive even if FF.net dies.
I also find the level of statistical knowledge exercised by FFnResearch horrifyingly poor. Please, for the sake of the internet, stop writing and go learn what on earth you're trying to say.
No comments:
Post a Comment