The hard part is keeping it simple. Says Farmer, "The more complex the problem is, the simpler the models that you end up having to use. It's easy to fit the data perfectly, but if you do that, you invariably end up just fitting to the flukes. The key is to generalize." Prediction machinery is ultimately theory-making machinery -- devices for generating abstractions and generalizations. Prediction machinery chews on the mess of seemingly random chicken-scratch data produced by complex and living things. If there is a sufficiently large stream of data over time, the device can discern a small bit of pattern. Slowly, the technology shapes an internal ad hoc model of how the data might be produced. The apparatus shuns "overfitting" the pattern on specific data and leans to the fuzzy fit of a somewhat imprecise generalization. Once it has a general fit -- a theory -- it can make a prediction. In fact, prediction is the whole point of theories. "Prediction is the most useful, the most tangible, and, in many respects, the most important consequence of having a scientific theory," Farmer declares.
Source: Wired, Interview with Doyne Farmer, “Cracking Wall Street“
Contributed by: Zaady