1

I have been designing a neural network to perform predictions on construction item costs. I've developed a core set of predictors that seem to describe the problem space well - they appear to be highly correlated with the dependent variable (the construction item cost) and generally uncorrelated with each other.

These predictors cover a range of features including:

  • Quantity of work
  • Time of year (month / quarter)
  • Location (by county or larger regions)
  • National construction cost index
  • Union worker prevailing wages
  • Material cost indices

The neural network architecture is multi-layer preceptron (MLP) trained using resilient back-propagation (RProp). Though I typically use less than ten inputs, the network has around 60-70, due to conversion of several inputs to 1-of-N form. I'm using only a single hidden layer and have generally restricted the neuron count from 9 to the number of inputs in the network. I iterate the training anywhere from 1000 - 8000 times, depending. I typically split the data 85% / 15%, test / train. I've not done cross-validation because I'm not yet comfortable that I really have the right number / combination of predictors to start comparing models for performance.

For evaluation, I use the SSE to keep an eye on bias / variance changes, and an overall "success" metric which records the number of times the network successfully predicts a cost within a given percentage of the actual value, typically 15%.

To this point, I've found a couple "sweet spots" in learning rates and hidden nodes that yield good results with the test data without overfitting the training data greatly. However, I seem to have hit a limit.

Right now, I can achieve 65% "success" on the test data (i.e., the network successfully predicts within 15% of the actual value 65% of the time), which is encouraging. I've achieved higher results, but the model starts to overfit the training data. That is, once the model hits about 75% success on the training data, generalization starts to degrade.

It seems I should be able to achieve better success on my test data than I am. But at this point, I have to admit that my testing has gotten a bit ad hoc and I'm not really sure how to go about more thoroughly searching for better parameters or identifying performance characteristics that might give me a clue as to what's wrong...

Joel Graff
  • 167
  • 1
  • 13
  • Have you looked into other algorithms than neural networks, like SVM, regression trees etc.? Or maybe applying meta-algorithms on top of your regressor? – alesc Apr 27 '15 at 13:17
  • I skipped on SVM, as it seems to be better suited for class prediction (whereas I'm doing numeric prediction). I can look up regression trees - not familiar with the term. On the point about meta-algorithms, can you provide a reference / examples? – Joel Graff Apr 27 '15 at 14:50
  • You can read this [Wiki](http://en.wikipedia.org/wiki/Meta_learning_(computer_science)) article. It is usually used in classification, where you supply a concrete and trained classifier and then you train a meta-algorithm on top of it. I have searched for meta-algorithms that can be applied for regression, but I haven't found anything useful. – alesc Apr 27 '15 at 15:02
  • Related: http://area51.stackexchange.com/proposals/93481/artificial-intelligence – kenorb Jun 27 '16 at 18:53
  • it maybe helpful to make the network deeper (this can boost MLP regression performance), it may also be worthwhile to experiment with different optimization algorithms (e.g. Adam, RMSprop). It may be helpful to predict log prices (as opposed to prices) in case there's a high price range. – Vadim Smolyakov Aug 13 '17 at 01:44
  • I did try increasing depth, trading off with the number of neurons in a single layer. I couldn't get markedly better performance by adding hidden layers. As for other optimization schemes, I was limited by what I had at hand, but did try a few. In the end, stacking networks proved to be most beneficial - the output of one was added to the original data set and fed into another. A pair of networks joined in this way achieved nearly 90% success. – Joel Graff Aug 14 '17 at 02:48

0 Answers0