README.todo

cum_loss4x300.txt
  LR: 0.005
  decay 0.5
    RMS prop
  2 batch size
  1000 epochs
  5 layers, 300 neurons per layer. (363,300 parameters total)
  1 input, 10 output. 
  Got it down to ~150. (worse tho)1


cum_loss_largebatch_20.txt
  LR: 0.005
  decay 0.5
    RMS prop
  20 batch size
  1000 epochs
  5 layers, 300 neurons per layer. 
  1 input, 10 output. 
  Got it down to ~150. (worse)2

  This took 78.05 seconds.


cum_loss_largebatch_50.txt
  LR: 0.005
  decay 0.5
    RMS prop
  50 batch size
  1000 epochs
  5 layers, 300 neurons per layer. 
  1 input, 10 output. 
  Got it down to ~150. 3

  This took 41.3 seconds. 

cum_loss_lr_0.0005.txt
  LR: 0.0005
  decay 0.9
    RMS prop
  50 batch size
  1000 epochs
  5 layers, 300 neurons per layer. 
  1 input, 10 output. 
  Got it down to ~25. (best result).4

  This took 38.7 seconds. 


cum_loss_lr_0.00005.txt
  LR: 0.0005
  decay 0.9
    RMS prop
  50 batch size
  1000 epochs
  5 layers, 300 neurons per layer. 
  1 input, 10 output. 
  Got it down to ~25. (best result).4

  This took 38.7 seconds. 



cum_loss_newinit.txt
  init: stddev at 2, scale whole thing by 0.01.
  LR: 0.0005
  decay 0.9
    RMS prop
  50 batch size
  2000 epochs
  5 layers, 300 neurons per layer. 
  1 input, 10 output. 
  Got it down to ~200. (best result).4

  This is now giving normally looking spectrum 

  This took 39.7 seconds. 


cum_loss_good.txt
  init: stddev at 0.1. Don't scale anything.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  2000 epochs
  5 layers, 300 neurons per
  1input, 10 output
  Got it down to ~2.0
  This is giving good spectrum (taking residuals)
  This took 40seconds.

cum_loss_dev0.5.txt
  init: stddev at 0.5. Don't scale anything.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  2000 epochs
  5 layers, 300 neurons per
  1input, 10 output

  Got it down to ~3.5
  This is giving good spectrum (taking residuals)
  This took 40seconds.

cum_loss_dev0.1scaled.txt
  init: stddev at 0.1. scaled by 0.5
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  2000 epochs
  5 layers, 300 neurons per
  1input, 10 output

  Got it down to ~2.5
  This is giving good spectrum (taking residuals)
  This took 40seconds.

cum_loss_lr_0.000005.txt
  init: stddev at 0.`. Don't scale anything.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  2000 epochs
  5 layers, 300 neurons per
  1input, 10 output

  Got it down to ~1000
  This took 40seconds.

cum_loss_longrun.txt
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  3000 epochs
  5 layers, 300 neurons per
  1input, 10 output

  Got it down to ~1.6
  This took 40seconds.

cum_loss_small.txt
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  2000 epochs
  5 layers, 100 neurons per
  1input, 10 output

  Got it down to ~150
  This took 11seconds.

cum_loss_small10000.txt
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  10000 epochs
  5 layers, 100 neurons per
  1input, 10 output

  Got it down to ~.276
  This took 56seconds.

cum_loss_3x50.txt
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  10000 epochs
  3 layers, 50 neurons per
  1input, 10 output

  Got it down to ~13.3
  This took 32seconds.

cum_loss_3x50_long.txt
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  50 000 epochs
  3 layers, 50 neurons per (5,100)
  1input, 10 output

  Got it down to ~.11
  This took 32seconds.


cum_loss_5x20_long.txt (overwritten)
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  50 000 epochs
  5 layers, 20 neurons per (1,820)
  1input, 10 output

  Got it down to ~.23
  This took 145seconds.
  We are going to use this network. 


cum_loss_5x20_long.txt (rerun)
  init: stddev at 0.1.
  LR: 0.00005
  decay: 0.9
    RMS prop
  50 batch
  50 000 epochs
  5 layers, 20 neurons per (1,820)
  1input, 10 output

  Got it down to ~1.06
  This took 144seconds.
  We are going to use this network. 
  These weights are saved as the copy versions for backup. 
  Cool beans. It works!


  Now I need to generate the y data that is fixed. 


large_complex_net_5x20.txt
  Used larger dataset
  init: stddev at 0.1
  LR: 0.00005
  decay: 0.9
    RMS prop
  200 batch
  100,000 epochs
  5 layers (20 neurons per) (1,820)
  1 input, 100 output

  Got it down to ~200
  This took 4538.9797399seconds.
  We are now going to test it.