Adagrad optimizer and test - #20
Conversation
| end | ||
|
|
||
| # build the net | ||
| layers = Layer[InputLayer((1,2)), DenseLayer(2), Sigmoid(), DenseLayer(1), Sigmoid()] |
There was a problem hiding this comment.
Could you explain for a little bit why you need the last two layers? (i.e. DenseLayer and Sigmoid?)
There was a problem hiding this comment.
They are the output layer of the XOR perceptron.
There was a problem hiding this comment.
I see, I think the last layer might not be quite necessary, but as long as it works.
| cache = [] | ||
| for i = 1:length(net.layers) | ||
| layer = net.layers[i] | ||
| param = getParam(layer) |
There was a problem hiding this comment.
I'm thinking that there might be a better way to check whether a layer is a learnable layer. Maybe you could experiment on using type information?
There was a problem hiding this comment.
I tried "typeof(layer) <: LearnableLayer" but it doesn't work. I am not sure if those layers, like DenseLayer, are declared as LearnableLayer.
| g = grad[j] | ||
| @assert size(c) == size(p) && size(c) == size(g) | ||
| c = c + g.^2 | ||
| # not sure |
There was a problem hiding this comment.
What's this comment referring to? Not sure about the smoothing epsilons?
There was a problem hiding this comment.
At first I was not sure about declaring iteration as a parameter and later I forgot to delete the comment. Resolved.
| @assert size(c) == size(p) && size(c) == size(g) | ||
| c = c + g.^2 | ||
| # not sure | ||
| p = p - this.base_lr(this.iter) * g ./ (sqrt(c) + 1e-10) |
There was a problem hiding this comment.
It would be nice if we could put this 1e-10 epsilon as a parameter for this layer.
2 participants
No description provided.