ReLU activation function support

[hatsunearu]

Is there any chance that FANN will support the ReLU function as a possible activation function?

[mushketer888]

It is also strange to me that such popular library doesnt have ReLU.

[hatsunearu]

Well, I moved onto Tensorflow because FANN isn't quite flexible.
I did implement ReLU and LReLU on my fork so if you want to try it out, go ahead.

https://github.com/hatsunearu/fann

[mushketer888]

ReLU is like simple rectifier as we know. it would be too easy just to write it in fann_activation.h. I just dont know do i need to include derivative also (math stuff)

[hatsunearu]

Huh?
If you look at the diff on my fork, you can see what I did to include ReLU and LReLU. The derivatives are calculated with an IFDEF that computes the slope.

[hatsunearu]

master...hatsunearu:master

I need to fix the indents someday and I'll do a PR, but considering the original author hasn't pulled any of the PRs lately, I don't think it's worth.

[mushketer888]

wow thanks!

[geekgirljoy]

Steffen seems like a busy guy. Have you messaged him about it and asked him if he would review the code if you did a pull request? I agree that adding ReLU and LReLU would only improve FANN and make it more useful to people. I vote for a PR, FANN needs ReLU and LReLU 👍 :)

[hatsunearu]

well he'll get a notification for a PR if someone made one, but considering he hasn't replied to any of the recent ones I doubt he'll pull my change.

I'll make it anyways if you all want to. In the meantime, you can use my fork.

[hatsunearu]

pull request created: #74

[geekgirljoy]

Thanks for adding this :-)

[radioman]

when using these both or one, the training doesn't progress, it's stuck at one number usualy 0.5 ???

fann_set_activation_function_hidden(ann, FANN_RELU);
fann_set_activation_function_output(ann, FANN_RELU);

[hatsunearu]

try changing the weight initialization. you might also be running into the issue where the gradient "dies" because the ReLU has 0 gradient where x<0.

[MatthewScholefield]

Just wondering, isn't ReLU essentially the same as the LINEAR_PIECE activation function in FANN?

[geekgirljoy]

Well, I moved onto Tensorflow because FANN isn't quite flexible.
I did implement ReLU and LReLU on my fork so if you want to try it out, go ahead.

@hatsunearu By chance you wouldn't know where one might obtain a PHP port of Tensorflow would you? :-P

[hatsunearu]

@geekgirljoy Suuuuper late to the thread, but you need to make some program to interface your PHP webapp to Tensorflow yourself. I haven't done this before, but try something like this: https://www.tensorflow.org/serving/ and also try searching something like "serving tensorflow" on Google.

Depending on your actual model, it might be best to make PHP call a Tensorflow script written in Python, appropriately package the result in Python, then return that back to PHP which appropriately packages it up and serves to the user.