Mercurial > hg > tvii

annotate tvii/activation.py @ 70:351fc97bb996

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
add error computation + test functions
author Jeff Hammel <k0scist@gmail.com>
date Sun, 17 Dec 2017 13:22:44 -0800
parents 857a606783e1
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
41
8c1648921827 [activation functions] placeholder
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
1 """
8c1648921827 [activation functions] placeholder
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
2 activation functions
8c1648921827 [activation functions] placeholder
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
3 """
8c1648921827 [activation functions] placeholder
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
4
43
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
5 # tanh:
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
6 # g(z) = tanh(z) = (exp(z) - exp(-z))/(exp(z) + exp(-z))
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
7 # g'(z) = 1 - (tanh(z))**2
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
8
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
9 # ReLU
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
10 def ReLU(z):
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
11 return max((0, z))
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
12
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
13 def ReLUprime(z):
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
14 return 1. if z > 0 else 0.
2f0caec46e26 [activation functions] add ReLU
Jeff Hammel <k0scist@gmail.com>
parents: 41
diff changeset
15