Mercurial > hg > tvii

annotate tvii/deep.py @ 80:3c7927f59b05

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
notes to self re deep neural networks
author Jeff Hammel <k0scist@gmail.com>
date Sun, 17 Dec 2017 13:43:42 -0800
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
80
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
1 """
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
2 Deep neural networks
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
3
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
4 Forward propagation for layer `l`
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
5
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
6 Input: a[l-1]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
7
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
8 Output: a[l], cache(z[l] {w[l], b[1]})
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
9
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
10 z[l] = w[l] ...
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
11
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
12 ---
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
13
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
14 Backward propagation for layer `l`:
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
15
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
16 Input: da[l]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
17 Output: da[l-1], dW[l], db[l]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
18
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
19 dz[l] = da[l]* g[l]'(z[l])
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
20 dw[l] = dz[l] a[l-1]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
21 db[l] = dz[l]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
22 dz[l-1] w[l].T dz[l]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
23
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
24 dz[l] = w[l+1].T dz[l+1] * g[l]' ( z[l] )
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
25
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
26 =>
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
27
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
28 dZ[l] dZ[l] * g[l]' ( Z[l] )
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
29
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
30 dW[l] (1/m) dZ[l] A[l-1].T
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
31
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
32 db[l] = (1/m) np.dum(dZ[l], axis=1, keepdims=True)
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
33 dA[l-1] = W[l].T * dZ[l]
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
34
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
35
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
36 For the final layerL
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
37 da[l] = - (y/a) + (1 - y)/(1-a)
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
38 dA[l] = (-(y(1)/a(1) + (1 - y(1))/(1 - a(1)) # first training example
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
39 ...)
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
40
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
41
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
42 The weight matrix for layer `l`, W[l] is
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
43 of the shape (n[l], n[l-1])
3c7927f59b05 notes to self re deep neural networks
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
44 """