Computational Graphs
I want to read about computational graphs, as they are related to neural networks and DL.
Date Created:
1
16
References
Notes
Computational Graphs:
- The descriptive language of deep learning models
- Functional description of the required computation
- Can be instantiated to do two types of computation:
- Forward computation
- Backward computation
A node is a { tensor, matrix, vector, scalar } value.
An edge represents a function argument (and also data dependency). They are just pointers to nodes.
A node with an incoming edge is a function of that edge's tail node.
A node knows how to compute its value and the value of its derivative with respect to each argument (edge) times a derivative of an arbitrary input ,
Functions can be nullary, unary, binary, ... n-ary. Often they are unary or binary.
Computed graphs are directed and acyclic (usually).
Variable names are just labelings of nodes.
- Forward propagation
- Loop over nodes in topological order
- Compute the value of the node given its inputs
- Given my inputs, make a prediction (or compute an
error
with respect to atarget output
)
- Backward propagation
- Loop over the nodes in reverse topological order starting with a final goal node
- Compute the derivatives of final goal node value with respect to each edge's tail node
- How does the output change if I make a small change to the inputs?
Constructing Graphs:
- Static Declaration
- Phase 1: define an architecture (maybe with some primitive flow control like loops and conditionals)
- Phase 2: run a bunch of data through it to train the model and/or make predictions
- Pros
- Offline optimization/scheduling of graphs is powerful
- Limits on operations mean better hardware support
- Cons
- Structured data (even simple stuff like sequences), even variable-sized data, is ugly
- You effectively learn a new programming language (
the Graph Language
) and you write programs in that language to process data
- Examples: Torch, Theano, TensorFlow
Comments
You have to be logged in to add a comment
User Comments
There are currently no comments for this article.