Computational Graphs

References

• Computational Graphs Lecture Cornell

Notes

Computational Graphs:

• The descriptive language of deep learning models
• Functional description of the required computation
• Can be instantiated to do two types of computation:
• • Forward computation
  • Backward computation

A node is a { tensor, matrix, vector, scalar } value.

An edge represents a function argument (and also data dependency). They are just pointers to nodes.

A node with an incoming edge is a function of that edge's tail node.

A node knows how to compute its value and the value of its derivative with respect to each argument (edge) times a derivative of an arbitrary input ,

Functions can be nullary, unary, binary, ... n-ary. Often they are unary or binary.

Computed graphs are directed and acyclic (usually).

Variable names are just labelings of nodes.

• Forward propagation
• • Loop over nodes in topological order
  • • Compute the value of the node given its inputs
  • Given my inputs, make a prediction (or compute an error with respect to a target output)
• Backward propagation
• • Loop over the nodes in reverse topological order starting with a final goal node
  • • Compute the derivatives of final goal node value with respect to each edge's tail node
  • How does the output change if I make a small change to the inputs?

Constructing Graphs:

• Static Declaration
• • Phase 1: define an architecture (maybe with some primitive flow control like loops and conditionals)
  • Phase 2: run a bunch of data through it to train the model and/or make predictions
  • Pros
  • • Offline optimization/scheduling of graphs is powerful
    • Limits on operations mean better hardware support
  • Cons
  • • Structured data (even simple stuff like sequences), even variable-sized data, is ugly
    • You effectively learn a new programming language (the Graph Language) and you write programs in that language to process data
  • Examples: Torch, Theano, TensorFlow