The computational graph is distinct from the systems it interacts with —tensors, modules, and optimizers. Its nodes are of class torch.autograd.Function (code here). Autograd constructs the graph as operations are performed, such that the parameters are leaf nodes.
By the end of the forward pass, the final operation is the root of a DAG. Hence, during the backwards pass, it can simply traverse from the root to the leaves, computing the gradient of each function with respect to its intermediate variables.