Tensor decomposition discovers latent structure in higher-order data sets and is the higher-order analogue of the matrix decomposition. Variations have proliferated, including symmetric tensor decomposition, tensor completion, higher-order singular value decomposition, generalized canonical tensor decomposition, streaming tensor decomposition, etc. All of these scenarios reduce to some form of nonconvex optimization. In this talk, we discuss the use of randomized optimization methods, including stochastic gradient methods and sketching. Such approaches show amazing promise, not only improving the speed of the methods but also the quality of the solution. We give examples of the intriguing successes as well as notable limitations of these methods. We close with a number of open questions, including important open theoretical challenges.