Search results
Jul 22, 2019 · Andrew Ng and Kian Katanforoosh (updated Backpropagation by Anand Avati) Deep Learning We now begin our study of deep learning. In this set of notes, we give an overview of neural networks, discuss vectorization and discuss training neural networks with backpropagation. 1 Neural Networks
This book delivers insights from AI pioneer Andrew Ng about learning foundational skills, working on projects, finding jobs, and joining the machine learning community. A practical roadmap to building your career in AI.
Introduction to deep learning. What is a (Neural Network) NN? Supervised learning with neural networks. Why is deep learning taking off? Neural Networks Basics. Binary classification. Logistic regression cost function. Gradient Descent. Derivatives. More Derivatives examples. Computation graph. Derivatives with a Computation Graph.
- Linear regression
- 1.1 LMS algorithm
- 1.2 The normal equations
- 1.3 Probabilistic interpretation
- L( ) = L( ; X; ~y) = p(~yjX; ):
- T x(i))2:
- T x(i))2;
- h (x))1 y
- Generalized linear models
- 3.2 Constructing GLMs
- 3.2.1 Ordinary least squares
- Generative learning algorithms
- 4.1 Gaussian discriminant analysis
- 4.2 Naive bayes
- 5.2 LMS (least mean squares) with features
- j (x(i)) j=1
- Support vector machines
- 6.4 The optimal margin classi er (option read-ing)
- 6.8 The SMO algorithm (optional reading)
- 7.2 Neural networks
- (x) + b[r]
- 7.3 Backpropagation
- 7.3.4 Two-layer neural network with vector notation
- 8.3.1 Preliminaries
- 9.2 Implicit regularization e ect
- 9.4 Bayesian statistics and regularization
- J(c; ) = X jjx(i)
- EM algorithms
- Pn 1fz(i) = jg
- Pn w(i)
- Pk p(x(i)jz(i) = l; ; )p(z(i) l=1 = l; )
- 11.3 General EM algorithms
- DKL(Qkpz) (11.15)
- 11.4 Mixture of Gaussians revisited
- Principal components analysis
- Independent components analysis
- 13.3 ICA algorithm
- 14.2 Pretraining methods in computer vision
- Reinforcement learning
- 15.1 Markov decision processes
- R(s0; a0) + R(s1; a1) + 2R(s2; a2) + :
- R(s0) + R(s1) + 2R(s2) + :
- E R(s0) + R(s1) + 2R(s2) +
- 15.4.2 Value function approximation
- Using a model or simulator
- V jj1
- F C C ; at B B C C @ @ t A A
- 16.3.2 Di erential Dynamic Programming (DDP)
- 16.4 Linear Quadratic Gaussian (LQG)
- P [f( )]
To make our housing example more interesting, let's consider a slightly richer dataset in which we also know the number of bedrooms in each house: Here, the x's are two-dimensional vectors in R2. For instance, x(i) is the 1 living area of the i-th house in the training set, and x(i) 2 is its number of bedrooms. (In general, when designing a learnin...
We want to choose so as to minimize J( ). To do so, let's use a search algorithm that starts with some \initial guess" for , and that repeatedly changes to make J( ) smaller, until hopefully we converge to a value of that minimizes J( ). Speci cally, let's consider the gradient descent algorithm, which starts with some initial , and repeatedly perf...
Gradient descent gives one way of minimizing J. Let's discuss a second way of doing so, this time performing the minimization explicitly and without resorting to an iterative algorithm. In this method, we will minimize J by explicitly taking its derivatives with respect to the j's, and setting them to zero. To enable us to do this without having to...
When faced with a regression problem, why might linear regression, and speci cally why might the least-squares cost function J, be a reasonable choice? In this section, we will give a set of probabilistic assumptions, under which least-squares regression is derived as a very natural algorithm. Let us assume that the target variables and the inputs ...
Note that by the independence assumption on the (i)'s (and hence also the y(i)'s given the x(i)'s), this can also be written n ) L( =
Hence, maximizing `( ) gives the same answer as minimizing
which we recognize to be J( ), our original least-squares cost function. To summarize: Under the previous probabilistic assumptions on the data, least-squares regression corresponds to nding the maximum likelihood esti-mate of . This is thus one set of assumptions under which least-squares re-gression can be justi ed as a very natural method that's...
Assuming that the n training examples were generated independently, we can then write down the likelihood of the parameters as ) L( = p(~y j X; ) n
So far, we've seen a regression example, and a classi cation example. In the regression example, we had yjx; N ( ; 2), and in the classi cation one, yjx; Bernoulli( ), for some appropriate de nitions of and as functions of x and . In this section, we will show that both of these methods are special cases of a broader family of models, called Genera...
Suppose you would like to build a model to estimate the number y of cus-tomers arriving in your store (or number of page-views on your website) in any given hour, based on certain features x such as store promotions, recent advertising, weather, day-of-week, etc. We know that the Poisson distribu-tion usually gives a good model for numbers of visit...
To show that ordinary least squares is a special case of the GLM family of models, consider the setting where the target variable y (also called the response variable in GLM terminology) is continuous, and we model the conditional distribution of y given x as a Gaussian N ( ; 2). (Here, may depend x.) So, we let the ExponentialF amily( ) distributi...
So far, we've mainly been talking about learning algorithms that model p(yjx; ), the conditional distribution of y given x. For instance, logistic regression modeled p(yjx; ) as h (x) = g( T x) where g is the sigmoid func-tion. In these notes, we'll talk about a di erent type of learning algorithm. Consider a classi cation problem in which we want ...
The rst generative learning algorithm that we'll look at is Gaussian discrim-inant analysis (GDA). In this model, we'll assume that p(xjy) is distributed according to a multivariate normal distribution. Let's talk brie y about the properties of multivariate normal distributions before moving on to the GDA model itself.
In GDA, the feature vectors x were continuous, real-valued vectors. Let's now talk about a di erent learning algorithm in which the xj's are discrete-valued. For our motivating example, consider building an email spam lter using machine learning. Here, we wish to classify messages according to whether they are unsolicited commercial (spam) email, o...
We will derive the gradient descent algorithm for tting the model T (x). First recall that for ordinary least square problem where we were to t T x, the batch gradient descent update is (see the rst lecture note for its deriva-tion): := +
by We often rewrite (x(j))T (x(i)) as h (x(j)); (x(i))i to emphasize that it's the inner product of the two feature vectors. Viewing i's as the new representa-tion of , we have successfully translated the batch gradient descent algorithm into an algorithm that updates the value of iteratively. It may appear that at every iteration, we still need to...
This set of notes presents the Support Vector Machine (SVM) learning al-gorithm. SVMs are among the best (and many believe are indeed the best) \o -the-shelf" supervised learning algorithms. To tell the SVM story, we'll need to rst talk about margins and the idea of separating data with a large \gap." Next, we'll talk about the optimal margin class...
Given a training set, it seems from our previous discussion that a natural desideratum is to try to nd a decision boundary that maximizes the (ge-ometric) margin, since this would re ect a very con dent set of predictions on the training set and a good \ t" to the training data. Speci cally, this will result in a classi er that separates the positi...
The SMO (sequential minimal optimization) algorithm, due to John Platt, gives an e cient way of solving the dual problem arising from the derivation of the SVM. Partly to motivate the SMO algorithm, and partly because it's interesting in its own right, let's rst take another digression to talk about the coordinate ascent algorithm.
Neural networks refer to broad type of non-linear models/parametrizations h (x) that involve combinations of matrix multiplications and other entry-wise non-linear operations. We will start small and slowly build up a neural network, step by step. A Neural Network with a Single Neuron. Recall the housing price prediction problem from before: given ...
(7.26) When is xed, then ( ) can viewed as a feature map, and therefore h (x) is just a linear model over the features (x). However, we will train the neural networks, both the parameters in and the parameters W [r]; b[r] are optimized, and therefore we are not learning a linear model in the feature space, but also learning a good feature map ( ) i...
In this section, we introduce backpropgation or auto-di erentiation, which computes the gradient of the loss rJ(j)( ) e ciently. We will start with an informal theorem that states that as long as a real-valued function f can be e ciently computed/evaluated by a di erentiable network or circuit, then its gradient can be e ciently computed in a simil...
As we have done before in the de nition of neural networks, the equations for backpropagation becomes much cleaner with proper matrix notation. Here we state the algorithm rst and also provide a cleaner proof via matrix cal-culus. Let
In this set of notes, we begin our foray into learning theory. Apart from being interesting and enlightening in its own right, this discussion will also help us hone our intuitions and derive rules of thumb about how to best apply learning algorithms in di erent settings. We will also seek to answer a few questions: First, can we make formal the bi...
The implicit regularization e ect of optimizers, or implicit bias or algorithmic regularization, is a new concept/phenomenon observed in the deep learning era. It largely refers to that the optimizers can implicitly impose structures on parameters beyond what has been imposed by the regularized loss. In most classical settings, the loss or regulari...
In this section, we will talk about one more tool in our arsenal for our battle against over tting. At the beginning of the quarter, we talked about parameter tting using maximum likelihood estimation (MLE), and chose our parameters according to n = arg max Y p(y(i)jx(i); ): MLE i=1 Throughout our subsequent discussions, we viewed as an unknown par...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
(17.2) We face a similar situations in the variational auto-encoder (VAE) setting covered in the previous lectures, where the we need to take the gradient w.r.t to a variable that shows up under the expectation | the distribution P depends on . Recall that in VAE, we used the re-parametrization techniques to address this problem. However it does no...
Standard notations for Deep Learning.pdf. File metadata and controls. 256 KB. Contains all course modules, exercises and notes of Deep Learning Specialization by Andrew Ng, and DeepLearning.ai in Coursera - Deep-Learning-AndrewNg-DeepLearning.AI/1 Neural Networks and Deep Learning/W1/1.
Learning deep energy models, Jiquan Ngiam, Zhenghao Chen, Pangwei Koh and Andrew Y. Ng. In Proceedings of the Twenty-Eighth International Conference on Machine Learning, 2011. [ pdf ]
People also ask
Who teaches deep learning at Coursera?
How good is deep learning?
What is a deep learning tutorial?
Who are the authors of multimodal deep learning?
Andrew NG Notes Collection. This is the first course of the deep learning specialization at Coursera which is moderated by DeepLearning.ai. The course is taught by Andrew Ng. Andrew NG Machine Learning Notebooks : Reading. Deep learning Specialization Notes in One pdf : Reading.
