Linear Classifier for Image Classification
Dataset: CIFAR10 [Alex Krizhevsky, “Learning Multiple Layers of Features from Tiny Images”, Technical Report, 2009.]
$$ $$f(x,W) = Wx+b$$ $$
It flatten a tensors into vector
Hard cases for a linear classifier
From Linear Classifiers to (Non-linear) Neural Networks
\(f = W_2max(0,W_1x)\)
Fully connected layer
3 layers
\(f= W_3max(0, W_2max(0,W_1x))\)
The function max(0, z) is called the activation function.
What if without the activation function?
- The model will be linear.
From Neural Networks to “Deep Learning”
CNNs
- CNNs are made up of neurons with learnable weights, as other to regular Neural Networks
- CNN architecture assumes that inputs are images
- Using specific assumptions for images
- So that we have local features
- Which allows us to
- encode certain properties in the architecture that makes the forward pass more efficient and
- significantly reduces the number of parameters needed for the network
Convolutional Neural Networks (CNNs)
CNN Architecture
- Convolution Layer
- Pooling Layer
- ReLU Layer, and
- Fully-Connected Layer
- Dropout layer（randomly deactive some perceptrons)
- Output layer
- The output of the Conv layer can be interpreted as holding neurons arranged in a 3D volume.
- The Conv layer's parameters consist of a set of learnable filters. Every filter is small spatially (along width and height), but extends through the full depth of the input volume.
- During the forward pass, each filter is slid (convolved) across the width and height of the input volume, producing a 2-dimensional activation map of that filter.
- The output of the Conv layer can be interpreted as holding neurons arranged in a 3D volume.ction) when they see some specific type of feature at some spatial position in the input.
CNN: Pooling Layer •
The function of pooling layer
- to progressively reduce the spatial size of the representation to reduce the number of parameters and computation in the network, and hence to also control overfitting
- The Pooling Layer operates
- independently on every depth slice of the input and resizes it spatially, typically using the MAX operation (ie: max pooling)
- The most common form is a pooling layer with filters of size 2x2 applied with a stride of 2, which downsamples every depth slice in the input by 2 along both width and height, discarding 75% of the activations
CNN: Output Layer
- The output layer produces the probability of each class given the input image
- This is the last layer containing the same number of neurons as the number of classes in the dataset
- The output of this layer passes through a Softmax activation function to normalize the outputs to a sum of one:
$$ $$\sigma (x_j) = \frac {e^{x_J}}{\sum_i e^{x_i}}$$ $$
Summary of CNNs
- ConvNets stack CONV,POOL,FC layers
- Trend towards smaller filters and deeper architectures
- Trend towards getting rid of POOL/FC layers (just CONV)
- Historically architectures looked like [(CONV-RELU)*N-POOL?]*M-(FCRELU)*K,SOFTMAX, where N is usually up to ~5, M is large, 0 <= K <= 2.
- but recent advances such as ResNet/GoogLeNet have challenged this paradigm
CNN: Training
- A loss function is used to compute the model’s prediction accuracy from the outputs
- Most commonly used: categorical cross-entropy loss function
\(-\sum_{c=1}^My_{o,c}\log(p_{o,c})\)
- The training objective is to minimise this loss
- The loss guides the backpropagation process to train the CNN model
- Gradient descent based methods, such as Stochastic gradient descent and the Adam optimizer, are commonly used algorithms for optimisation
Batch Normalization
- Enables stable training
- Reduces the internal covariate shift (ICS)
- Accelerates the training process
- Reduces the dependence of gradients on the scale of the parameters
Choice of Optimizers
- Choosing right optimizer helps to update the model parameters and reducing the loss in much less effort
- Most DL frameworks supports various optimizers:
- Stochastic Gradient Descent (SGD)
- Momentum
- Nesterov Accelerated Gradient
- AdaGrad
- AdaDelta
- Adam
- RMSProp
Tuning Hyperparameters
- Hyperparameters are all parameters which can be arbitrarily set by the user before starting training
- Hyperparameters are like knobs or dials of the network (model)
- An optimization problem: We aim to find the right combinations of their valueswhich can help us to find either the minimum (e.g., loss) or the maximum (e.g.,accuracy) of a function
- Many hyperparameters to tune:
- Learning rate
- No. of epochs
- Dropout rate
- Batch size
- No. of hidden layers and units
- Activation function
- Weight initialization …
