AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

G検定 · LESSON 3

Overview of deep learning11%（当サイト設定） · 6 topics

Learn the basics of neural networks, activation functions, loss functions, regularization, backpropagation and optimization methods.

3.1Neural networks and deep learning

Neural networks and deep learning (simple perceptron, multilayer perceptron, CPU/GPU/TPU)

Concept

A simple perceptron has only input and output layers and can solve only linearly separable problems (it can't solve XOR). A multilayer perceptron with hidden layers can handle nonlinear problems, and making the layers deep gives deep learning. Training needs huge amounts of matrix computation, so GPUs, good at parallel computation, and TPUs, developed by Google for deep learning, are used.

How it works / steps

  1. Understand the limit of the simple perceptron (linear separability)
  2. Understand that hidden layers enable nonlinear representations
  3. Distinguish what CPUs, GPUs and TPUs are good at

Workplace example

Move image training from CPU to GPU to greatly shorten training time with parallel matrix computation.

Common confusions and exam tips

Mini quiz

Q1. Which best describes the limitation of a simple perceptron?

  1. Its training computation was so large that computers of the time could not handle it
  2. Its output is continuous, so it cannot be used for classification at all
  3. It can solve only linearly separable problems, not problems such as XOR that aren't linearly separable
  4. It is limited to two inputs and cannot handle three or more
Show answer and explanation

Answer: C It can solve only linearly separable problems, not problems such as XOR that aren't linearly separable

A simple perceptron can solve only problems separable by a line (hyperplane). Adding hidden layers in a multilayer perceptron made nonlinear problems possible.

Q2. You are choosing hardware for deep learning computation. Which is the general-purpose processor that excels at running complex processing quickly in sequence and runs the OS and ordinary applications?

  1. TPU
  2. CPU
  3. FPGA
  4. GPU
Show answer and explanation

Answer: B CPU

A CPU runs complex processing in order on a few cores. GPUs and TPUs suit massive parallel computation. (Deciding point: a general-purpose processor good at sequential, complex processing.)

Reference links

3.2Activation functions

Activation functions (sigmoid, tanh, ReLU, Leaky ReLU, softmax, vanishing gradient problem)

Concept

Activation functions transform neuron outputs nonlinearly. The sigmoid (0–1) has a small maximum derivative of 0.25, so with deep layers gradients tend to vanish (the vanishing gradient problem). tanh (−1 to 1) has a maximum derivative of 1, an improvement. ReLU (0 below 0, unchanged when positive) rarely causes vanishing gradients and is widely used; Leaky ReLU gives a slight slope in the negative region too. Softmax is used in the output layer for multi-class classification, making outputs sum to 1 so they can be treated as probabilities.

How it works / steps

  1. Use ReLU-type functions by default in hidden layers
  2. For output layers: sigmoid for binary classification, softmax for multi-class, identity for regression
  3. If gradients vanish, review the activation functions and normalization

Workplace example

For classifying images of 10 kinds of products, use ReLU in hidden layers and softmax in the output layer to output each class's probability.

Common confusions and exam tips

Mini quiz

Q1. You are choosing an activation function. In hidden layers of a deep network, you want a function that rarely causes vanishing gradients and is cheap to compute. Which fits best?

  1. ReLU function
  2. Softmax function
  3. Identity function
  4. Sigmoid function
Show answer and explanation

Answer: A ReLU function

ReLU outputs the input as-is if positive (derivative 1) and 0 if negative, so gradients rarely vanish. For learning stalling on the negative side, use Leaky ReLU. (Deciding point: deep hidden layers, avoiding vanishing gradients.)

Q2. You are choosing an activation function. In the output layer of a regression predicting house prices (unbounded continuous values), you want to output values as-is. Which fits best?

  1. Softmax function
  2. Identity function
  3. ReLU function
  4. Sigmoid function
Show answer and explanation

Answer: B Identity function

For regression output layers, the identity function that returns the input as-is is standard. (Deciding point: regression output layer.)

Reference links

3.3Loss functions

Loss functions (mean squared error, cross-entropy, KL divergence, Contrastive Loss, Triplet Loss)

Concept

A loss function turns the gap between prediction and answer into a number, and training reduces it. Mean squared error is used for regression and cross-entropy for classification. KL divergence measures the difference between two probability distributions and is used in VAEs and elsewhere. Metric learning uses Contrastive Loss, which pulls similar pairs together and pushes different pairs apart, and Triplet Loss, which uses triples of anchor, positive and negative.

How it works / steps

  1. Choose a loss function suited to the task type (regression, classification, metric learning)
  2. Use KL divergence to compare probability distributions
  3. Use metric learning losses to learn similarity, as in face matching

Workplace example

For employee ID face matching, train with Triplet Loss so images of the same person sit close and images of different people sit far apart.

Common confusions and exam tips

Mini quiz

Q1. You are choosing a loss function. For face authentication, you want to learn embeddings from triplets of anchor, same person and different person so the same person's images are close and others' are far. Which fits best?

  1. Triplet loss
  2. Cross-entropy
  3. Kullback-Leibler divergence (KL)
  4. Mean squared error
Show answer and explanation

Answer: A Triplet loss

Triplet loss is a metric learning loss that pulls the anchor and positive together and pushes the negative away. Learning from pairs is contrastive loss. (Deciding point: learning distances from triplets.)

Q2. You are choosing a loss function. You want to measure how different two probability distributions are, as a term pulling a VAE's latent distribution toward a standard normal distribution. Which fits best?

  1. Mean squared error
  2. Triplet loss
  3. Kullback-Leibler divergence (KL)
  4. Cross-entropy
Show answer and explanation

Answer: C Kullback-Leibler divergence (KL)

KL divergence expresses the difference between two distributions (asymmetric). It is used as the regularization term in the VAE loss. (Deciding point: difference between probability distributions.)

Reference links

3.4Regularization

Regularization (L1 and L2 regularization, lasso regression, dropout)

Concept

Regularization prevents overfitting. L1 regularization adds the sum of absolute weights to the loss and tends to set unneeded weights to 0 (a feature selection effect); L1 applied to linear regression is lasso regression. L2 regularization adds the sum of squared weights and shrinks weights overall (L2 linear regression is ridge regression). Dropout randomly disables neurons during training, gaining an effect like an ensemble of many models.

How it works / steps

  1. Check for overfitting from the gap between training and validation error
  2. Use L1 to narrow features, L2 to smooth the whole
  3. Use dropout in neural networks

Workplace example

In a sales forecast with 500 features, lasso regression sets many feature coefficients to 0, narrowing to the 30 that matter.

Common confusions and exam tips

Mini quiz

Q1. You are choosing a regularization method to curb overfitting. During neural network training, you want to randomly disable a fixed share of neurons to get an effect close to an ensemble. Which fits best?

  1. Dropout
  2. L1 regularization
  3. L2 regularization
  4. Early stopping
Show answer and explanation

Answer: A Dropout

With dropout, a different network is learned each time, curbing overfitting. All neurons are used at inference. (Deciding point: randomly disabling neurons.)

Q2. You are choosing a regularization method to curb overfitting. As training continues, validation error starts to rise, so you want to cut training off at that point. Which fits best?

  1. L1 regularization
  2. Early stopping
  3. L2 regularization
  4. Dropout
Show answer and explanation

Answer: B Early stopping

Early stopping halts training when validation error stops improving. (Deciding point: stopping when validation error worsens.)

Reference links

3.5Backpropagation

Backpropagation (chain rule, vanishing and exploding gradients, credit assignment problem)

Concept

Backpropagation passes output error backward from the output layer to the input layer, efficiently computing each weight's gradient with the chain rule of differentiation. With deep layers, gradients can become too small (vanishing gradients) or too large (exploding gradients). The credit assignment problem is deciding how responsible each weight is for the error, and backpropagation is one solution.

How it works / steps

  1. Understand computing derivatives of composite functions with the chain rule
  2. Address vanishing gradients with ReLU, normalization and skip connections
  3. Address exploding gradients with gradient clipping

Workplace example

Exploding gradients occurred while training a deep RNN, so use gradient clipping to cap gradient size.

Common confusions and exam tips

Mini quiz

Q1. Which best explains why vanishing gradients easily occur when sigmoid is used in hidden layers?

  1. Because outputs are never negative, all weights converge to the same value
  2. Because outputs always sum to 1, multiple classes cannot be learned at once
  3. Because its derivative peaks at a small 0.25, gradients approach 0 as layers stack
  4. Because it cannot be differentiated at input 0, stopping training computation
Show answer and explanation

Answer: C Because its derivative peaks at a small 0.25, gradients approach 0 as layers stack

Backpropagation multiplies each layer's derivatives, so small values stacking up make gradients vanish on the input side. tanh's derivative peaks at 1, improving on sigmoid.

Q2. Which best explains why the Leaky ReLU function was proposed as an improvement on ReLU?

  1. To cap the slope for positive inputs at 0.25 or below so gradients don't explode
  2. To keep output constant regardless of input size and stabilize training
  3. To keep output in the 0–1 range so each neuron's output can be treated as a probability
  4. To give negative inputs a small slope, preventing neurons that stop learning
Show answer and explanation

Answer: D To give negative inputs a small slope, preventing neurons that stop learning

With ReLU, gradients are 0 for negative inputs, so neurons that stop learning (dying ReLU) can occur. Leaky ReLU has a slight slope on the negative side.

Reference links

3.6Optimization methods

Optimization methods (gradient descent, SGD, momentum, AdaGrad, RMSprop, Adam, learning rate, early stopping, double descent, hyperparameter search)

Concept

Gradient descent updates weights little by little opposite to the error gradient. There is batch (all data), stochastic gradient descent (SGD, one at a time) and mini-batch (small amounts). Momentum adds inertia to damp oscillation; AdaGrad lowers the learning rate of weights with large past updates; RMSprop fixes that decline going too far; Adam combines momentum and RMSprop. The learning rate is the update step size. Early stopping halts training when validation error starts to rise. Double descent is the phenomenon where error first rises and then falls again as models grow. Hyperparameters are searched with grid search, random search and Bayesian optimization.

How it works / steps

  1. First try common methods such as Adam with a standard learning rate
  2. Monitor validation error and set early stopping
  3. Search hyperparameters efficiently with random search or Bayesian optimization

Workplace example

Training error oscillates and won't fall, so lower the learning rate and switch the optimizer from SGD with momentum to Adam.

Common confusions and exam tips

Mini quiz

Q1. Which method efficiently computes each weight's gradient with respect to the output error in neural network training?

  1. Forward propagation
  2. Gradient clipping
  3. Stochastic gradient descent
  4. Backpropagation
Show answer and explanation

Answer: D Backpropagation

Backpropagation uses differentiation of composite functions (the chain rule). It also solves the problem of deciding which weights are responsible for the error (the credit assignment problem). The forward computation is forward propagation. [Other options] Forward propagation computes outputs from inputs. SGD updates weights using the gradients. Gradient clipping shrinks overly large gradients.

Q2. You are choosing a gradient descent optimizer. You want to combine both inertia (moving average of gradients) and per-parameter learning rate adjustment (moving average of squared gradients). Which fits best?

  1. Stochastic gradient descent (SGD)
  2. Adam
  3. AdaGrad
  4. Momentum
Show answer and explanation

Answer: B Adam

Adam combines momentum and RMSprop and is widely used. AdaBound and AMSBound improve on Adam. (Deciding point: combining momentum and RMSprop.)

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)