PMLE · LESSON 4
Section 4: Serving and scaling models約20% · 2 topics
Learn batch and online inference, and model serving and scaling.
4.1Batch and online inference
Batch and online inference (Agent Platform, Model Garden, Cloud Run, GKE), prebuilt and custom containers, Model Registry, A/B testing and canaries, inference pre- and post-processing
Concept
Batch inference predicts large amounts of data together and outputs results to BigQuery or Cloud Storage (when immediacy isn't needed). Online inference predicts per request with low latency at an endpoint. Serving platforms include Agent Platform endpoints, deployment from Model Garden, Cloud Run and GKE. Use prebuilt or custom containers and manage model versions in Model Registry. Try new models on part of the traffic with A/B tests or canaries.
How it works / steps
- Decide batch or online from immediacy requirements
- Register models and versions in Model Registry
- Split traffic at the endpoint and try new versions with a canary
- Keep inference pre- and post-processing consistent with containers or pipelines
Workplace example
Nightly churn scores for all customers are output to BigQuery with batch inference, while web recommendations are returned within 100 ms from an online endpoint.
Common confusions and exam tips
- Batch inference doesn't need an always-on endpoint
- A canary sends a little traffic to the new version to validate it
- If preprocessing differs from training, predictions drift (training-serving skew)
Mini quiz
Q1. What is the most appropriate advantage of using Model Registry?
- It automatically detects and masks PII in training data
- It saves notebook co-editing comments per version
- It automatically optimizes and rewrites the training loss function
- It manages model versions and aliases, making clear which version is in production
Show answer and explanation
Answer: D It manages model versions and aliases, making clear which version is in production
An alias such as “default” points to the production version.
Q2. What is the most appropriate use of traffic splitting?
- Splitting one request by character count and processing parts separately
- Dividing network bandwidth evenly among employees
- Routing requests by percentage to multiple models on the same endpoint
- Automatically splitting training data into training and validation sets
Show answer and explanation
Answer: C Routing requests by percentage to multiple models on the same endpoint
Enables canaries and A/B tests.
Reference links
4.2Scaling online serving
Scaling online serving (Feature Store, public and private endpoints, CPU, GPU, TPU and edge, scaling with throughput, tuning for production)
Concept
Online serving fetches the latest features from Feature Store with low latency and serves through a public or private endpoint (called from inside a VPC). Choose hardware from CPU, GPU, TPU, or edge running on devices. Match throughput with minimum and maximum replica counts and autoscaling, and tune for production with model optimization (such as quantization) and machine type adjustment.
How it works / steps
- Set latency and throughput targets
- Fetch online features from Feature Store
- Use a private endpoint if it's only called internally
- Prepare for peaks with minimum replicas and autoscaling
Workplace example
To prepare for 10x traffic during a sale, autoscale with a minimum of 2 and a maximum of 20 replicas, and secure latency with GPU machines.
Common confusions and exam tips
- With a minimum of 0 replicas, the first request can be slow
- Private endpoints don't go through the internet
- Running at the edge requires making the model lightweight
Mini quiz
Q1. What is the most appropriate advantage of using Feature Store for online serving?
- It automatically retrains model weights at the millisecond level
- It automatically forwards inference requests to another region
- It always makes endpoint fees free
- It retrieves the latest features with low latency
Show answer and explanation
Answer: D It retrieves the latest features with low latency
Fetch the latest values from the online store.
Q2. What is the most appropriate way to raise utilization in GPU serving?
- Always assign one request per GPU and make the rest wait
- Free and reallocate GPU memory on every request
- Process requests together (batching) to use GPU parallelism
- GPUs can't be used for inference, so use them only for training
Show answer and explanation
Answer: C Process requests together (batching) to use GPU parallelism
Dynamic batching raises throughput.
Reference links
Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.
Notes (unofficial, original questions, no pass guarantee)
- The questions on this site are original practice questions written by this site based on publicly available skills outlines and exam guides. They are not actual exam questions.
- Google Cloud, Gemini, Vertex AI and Google Workspace are trademarks of Google LLC. This site is not affiliated with or endorsed by Google.
- This content is a study aid and does not guarantee a pass. Exam content, format and prerequisites may change without notice; always check the latest information on the official page before taking the exam.
- Your accuracy on the mock exam cannot be converted to the official (scaled) score and does not predict a pass or fail.
- Progress is saved only in this browser (localStorage) unless you sign in with Google to sync it. You can delete it from the Cookie & storage settings page (Japanese).