PyTorch, No Tears

Preface

This book is to teach students how program in PyTorch. To follow along and execute the code samples, you will need Docker installed. The Docker container is located on Docker Hub. After you have installed Docker, you may run the container as follows.

docker run -it \
 -p 8888:8888 \
 -p 6006:6006 \
 --gpus all \
 oneoffcoder/book-pytorch-intro-gpu

Note that this Docker container has Jupyter Lab running on port 8888. You may access Jupyter Lab at http://localhost:8888 when the Docker container is running.

The diagram below shows the PyTorch learning path from tensors and automatic differentiation into data pipelines, optimization, model building, and more specialized deep learning applications.

That ordering mirrors real PyTorch work: understand tensors first, then build the training loop, and only then take on larger end-to-end architectures.

Foundations

• 1. Environment
  • 1.1. nvidia-smi
  • 1.2. nvcc
  • 1.3. PyTorch
• 2. Tensor
  • 2.1. Creation
  • 2.2. Numpy bridge
  • 2.3. Indexing, slicing, joining and mutating
  • 2.4. Random sampling
  • 2.5. Operations
  • 2.6. Basic math
  • 2.7. Broadcasting
  • 2.8. Device-agnostic tensors
  • 2.9. Serialization
• 3. Autograd

Data

• 1. Data
  • 1.1. Dataset
  • 1.2. ImageFolder
  • 1.3. DataLoader
  • 1.4. Datasets
• 2. Datasets and Caching
  • 2.1. Local datasets
  • 2.2. Balanced sampling
  • 2.3. Predownload remote datasets
• 3. Data Pipelines at Scale
  • 3.1. Shard an iterable dataset by worker
  • 3.2. Worker seeding
• 4. Transformation
  • 4.1. PIL transforms
  • 4.2. Tensor transforms
  • 4.3. Conversion transforms

Training

• 1. Loss Functions
  • 1.1. L1
  • 1.2. Smooth L1
  • 1.3. Mean-Squared Error
  • 1.4. Cross-Entropy
  • 1.5. Negative Log-Likelihood
  • 1.6. Poisson Negative Log-Likelihood
  • 1.7. Kullback-Leibler Divergence
  • 1.8. Binary Cross-Entropy
  • 1.9. Binary Cross-Entropy with Logits
  • 1.10. Margin Ranking
  • 1.11. Soft Margin
• 2. Optimizer
  • 2.1. Stochastic gradient descent
  • 2.2. AdaDelta
  • 2.3. AdaGrad
  • 2.4. Adam
  • 2.5. AdamW
  • 2.6. Adamax
  • 2.7. ASGD
  • 2.8. RMSprop
  • 2.9. Rprop
• 3. Scheduler
  • 3.1. LambdaLR
  • 3.2. StepLR
  • 3.3. MultiStepLR
  • 3.4. ExponentialLR
  • 3.5. CyclicLR
  • 3.6. CosineAnnealingWarmRestarts
• 4. Learning
  • 4.1. Training and testing
  • 4.2. Validation
• 5. Training Recipe
  • 5.1. Setup
  • 5.2. Model
  • 5.3. One epoch functions
  • 5.4. Fit loop
• 6. Early Stopping
  • 6.1. Setup
  • 6.2. Data
  • 6.3. Model
  • 6.4. Training With Patience
  • 6.5. Test Performance
• 7. Class Imbalance
  • 7.1. Setup
  • 7.2. Create an Imbalanced Dataset
  • 7.3. Helper Functions
  • 7.4. Naive Training
  • 7.5. Weighted Loss
• 8. Debugging Training
  • 8.1. Check one batch
  • 8.2. Overfit one batch
  • 8.3. Measure gradients
• 9. Ragged Sequences
  • 9.1. Packed RNN input
  • 9.2. Attention mask
• 10. Multi-Task Learning

Performance and Scaling

• 1. Training Performance
  • 1.1. Non-blocking transfer
  • 1.2. Mixed precision and gradient clipping
  • 1.3. Gradient accumulation
  • 1.4. Evaluation mode
• 2. Memory Optimization
  • 2.1. Parameter footprint
  • 2.2. Gradient accumulation
  • 2.3. Activation checkpointing
• 3. Profiling
  • 3.1. Profile one train step
  • 3.2. Export a trace
• 4. Compiled Models
  • 4.1. Warmup and compare outputs
  • 4.2. Micro-benchmark
• 5. Distributed Training
  • 5.1. Initialize a local process group
  • 5.2. Wrap the model and use a distributed sampler
  • 5.3. Rank-zero checkpointing

Models, Persistence, and Deployment

• 1. Models and Convolutional Neural Networks
  • 1.1. ResNet18
  • 1.2. ResNet152
  • 1.3. AlexNet
  • 1.4. VGG19_BN
  • 1.5. SqueezeNet1_1
  • 1.6. DenseNet201
  • 1.7. GoogleNet
  • 1.8. ShuffleNet_v2_x1_0
  • 1.9. ResNext101_32x8d
  • 1.10. Wide_ResNet101_2
  • 1.11. MNASNet1_0
• 2. Saving and Loading Models
  • 2.1. Saving
  • 2.2. Loading
• 3. Checkpoint and Resume
  • 3.1. Train, save, reload, resume
• 4. Model Zoo
  • 4.1. Discover torchvision models
  • 4.2. Replace the classifier head
  • 4.3. Freeze the backbone
  • 4.4. Weight metadata
• 5. Parameter-Efficient Finetuning
  • 5.1. Frozen backbone with a trainable adapter
  • 5.2. Train only the adapter and head
• 6. Fine-Tuning Patterns
  • 6.1. Setup
  • 6.2. Data and a Pretend Pretrained Backbone
  • 6.3. Helpers
  • 6.4. Linear Probe
  • 6.5. Partial Unfreeze
  • 6.6. Full Fine-Tune
  • 6.7. Comparison
• 7. Custom Autograd
  • 7.1. A custom square function
  • 7.2. Gradient check
• 8. Quantization
  • 8.1. Dynamic quantization
  • 8.2. Parameter footprint
• 9. Export and Deploy
  • 9.1. torch.export
  • 9.2. ONNX export
• 10. Serving API
  • 10.1. Single request and batch request
  • 10.2. API shape
• 11. Inference Batching
  • 11.1. Setup
  • 11.2. Helper
  • 11.3. Batch Size Comparison
  • 11.4. no_grad() vs Tracking Gradients

Evaluation and Reliability

• 1. Calibration and Metrics
  • 1.1. Expected calibration error
• 2. Experiment Tracking
  • 2.1. Setup
  • 2.2. Data
  • 2.3. Tracking Helpers
  • 2.4. Compare Two Runs
  • 2.5. Read Back the Saved Metadata
• 3. Uncertainty
  • 3.1. Predictive entropy
  • 3.2. Monte Carlo dropout
• 4. Reproducibility
  • 4.1. Reset seeds and regenerate
  • 4.2. Save metadata with checkpoints
• 5. Testing PyTorch Code
  • 5.1. Tiny forward and backward checks
  • 5.2. Checkpoint reload test
  • 5.3. What to test in real projects

Text and Language

• 1. Text Data
  • 1.1. Tokenize and build a vocabulary
  • 1.2. Dataset and collate
  • 1.3. Move a text batch to device
• 2. Embeddings
  • 2.1. Mean-pooled text classifier
  • 2.2. EmbeddingBag
• 3. Sequence Classification
  • 3.1. GRU classifier
• 4. Transformer Language Model
  • 4.1. Causal transformer
  • 4.2. Generate characters
• 5. Recurrent Neural Networks
  • 5.1. Data
  • 5.2. Model
  • 5.3. Training
  • 5.4. Evaluation

Graphs

• 1. Graph Data
  • 1.1. A graph as tensors
  • 1.2. Edge list to adjacency
  • 1.3. Batch multiple graphs
• 2. Message Passing
  • 2.1. Sum and mean aggregation
  • 2.2. A message-passing layer
  • 2.3. Add self loops
• 3. Graph Convolution
  • 3.1. Synthetic node classification graph
  • 3.2. GCN layer
• 4. Graph Classification
  • 4.1. A tiny graph dataset
  • 4.2. Graph classifier
• 5. Heterogeneous Graphs
  • 5.1. Separate tensors by node type and edge type
  • 5.2. Relation-specific projections
• 6. PyTorch Geometric
  • 6.1. Dependency boundary
  • 6.2. The common data shape
  • 6.3. Layer boundary
  • 6.4. When to install it

Forecasting and Recommendation

• 1. Time Series
  • 1.1. Build windows from a signal
  • 1.2. Train a simple forecaster
• 2. Recommendation Systems
  • 2.1. User-item interactions
  • 2.2. Matrix factorization with embeddings

Vision and Generative Models

• 1. Generative Adversarial Network
  • 1.1. Parameters
  • 1.2. Data
  • 1.3. Generator network
  • 1.4. Discriminator
  • 1.5. Loss function and optimizer
  • 1.6. Training
  • 1.7. Visualization
  • 1.8. Saving
• 2. Neural Transfer
  • 2.1. Input images
  • 2.2. Loss functions
  • 2.3. Model
  • 2.4. Normalization
  • 2.5. Loss
  • 2.6. Optimizer
  • 2.7. Transfer
  • 2.8. Results
  • 2.9. Visualize
• 3. Object Detection
  • 3.1. Data
  • 3.2. Model
  • 3.3. Optimizer and scheduler
  • 3.4. Helper code
  • 3.5. Training and testing
  • 3.6. Predictions
• 4. TensorBoard
  • 4.1. Add image
  • 4.2. Add model
  • 4.3. Add embedding
  • 4.4. Add training loss
  • 4.5. Add precision-recall curve

About

One-Off Coder is an educational, service and product company. Please visit us online to discover how we may help you achieve life-long success in your personal coding career or with your company’s business goals and objectives.

• Website

• Facebook

• Twitter

• Instagram

• YouTube

• LinkedIn

Copyright

This work is licensed under a Creative Commons Attribution 4.0 International License by One-Off Coder.



The full source code is available on GitHub.

Cite this book as follows.:

@misc{oneoffcoder_pytorch_intro_2019,
title={PyTorch, No Tears},
url={https://learn-pytorch.oneoffcoder.com},
author={One-Off Coder},
year={2019},
month={Oct}}

Author

Jee Vang, Ph.D.

• Patreon