Google Colab

google colab
Quick Primer on Colab Jupyter

Start

Firstly, you need to have a google drive. (Of course you also have to get a google account before this.)

New a folder as your work space in your google drive.

  • If you already have a .ipynb file, you can double click it directly (or right click it > Open with > Google Colaboratory) to open this .ipynb file with Colab.

  • If you want to new a .ipynb file, just right click blank space > More > Google Colaboratory, then google drive will create a Untitled.ipynb file and open it with Colab automatically.

The use of Colab is vary similar with jupyter notebook.

Some skills

You can add ! before a command to indicate it should be run in shell. For example

1
! pip list

By runing this command, Colab will return the names of all the packages it has installed including most common packages.

1
2
3
4
5
6
7
8
...
matplotlib                       3.7.1
numpy                            1.22.4
pandas                           1.5.3
Pillow                           9.4.0
tensorflow                       2.12.0
torch                            2.0.1+cu118
...

If you want to install another package, take the lightning for an example, use the command below.

1
! pip install lightning

Once you finish the install, you can import it in your python code by add this command.

1
import lightning

Note: Using short cut key Ctrl + Enter to run the code block quickly.

Mount to google drive

1
2
3
from google.colab import drive
drive.mount('/content/drive')
# Mounted at /content/drive

Change working directory

Using os.getcwd() or !pwd to get current working directory

1
2
3
import os
os.getcwd()
# /content
1
2
!pwd
# /content

Using os.chdir('...') or magic cd %cd '...' to change working directory to your folder of Google Driver.

1
os.chdir('/content/drive/MyDrive/Colab Notebooks')
1
%cd '/content/drive/MyDrive/Colab Notebooks'

Note: Note that !cd '...' doesn’t work for this purpose because it will take effect only in the current code block.

magic-cd | Ipython
Linux pwd命令 | Runoob
Linux cd 命令 | Runoob

How to Change Python Version in Google Colab

1
! cat /etc/os-release # check the system version of Colab
1
! python --version # check the default version of Python in Colab
1
! ls /usr/bin/python* # check the availabel versions of python in Colab(Ubuntu)

How to Change Python Version in Google Colab

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
!cat /etc/os-release
# PRETTY_NAME="Ubuntu 22.04.2 LTS"
# ...

# Check the default python version of colab
!python --version
# Python 3.10.12

# Check the python version you have
!ls /usr/bin/python*
# /usr/bin/python3  /usr/bin/python3.10-config  /usr/bin/python3-config
# /usr/bin/python3.10

# Install the python version you wish to list (python3.8 in my case)
!sudo apt-get update -y
!sudo apt-get install python3.8
# ...

# Check the availabel versions of python again
! ls /usr/bin/python*
# /usr/bin/python3     /usr/bin/python3.10-config  /usr/bin/python3-config
# /usr/bin/python3.10  /usr/bin/python3.8

# Switch source for python3 command
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
# update-alternatives: using /usr/bin/python3.8 to provide /usr/bin/python3 (python3) in auto mode

# Choose one of the given alternatives:
!sudo update-alternatives --config python3
# There is only one alternative in link group python3 (providing /usr/bin/python3): /usr/bin/python3.8
# Nothing to configure.

# Check the result
!python3 --version
# Python 3.8.18

# Check the pip version
!pip --version
# ...
# ModuleNotFoundError: No module named 'pip'

# Attention: Install pip (... needed!)
!sudo apt install python3-pip
# ...

# check pip version
pip --version
# ...
# ModuleNotFoundError: No module named 'distutils.cmd'

# install python3.8-distutils
!sudo apt install python3.8-distutils

# check pip version
pip --version
# ...
# pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

Alternative reading
Update-alternatives Command: A Comprehensive Guide for Linux Users

1
!sudo update-alternatives: --install <link> <name> <path> <priority>

unzip

Upload the .zip file to your google driver and open a .ipynb file in colab, then mount your google driver to colab and execute the following command.
The first path is where your package is ('.../Inception-v4.zip' in my case), the second path is the destination folder.
Note that the destination folder must be under '/content/drive/MyDrive/' which is your owe google driver directory.

1
!unzip '/content/drive/MyDrive/Colab Notebooks/Inception-v4.zip' -d '/content/drive/MyDrive/Colab Notebooks'

How to find the location of package that you pip installed

1
!python -m pip show <package_name>

Take timm for example

1
2
3
4
5
6
7
8
9
10
11
12
!python -m pip show timm

Name: timm
Version: 0.9.12
Summary: PyTorch Image Models
Home-page: https://github.com/huggingface/pytorch-image-models
Author: Ross Wightman
Author-email: ross@huggingface.co
License: 
Location: /usr/local/lib/python3.10/dist-packages
Requires: huggingface-hub, pyyaml, safetensors, torch, torchvision
Required-by:

Reload your modified python file

Jupyter Notebook Reload Module: A Comprehensive Guide

If you have imported a python file and later make changes to it, you’ll need to reload it in your Jupyter Notebook (or Colab) to take advantage of any recent changes.

Here’s the scenario. You are working in a Jupyter Notebook and you’ve imported a custom python file. While working in your notebook, you make some changes to the python file and want to work with those new changes in your Jupyter Notebook. After saving your python file, you run your import some_file code again. However, your recent changes don’t get imported.

You could restart your entire kernel, or you can simply reload the file by running this code:

1
2
3
4
import python_file
...
import importlib
importlib.reload(python_file)

Once you run the code, you’ll see that any changes you made to your python file are correctly loaded into your jupyter notebook.

Using the %load_ext and %autoreload Magic Commands

1
2
%load_ext autoreload
%autoreload 2

TypeError

1
TypeError: reload() argument must be a module

The reason why you got this error is that this module hadn’t been imported, or you used from python_file import * rather than import python_file so you didn’t import the python file (the module) actually.

There is two way to solve this problem.

The first is using import python_file to take place of from python_file import *;

The second is using sys.modules['python_file'] to reload your module. Then reimport your python_file.

1
2
3
4
5
6
from python_file import *
...
import sys
import importlib
importlib.reload(sys.modules['python_file'])
from python_file import *
1
2
3
4
5
6
from python_file import xxx
...
import sys
import importlib
importlib.reload(sys.modules['python_file'])
from python_file import xxx

e.g.

1
2
3
4
5
6
7
8
from geoseg.datasets.potsdam_dataset import PotsdamDataset

# modify PotsdamDataset

import sys
import importlib
importlib.reload(sys.modules['geoseg.datasets.potsdam_dataset'])
from geoseg.datasets.potsdam_dataset import PotsdamDataset

Cuda in Colab

Click top left corner Edit > Notebook settings >

1
2
Hardware accelerator: None/GPU/TPU
GPU type: T4/A100/V100

Use this command to test if GPU is available. You should get True if you set correctly.

1
2
import torch
torch.cuda.is_available() # True

Get the number of GPU

1
torch.cuda.device_count() # 1

Get the index of current device

1
torch.cuda.current_device() # 0

Get the name of device

1
torch.cuda.get_device_name(0) # Tesla T4

Using this command to open nvidia-smi (Invidia System Management Interface) to check more information about your GPU.

1
!/opt/bin/nvidia-smi

By default, tensors are generated on the CPU. You have to manually make sure that the operation is done through the GPU.
Using .cuda() you can convert (copy) Tensor from CPU to GPU.
If there are multiple Gpus, use .cuda(i) to represent the i-th GPU, and i defaults to 0.
Besides, you can use .is_cuda or .is_cpu to check whether a tensor is on GPU or CPU.

1
2
3
4
5
6
x = torch.tensor([1, 2, 3])
x.is_cuda # False
x.is_cpu  # True
x = x.cuda(0)
x.is_cuda # True
x.is_cpu  # False

Using .cpu() you can convert(copy) a Tensor back to CPU from GPU. And you can print a Tensor’s device attribute to check it is on a CPU or a GPU.

1
2
3
4
5
6
x = torch.tensor([1, 2, 3])
print(x.device) # cpu
x = x.cuda(0)
print(x.device) # cuda:0
x = x.cpu()
print(x.device) # cpu

You can specify a tensor’s device attribute when you create it.

1
2
x = torch.tensor([1, 2, 3], device="cuda:0")
x.is_cuda # True

device and .to(device) is commonly used in training with GPU. And This method is more recommended.

1
2
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device, type(device)) # cuda:0 <class 'torch.device'>
1
2
3
4
x = torch.tensor([1, 2, 3], device=device)
or
x = torch.tensor([1, 2, 3]).to(device)
x.is_cuda # True

If a calculation of a Tensor on a GPU device is performed, the result of the calculation will also be stored in that device:

1
2
3
4
x = torch.tensor([1, 2, 3]).cuda()
print(x.device) # cuda:0
y = x**2
print(y.device) # cuda:0

However, Tensor on different devices cannot be calculated directly, including Tensor on the GPU cannot be calculated directly with Tensor on the CPU, and Tensor on different GPU devices cannot also be calculated directly:

1
2
3
4
5
6
7
x = torch.tensor([1, 2, 3]).cpu()
y = torch.tensor([1, 2, 3]).cuda()
print(x.device) # cpu
print(y.device) # cuda:0
z = x + y

# RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

By default, your model is generated on CPU

1
2
3
4
5
model = torch.nn.Linear(3, 1)
for param in model.parameters():
  print(param.device)  # cpu
  print(param.is_cpu)  # True
  print(param.is_cuda) # False
1
2
3
4
5
6
7
for param in model.parameters():
  print(param, param.size())

# Parameter containing:
# tensor([[ 0.1516, -0.0096,  0.1179]], requires_grad=True) torch.Size([1, 3])
# Parameter containing:
# tensor([0.2088], requires_grad=True) torch.Size([1])

You can convert your model to a GPU via .cuda() or .to(device) command. What .cuda() or .to(device) do is moving all model parameters and buffers to the GPU.
torch.nn.Module.cuda

1
2
3
4
5
model = torch.nn.Linear(3, 1).cuda(0)
for param in model.parameters():
  print(param.device)  # cduda:0
  print(param.is_cpu)  # False
  print(param.is_cuda) # True

How to run your .py file rather than .ipynb on Colab

First change directory to where your .py file on your google driver

1
2
3
4
import os
path="/content/drive/My Drive/..."
os.chdir(path)
os.listdir(path)

Then use the shell command to run your .py file.

1
!python filename.py

empty_cache

torch.cuda.empty_cache()

pandas

pandas.DataFrame

1
2
3
import pandas as pd

pd.DataFrame()

sklearn

sklearn.model_selection.train_test_split

1
2
3
from sklearn.model_selection import train_test_split

train_test_split()

segmentation_models_pytorch

segmentation_models.pytorch | GitHub
segmentation_models_pytorch’s documentation

1
2
3
4
5
6
7
8
9
10
11
import segmentation_models_pytorch as smp

model = smp.Unet(
  encoder_name = 'inceptionv4', # choose 'inceptionv4' as encoder
  encoder_weights = 'imagenet', # use `imagenet` pre-trained weights for encoder initialization
  in_channels=3, # model input channels (1 for gray-scale images, 3 for RGB, default is 3)
  classes=23,    # model output channels (number of classes in your dataset)
  activation = None,
  encoder_depth=5,
  decoder_channels=[256, 128, 64, 32, 16]
)

smp.Unet() HTTP Error 502

When you are invoking smp.Unet() or other model of smp, you get a http error as follows:

1
2
3
4
5
6
7
8
9
10
Downloading:
"http://data.lip6.fr/cadene/pretrainedmodels/inceptionv4-8e4777a0.pth" 
to
C:\Users\xiaophai/.cache\torch\hub\checkpoints\inceptionv4-8e4777a0.pth

Traceback (most recent call last):
  ...
  File "...\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 502: Bad Gateway

The reason for this is that the certificate of the website has expired, as shown in the following picture (the date is 2023-8-6).

http://data.lip6.fr/cadene/pretrainedmodels/inceptionv4-8e4777a0.pth

You need to download the .pth file manually (for example inceptionv4-8e4777a0.pth in my case) from the website, and move or copy it to where it should be (check the terminal to get the website and the path it should go).

In the case of Colab

In the case of Colab, you first need to upload the .pth file to your Google Driver. Then use the following commands to copy the .pth file to the destination folder.

If you have not used torch hub before, you need to create a folder under this path (you can get it from the Error information) for torch hub first.

1
!mkdir -p '/root/.cache/torch/hub/checkpoints/'

Otherwise, you may get this error if you excute copy command directly.

1
cp: cannot create regular file '/root/.cache/torch/hub/checkpoints/': No such file or directory

Now you can use the following command to copy the .pth file to the destination folder. The first path is the path of your source file (…/inceptionv4-834777a0.pth in my case), and the second path is the path of your destination folder.

1
2
3
!cp -r
'/content/drive/MyDrive/Colab Notebooks/Inception-v4/inceptionv4-8e4777a0.pth'
'/root/.cache/torch/hub/checkpoints/'

Linux mkdir 命令 | Runoob
Linux cp 命令 | Runoob

albumentations

Albumentations documentation

1
2
3
4
5
6
7
8
9
10
11
12
import albumentations as A

t_train = A.Compose([
  A.Resize(704, 1056, interpolation=cv2.INTER_NEAREST),
  A.HorizontalFlip(),
  A.GridDistortion(p=0.2)
])
t_val = A.Compose([
  A.Resize(704, 1056, interpolation=cv2.INTER_NEAREST),
  A.HorizontalFlip(),
  A.GridDistortion(p=0.2)
])

torch.hub

Pytorch Hub
pytorch_vision_resnet

Module.train&eval

torch.nn.Module.train

torch.nn.Module.eval

torch.save&load

torch.save
torch.load
Saving and loading pytorch tensors and module states

saving and loading tensors

1
2
3
4
5
6
7
8
9
import torch

t1 = torch.tensor([1, 2])
# saving t1 as tensor.pt
torch.save(t1, 'tensor.pt')
# loading tensor.pt to t2
t2 = torch.load('tensor.pt')
print(t2)
# tensor([1, 2])

By convention, PyTorch files are typically written with a ‘.pt’ or ‘.pth’ extension.

  • you can also save multiple tensors as part of Python objects like tuples, lists, and dicts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import torch

t1 = torch.tensor([1, 2])
t2 = torch.tensor([3, 4])

# lists
list = [t1, t2]
torch.save(list, 'tensor_list.pth')

list_load = torch.load('tensor_list.pth')
print(list_load)
# [tensor([1, 2]), tensor([3, 4])]
print(list_load[0]) # tensor([1, 2])
print(list_load[1]) # tensor([3, 4])

# dicts
dict = {'tensor1':t1, 'tensor2':t2}
torch.save(dict, 'tensor_dict.pth')

dict_load = torch.load('tensor_dict.pth')
print(dict_load)
# {'tensor1': tensor([1, 2]), 'tensor2': tensor([3, 4])}
print(dict_load['tensor1']) # tensor([1, 2])
print(dict_load['tensor2']) # tensor([3, 4])
  • Saving and loading tensors preserves views
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import torch

numbers = torch.arange(0, 10)
evens = numbers[0:10:2]

# numbers and evens share the same “storage”
print(numbers.data_ptr()==evens.data_ptr()) # True

print(numbers)
# tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(evens)
# tensor([0, 2, 4, 6, 8])
numbers *= 2
print(numbers)
# tensor([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18])
print(evens)
# note that evens also changed
# tensor([ 0,  4,  8, 12, 16])

torch.save([numbers, evens], 'tensors.pt')
loaded_numbers, loaded_evens = torch.load('tensors.pt')

# Saving and loading tensors preserves views
# numvers and evens still share the same “storage”
print(loaded_numbers.data_ptr()==loaded_evens.data_ptr()) # True

print(loaded_numbers)
# tensor([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18])
print(loaded_evens)
# tensor([ 0,  4,  8, 12, 16])
loaded_evens *= 2
print(loaded_numbers)
# note that loaded_numbers also changed 
# tensor([ 0,  2,  8,  6, 16, 10, 24, 14, 32, 18])
print(loaded_evens)
# tensor([ 0,  8, 16, 24, 32])
  • use .clone() to avoid unnecessary waste of storage
1
2
3
4
5
6
7
8
9
import torch

large = torch.arange(0, 1000)
small = large[0:5]
print(large.storage().size()) # 1000
print(small.storage().size()) # 1000

small = large[0:5].clone()
print(small.storage().size()) # 5
1
2
3
4
5
6
7
import torch

large = torch.arange(0, 1000)
small = large[0:5]
torch.save(small.clone(), 'small.pt')
loaded_small = torch.load('small.pt')
print(loaded_small.storage().size()) # 5

Saving and loading Modules

Tutorial: Saving and loading modules

  • saving and loading state_dict()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(3, 2)
        self.layer2 = nn.Linear(2, 1)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        return x
    
model = MyModule()
print(model.state_dict())
# OrderedDict([
# ('layer1.weight', tensor([[ 0.3531, -0.5274, -0.2317],
#                           [ 0.5642,  0.1380, -0.1637]])),
# ('layer1.bias', tensor([-0.4267,  0.5400])),
# ('layer2.weight', tensor([[-0.2539,  0.5889]])),
# ('layer2.bias', tensor([0.6985]))
# ])
torch.save(model.state_dict(), 'mymodel_state_dict.pt')
loaded_state_dict = torch.load('mymodel_state_dict.pt')

new_model = MyModule()
new_model.load_state_dict(loaded_state_dict)
  • saving and loading entire model
1
2
3
4
5
# save:
torch.save(model, 'mymodel.pt')
# load:
model = torch.load('mymodel.pt')
model.eval()

Load a model saved from a CUDA device to a CPU device

If you load a model saved from a CUDA device to a CPU device directly, you will get a RuntimeError as following:

1
2
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False.
If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

You have to check torch.load() again to see its map_location attribute and add it to your torch.load

1
torch.load('xxx.pt', map_location=torch.device('cpu'))

Train, Validation and Test Sets

About Train, Validation and Test Sets in Machine Learning

Swin-Transformer for Classification

Data Set: flower-photos

1
2
3
4
5
6
7
8
# Folder Structure
flower_photos  : 3670
├── daisy      : 633
├── dandelion  : 898
├── roses      : 641
├── sunflowers : 699
├── tulips     : 799
└── LICENSE.txt
1
2
3
4
5
6
7
8
9
# read_split_data(root: str, val_rate: float = 0.2)
classes    : num | calculation   | validation | train
daisy      : 633 | 633*0.2=126.6 | 126        | 507
dandelion  : 898 | 898*0.2=179.6 | 179        | 719
roses      : 641 | 641*0.2=128.2 | 128        | 513
sunflowers : 699 | 699*0.2=139.8 | 139        | 560
tulips     : 799 | 799*0.2=159.8 | 159        | 640
# ---------------------------------------------------
total     : 3670 |               | 731        | 2939

Structure
Input Image: 3x224x224
Number of Classes: 5
Patch Partition: 56x56x48, 2244×2244×48\frac{224}{4}\times\frac{224}{4}\times 48
Stage 1

  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:

Stage 2

  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:

Stage 3

  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:

Stage 4

  • Linear Embedding:
  • Swin Transformer Block:
  • Linear Embedding:
  • Swin Transformer Block:

Stage 1: Linear Embedding + SwinTransformerBlock*2
Stage 2: PatchMerging + SwinTransformerBlock*2
Stage 3: PatchMerging + SwinTransformerBlock*6
Stage 4: PatchMerging + SwinTransformerBlock*2