Understanding pretrain and finetune of transfer learning

This article mainly introduces two practical concepts in transfer learning: pretraining and finetuning.

Supposing that you have the experience of developping a CNN using pytorch.

For example in function forward, you write

h = self.conv1(x)
r1 = self.pool1(h)

h = self.conv2(r1)
r2 = self.pool2(h)

Pretraining means that the 1st layer like conv1 and pool1 is done for you by the pretrained model. And the parameters of them are fixed. What you need to do is just finetuning, i.e. you input the model with x as input but the r1 is fixed so your truly take r1 as input and train the later layers’ parameters.

Usually, the pretrained model is in large scale. To get the inside layers of this network, we use some functions like mx.symbol.FullyConnected to get the fine tuned data.