Cross correlation in Machine learning

2 minutes read (About 264 words)

The discrict 2D cross correlation is very similar to 2D convolution.

check by MATLAB code

M1 = [17 24  1  8 15;
      23  5  7 14 16;
       4  6 13 20 22;
      10 12 19 21  3;
      11 18 25  2  9];
M2 = [8 1 6;
      3 5 7;
      4 9 2];
D = xcorr2(M1, M2);
D

But there is some difference between MATLAB and PyTorch. The relevant functions in PyTorch are torch.nn.functional.conv2d and torch.nn.Conv2d. Because torch.nn.functional.conv2d can setup paddings as an argument, so the result matrix can be smaller. Let’s do a simple experiment to check it.

1
2
3
4
5
6
7
8
9
10
11
Python 2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 19:04:19) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torch.nn as nn
>>> import torch.nn.functional as F
>>> inputs = torch.randn(1, 4, 5, 5)
>>> filters = torch.randn(6, 4, 3, 3)
>>> results = F.conv2d(inputs, filters, padding=2)
>>> results.size()
(1, 6, 7, 7)

We will check the value for first filter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
>>> inputs
tensor([[[[-2.0126, -1.2142, -0.5943,  0.9880,  1.3990],
          [ 1.2635,  1.6013, -0.4639,  0.1469,  1.7033],
          [ 1.0538,  0.8419,  2.0627,  1.7289,  0.0493],
          [-0.7360, -1.0318, -2.6838,  0.1069, -1.1440],
          [-0.2589,  2.0517,  0.2611, -1.7295,  0.0654]],

         [[-1.8845, -1.9481,  0.3805, -0.6467,  0.6878],
          [-0.3418, -1.3715,  0.4947,  0.4899,  0.2835],
          [-0.8665,  0.6921,  0.1421,  0.1243, -0.9729],
          [ 0.2235, -0.3882, -0.4885,  1.1147, -0.2176],
          [-0.8032, -0.1928, -0.6535, -0.0107,  0.6291]],

         [[ 1.0086,  1.0685, -0.2147, -0.0510,  0.2829],
          [ 0.1978, -1.4479,  0.8483, -0.7487, -0.5068],
          [ 1.0150,  0.2064, -0.1610,  1.1604, -0.0933],
          [-0.4072,  0.2518,  0.2360,  0.0037,  0.0191],
          [ 1.3504, -0.1713, -1.0858,  1.7440, -1.0150]],

         [[-1.1781,  0.9697, -0.0453, -0.2083, -0.9241],
          [-0.3955,  0.0400, -0.4108, -0.8875, -0.3455],
          [-0.6430,  0.3412,  0.6656,  0.6985,  0.2691],
          [ 1.3433, -1.4387, -1.2028, -0.2997,  0.6933],
          [-0.4480, -0.7450,  0.9386, -1.3733,  1.1912]]]])
>>> filters[0]
tensor([[[-1.1877,  0.2613,  1.7033],
         [ 0.5515, -0.5155, -0.8936],
         [-1.1671, -1.3611, -0.0778]],

        [[-1.0069,  1.0255,  0.9102],
         [-0.4946,  0.8556,  1.4657],
         [ 0.1137, -0.2616,  0.5006]],

        [[ 1.4465,  1.2084,  2.2803],
         [ 0.2097,  0.0633,  0.7705],
         [-0.7784, -0.9365, -0.9489]],

        [[ 0.7263,  1.7227,  1.2436],
         [-0.9810,  0.2232,  2.0970],
         [ 0.4908,  0.0761, -1.2843]]])
>>> results[0, 0]
tensor([[ -0.2308,  -0.9420,   2.5055,   1.6621,   0.8917,  -3.7088,  -2.2284],
        [ -2.6064,  -1.0220,  -1.0562,  -1.2500,  -2.6156,  -1.1755,  -0.3335],
        [ -7.2688, -11.5212,  -0.2480,   0.6509,  -6.4913,  -3.3876,  -1.5478],
        [ -2.1471,   0.4592,   3.3012,   0.6832,   0.4210,  -1.7197,  -1.4319],
        [  4.9215,   0.9659,  -0.4123,   8.2638,   4.4465,   0.5087,   1.1521],
        [ -1.1531,  -5.8456,  -8.2622,   0.1412,   5.1500,   0.7043,   0.4526],
        [  1.3501,   1.9708,   0.1334,  -4.0153,  -1.4115,   5.0781,  -1.3142]])

Do the same thing on Matlab

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
>> inputs_0 = [[-2.0126, -1.2142, -0.5943,  0.9880,  1.3990];
          [ 1.2635,  1.6013, -0.4639,  0.1469,  1.7033];
          [ 1.0538,  0.8419,  2.0627,  1.7289,  0.0493];
          [-0.7360, -1.0318, -2.6838,  0.1069, -1.1440];
          [-0.2589,  2.0517,  0.2611, -1.7295,  0.0654]];
>> inputs_1 = [[-1.8845, -1.9481,  0.3805, -0.6467,  0.6878];
          [-0.3418, -1.3715,  0.4947,  0.4899,  0.2835];
          [-0.8665,  0.6921,  0.1421,  0.1243, -0.9729];
          [ 0.2235, -0.3882, -0.4885,  1.1147, -0.2176];
          [-0.8032, -0.1928, -0.6535, -0.0107,  0.6291]];
>> inputs_2 = [[ 1.0086,  1.0685, -0.2147, -0.0510,  0.2829];
          [ 0.1978, -1.4479,  0.8483, -0.7487, -0.5068];
          [ 1.0150,  0.2064, -0.1610,  1.1604, -0.0933];
          [-0.4072,  0.2518,  0.2360,  0.0037,  0.0191];
          [ 1.3504, -0.1713, -1.0858,  1.7440, -1.0150]];
>> inputs_3 = [[-1.1781,  0.9697, -0.0453, -0.2083, -0.9241];
          [-0.3955,  0.0400, -0.4108, -0.8875, -0.3455];
          [-0.6430,  0.3412,  0.6656,  0.6985,  0.2691];
          [ 1.3433, -1.4387, -1.2028, -0.2997,  0.6933];
          [-0.4480, -0.7450,  0.9386, -1.3733,  1.1912]];
>> filters_0 = [[-1.1877,  0.2613,  1.7033];
         [ 0.5515, -0.5155, -0.8936];
         [-1.1671, -1.3611, -0.0778]];
>> filters_1 = [[-1.0069,  1.0255,  0.9102];
         [-0.4946,  0.8556,  1.4657];
         [ 0.1137, -0.2616,  0.5006]];
>> filters_2 = [[ 1.4465,  1.2084,  2.2803];
         [ 0.2097,  0.0633,  0.7705];
         [-0.7784, -0.9365, -0.9489]];
>> filters_3 = [[ 0.7263,  1.7227,  1.2436];
         [-0.9810,  0.2232,  2.0970];
         [ 0.4908,  0.0761, -1.2843]];
>> c_0 = xcorr2(inputs_0, filters_0);
>> c_1 = xcorr2(inputs_1, filters_1);
>> c_2 = xcorr2(inputs_2, filters_2);
>> c_3 = xcorr2(inputs_3, filters_3);
>> c_0 + c_1 + c_2 + c_3
ans =
   -0.2308   -0.9419    2.5054    1.6621    0.8919   -3.7085   -2.2283
   -2.6062   -1.0219   -1.0561   -1.2497   -2.6155   -1.1754   -0.3335
   -7.2686  -11.5207   -0.2484    0.6508   -6.4911   -3.3877   -1.5477
   -2.1475    0.4589    3.3011    0.6829    0.4209   -1.7198   -1.4318
    4.9216    0.9663   -0.4121    8.2640    4.4465    0.5085    1.1521
   -1.1531   -5.8454   -8.2619    0.1415    5.1501    0.7042    0.4525
    1.3501    1.9708    0.1334   -4.0152   -1.4114    5.0780   -1.3141

No suprisingly we will find that ans on matlab is equivalent to results[0, 0] on python.

For torch.nn.Conv2d, PyTorch add random weights to the cross correlation bigot multiplication, which make it more complex. Look at the following codes run in python terminal:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
>>> import torch
>>> import torch.nn as nn
>>> import os
>>> os.system("clear")
>>> input = torch.ones(2,3,4)
>>> input
tensor([[[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]],

        [[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]]])
>>> m = nn.Conv1d(3,1,1)
>>> m2= nn.Conv1d(3,1,1)
>>> m(input)
tensor([[[-0.3817, -0.3817, -0.3817, -0.3817]],

        [[-0.3817, -0.3817, -0.3817, -0.3817]]], grad_fn=<SqueezeBackward1>)
>>> m2(input)
tensor([[[0.4615, 0.4615, 0.4615, 0.4615]],

        [[0.4615, 0.4615, 0.4615, 0.4615]]], grad_fn=<SqueezeBackward1>)
>>> input[1]
tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])
>>> input[1] = 2 * input[1]
>>> input
tensor([[[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]],

        [[2., 2., 2., 2.],
         [2., 2., 2., 2.],
         [2., 2., 2., 2.]]])
>>> m(input)
tensor([[[-0.3817, -0.3817, -0.3817, -0.3817]],

        [[-0.3176, -0.3176, -0.3176, -0.3176]]], grad_fn=<SqueezeBackward1>)
#machine learning
Understanding forward and backprop in neural networks Vim config file on Linux desktop

Comments

You forgot to set the shortname for Disqus. Please set it in _config.yml.
Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×