{"id":292,"date":"2019-02-24T11:11:13","date_gmt":"2019-02-24T11:11:13","guid":{"rendered":"https:\/\/tutorials.retopall.com\/?p=292"},"modified":"2019-03-28T21:06:54","modified_gmt":"2019-03-28T21:06:54","slug":"convolutional-neural-networks","status":"publish","type":"post","link":"https:\/\/tutorials.retopall.com\/index.php\/2019\/02\/24\/convolutional-neural-networks\/","title":{"rendered":"Convolutional Neural Networks"},"content":{"rendered":"\n<p>Convolutional Neural Networks are widely used for image classification. With some filters we can simplify an colored image with its most important parts. The main feature of a Convolutional Network is the convolution operation where each filters goes over the entire input image and creates another image.<\/p>\n\n\n\n<p> Also you can watch the video where I explain how they work in a simple way. The Convolutional Neural Network tutorials also will explain the code to create it and represent it in a 3D visualization.<\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" width=\"860\" height=\"484\" src=\"https:\/\/www.youtube.com\/embed\/oC41-M1rYZE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>A ConvNet or Convolutional Neural Network (CNN) is a complex model of artificial intelligence that outputs some prediction of certain inputs. It includes a<a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/03\/01\/feed-forward-neural-network\/\"> Feed-Forward Neural Network<\/a> layer but also some special layers we will analyze soon. <\/p>\n\n\n\n<p>They are used for important companies such as Google, Nvidia for image classification, autonomous cars, speech recognition, face detection and camera. The convolution operation is an operator that can easily create different image effects with some filters that change the aspect of the image. In Artificial Intelligence is often used for edge detection.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Convolutional Neural Network<\/h4>\n\n\n\n<p>Convolutional Neural Networks are mainly used to analyze RGB images. These images will perform as the input data of the neural network. RGB images often require more performance to be analyzed. There are many more weights to process than in the traditional  multilayer <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/03\/01\/feed-forward-neural-network\/\">Feed-Forward Neural Network.<\/a> It will have a higher computational cost. With ConvNets, we can simplify the image and get the most important parts of it to decrease the input data that will be processed afterwards.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1-1024x584.png\" alt=\"\" class=\"wp-image-659\" width=\"825\" height=\"470\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1-1024x584.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1-300x171.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1-768x438.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1.png 1734w\" sizes=\"auto, (max-width: 825px) 100vw, 825px\" \/><\/figure><\/div>\n\n\n\n<p>This networks will have different layer depending on their functionalities and features:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Convolutional Layer<\/strong><\/li><li><strong>Activation Layer<\/strong><\/li><li><strong>Pooling Layer<\/strong><\/li><li><strong>Fully connected Layer<\/strong><\/li><li><strong>Output Layer<\/strong><\/li><\/ul>\n\n\n\n<p>Some of this layers will be repeated in a ConvNet architecture more than one time.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvREluPooling-1024x616.png\" alt=\"\" class=\"wp-image-611\" width=\"601\" height=\"361\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvREluPooling-1024x616.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvREluPooling-300x180.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvREluPooling-768x462.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvREluPooling.png 1174w\" sizes=\"auto, (max-width: 601px) 100vw, 601px\" \/><figcaption>Le-Net Architecture<\/figcaption><\/figure><\/div>\n\n\n\n<p>In the image we have the most typical layers of the Convolutional Neural Network. The input data is an image RGB and will proccess it with the different layers.<\/p>\n\n\n\n<p>The activation layer, fully connected layer and output layer are very similar to the Feed-Forward Neural Network. However they have also some differences.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">FEATURE IMAGES<\/h4>\n\n\n\n<p>To undestand deeply how these neural networks work, we will need to define some terms:<\/p>\n\n\n\n<p>Each image that is created with the convolution operation, pooling or activation function and the input image will be a <strong>feature image<\/strong>. This images will have some size. They will have a normal <strong>width (W)<\/strong>and <strong>height (H) <\/strong>but also <strong>depth<\/strong> or number of <strong>channels (C) <\/strong>of the image.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/parts.png\" alt=\"\" class=\"wp-image-612\" width=\"261\" height=\"286\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/parts.png 584w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/parts-274x300.png 274w\" sizes=\"auto, (max-width: 261px) 100vw, 261px\" \/><\/figure><\/div>\n\n\n\n<p>In the input RGB image will have 3 channels: red, green and blue. This  size will be represented with the next notation <strong>D@WxH<\/strong><\/p>\n\n\n\n<p>We will start with those of the Convolutional Neural Network.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"convolution-operation\">CONVOLUTIONAL LAYER<\/h3>\n\n\n\n<p>This is the most important layer. It&#8217;s defined as an special operation that we will apply to a channel with a filter. This <strong>filter (F) <\/strong>will have weights and a bias that will be the parameters with which the neural network will learn.<\/p>\n\n\n\n<p>These filters or also named <strong>kernels<\/strong> that will be used to create the output image of the layer. They will have to be the same depth as the input image. However the size <strong>k x k<\/strong> of the kernels will need to be less than the size of the feature image.<\/p>\n\n\n\n<p>We can apply as many filters as we want. In the next example there are 6 filters. The output image will have the depth of the amount of filters we have, in this case 6.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvNet-1-1024x715.png\" alt=\"\" class=\"wp-image-614\" width=\"494\" height=\"344\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvNet-1-1024x715.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvNet-1-300x209.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvNet-1-768x536.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvNet-1-100x70.png 100w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvNet-1.png 1262w\" sizes=\"auto, (max-width: 494px) 100vw, 494px\" \/><\/figure><\/div>\n\n\n\n<p>Convolutional operation is similar to the cross-correlation but the kernel will be &#8220;flipped&#8221;. The input image will be denoted with <strong>I<\/strong> and the output image <strong>C<\/strong>.<\/p>\n\n\n\n<p>So the convolutional operation will be will be the equivalent in to the<strong> feed-forward <\/strong>function of a multilayer neural network:<\/p>\n\n\n\n<p>$$C(x,y)=I * F=\\sum_{m=0}^{k-1}{\\sum <br>_{n=0}^{k-1} {\\sum_{C}{K_{m,n,c}\\cdot I_{i+m,j+n,c}+b}}} $$<\/p>\n\n\n\n<p>The filter values will be firstly initialized with a normal distribution or complete random in the range [-1,1]. Then the weights and bias of the filter will be modified in the training. <\/p>\n\n\n\n<p>If we want to represent the convolutional layer in a 3D operation:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Conv3DOperation-1024x508.png\" alt=\"\" class=\"wp-image-617\" width=\"566\" height=\"281\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Conv3DOperation-1024x508.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Conv3DOperation-300x149.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Conv3DOperation-768x381.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Conv3DOperation.png 1322w\" sizes=\"auto, (max-width: 566px) 100vw, 566px\" \/><\/figure><\/div>\n\n\n\n<p>In the convolution operation, the input image will be the first blue square. We will get a little square that will be the <strong>same size <\/strong>of the filter (purple square) in width, height and depth and we will get a unique value (gray square).  We will do this repeatedly to cover the entire input image.<\/p>\n\n\n\n<p>The final <strong>depth <\/strong>of the output image of the convolution layer will be <strong>1<\/strong> but the witdh and the height will vary.<\/p>\n\n\n\n<p>A convolutional layer will have some hyper-parameters:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Input image size (I)<\/strong><\/li><li><strong>Kernel size (K)<\/strong><\/li><li><strong>Stride (S)<\/strong><\/li><li><strong>Padding (P)<\/strong><\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">STRIDE (S)<\/h4>\n\n\n\n<p>It&#8217;s an integer number that will be the number of steps we need to move to get an extract of the input image. When the extract has reached the latest position in the row it will go the stride number steps in the column movement.<\/p>\n\n\n\n<p>In this example, the stride is 2 so we will move 2 steps each time. Each color represent a new extract of the input image. The size of this extract is 3x3x1 so the filter will have the same size.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Stride.png\" alt=\"\" class=\"wp-image-619\" width=\"369\" height=\"286\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Stride.png 880w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Stride-300x233.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Stride-768x596.png 768w\" sizes=\"auto, (max-width: 369px) 100vw, 369px\" \/><\/figure><\/div>\n\n\n\n<h4 class=\"wp-block-heading\">PADDING (P)<\/h4>\n\n\n\n<p>We will put a zero padding around the image. The padding integer number will be the size of it.<\/p>\n\n\n\n<p>For example, if we have padding 1 we will put only one line of zeros around the image:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Padding-1024x640.png\" alt=\"\" class=\"wp-image-620\" width=\"502\" height=\"313\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Padding-1024x640.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Padding-300x188.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Padding-768x480.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Padding.png 1062w\" sizes=\"auto, (max-width: 502px) 100vw, 502px\" \/><\/figure><\/div>\n\n\n\n<p>With all of this paramets we will be able to calculate the final output image size:<\/p>\n\n\n\n<p>$$size=\\frac{I+2P-K}{S}+1$$<\/p>\n\n\n\n<p>For example if we have a padding of 1, stride 1 and an input image of 28x28x3 and the kernel 5x5x3, the output size when we apply the convolution operation of the image will be 24.<\/p>\n\n\n\n<p>Now we will see a mathematical example of the convolution operation:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvOperation-1024x622.png\" alt=\"\" class=\"wp-image-621\" width=\"609\" height=\"370\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvOperation-1024x622.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvOperation-300x182.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvOperation-768x466.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/ConvOperation.png 1346w\" sizes=\"auto, (max-width: 609px) 100vw, 609px\" \/><\/figure><\/div>\n\n\n\n<p>We will operate the convolution of $I*F$ and get the output green image with the convolutional operation formula.<\/p>\n\n\n\n<p>Now we will move to the Activation Layer:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">ACTIVATION LAYER<\/h3>\n\n\n\n<p>When we apply the convolution operation, we will normally use an activation layer to handle which values will be activated and which not.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Activation.png\" alt=\"\" class=\"wp-image-629\" width=\"379\" height=\"358\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Activation.png 781w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Activation-300x283.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Activation-768x725.png 768w\" sizes=\"auto, (max-width: 379px) 100vw, 379px\" \/><\/figure><\/div>\n\n\n\n<p>There are multiple <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/03\/25\/activation-functions-neural-networks\/\">activation functions<\/a> but the most used for ConvNets is the <strong>ReLU <\/strong>function:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1-1024x601.png\" alt=\"\" class=\"wp-image-624\" width=\"627\" height=\"367\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1-1024x601.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1-300x176.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1-768x451.png 768w\" sizes=\"auto, (max-width: 627px) 100vw, 627px\" \/><\/figure><\/div>\n\n\n\n<p>The RELU also has some variants such as <strong>PReLU:<\/strong><\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-1024x569.png\" alt=\"\" class=\"wp-image-625\" width=\"643\" height=\"357\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-1024x569.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-300x167.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-768x427.png 768w\" sizes=\"auto, (max-width: 643px) 100vw, 643px\" \/><\/figure><\/div>\n\n\n\n<p>Also we can use the <strong>sigmoid <\/strong>function but this activation is usually used in other types of network such as the Feed-Forward Neural Network.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1024x675.png\" alt=\"\" class=\"wp-image-626\" width=\"677\" height=\"445\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1024x675.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-300x198.png 300w\" sizes=\"auto, (max-width: 677px) 100vw, 677px\" \/><\/figure><\/div>\n\n\n\n<p>To calculate the output of the activation layer we will apply the activation function to each value and get a feature image of the same size as the input but with the values activated.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/RElu.png\" alt=\"\" class=\"wp-image-628\" width=\"482\" height=\"312\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/RElu.png 905w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/RElu-300x195.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/RElu-768x499.png 768w\" sizes=\"auto, (max-width: 482px) 100vw, 482px\" \/><\/figure><\/div>\n\n\n\n<p>There is also a very important layer to simplify the image:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">POOLING LAYER<\/h3>\n\n\n\n<p>This layer will usually be applied after the activation layer. It&#8217;s used to reduce the size in width and height of an image.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Pooling.png\" alt=\"\" class=\"wp-image-631\" width=\"413\" height=\"399\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Pooling.png 781w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Pooling-300x290.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Pooling-768x741.png 768w\" sizes=\"auto, (max-width: 413px) 100vw, 413px\" \/><\/figure><\/div>\n\n\n\n<p> In this example the image have been reduced by the half of the original size.<\/p>\n\n\n\n<p>In the pooling layer there are some hyper parameters:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Size of pooling<\/strong> (similar to the size of the filter)<\/li><\/ul>\n\n\n\n<p>With the extract of the original image with the size of pooling we will apply the pooling)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Stride <\/strong>(same as the convolutional stride)<\/li><\/ul>\n\n\n\n<p>The pooling can be done in two main methods:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Max pooling<\/strong><\/li><\/ul>\n\n\n\n<p>In the extract of the image we will only take the highest value and assigned it to the output image<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Average pooling<\/strong><\/li><\/ul>\n\n\n\n<p>However, with the average value, the ouput value will be the average of the extract of the image.<\/p>\n\n\n\n<p>This is an example of the average pooling size of 2&#215;2 with stride 2:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Average-Pooling.png\" alt=\"\" class=\"wp-image-632\" width=\"504\" height=\"372\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Average-Pooling.png 816w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Average-Pooling-300x222.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Average-Pooling-768x568.png 768w\" sizes=\"auto, (max-width: 504px) 100vw, 504px\" \/><\/figure><\/div>\n\n\n\n<p>Now we will start with the other layers that are very similar to the feed-forward network.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">FULLY CONNECTED LAYER<\/h3>\n\n\n\n<p>When we finally do all the previous layers we will finally get output final image and all the image will be reduced to an unidimensional array for the input array of the fully connected layer. This is also called <strong>flattening <\/strong>and we can apply a <strong>batch normalization.<\/strong> This will only be done in the first fully connected layer.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Flattening-1.png\" alt=\"\" class=\"wp-image-639\" width=\"211\" height=\"366\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Flattening-1.png 540w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Flattening-1-173x300.png 173w\" sizes=\"auto, (max-width: 211px) 100vw, 211px\" \/><\/figure><\/div>\n\n\n\n<p>Each fully connected layer will be completely connected  to the next layer. This layers will have some predifined neurons that will store a value.<\/p>\n\n\n\n<p>Wen we want to update the network, in this layer we will do the feed-forward function.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Fully-Connected.png\" alt=\"\" class=\"wp-image-645\" width=\"279\" height=\"318\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Fully-Connected.png 669w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Fully-Connected-263x300.png 263w\" sizes=\"auto, (max-width: 279px) 100vw, 279px\" \/><\/figure><\/div>\n\n\n\n<p>The<strong> feed-forward <\/strong>function in this layer is the same as the feed-forward explained in the <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/03\/01\/feed-forward-neural-network\/\">Feed-Forward Neural Network tutorial<\/a>. With the input that in this case will be the output of another layer and the weights that connected two fully connected layers we will calculate the next neurons values.<\/p>\n\n\n\n<p>Also, as we will apply an <strong>activation function<\/strong> such as the ReLU, Sigmoid or Hiperbolic Tanh after the feed-forward.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/NeuralNetwork.png\" alt=\"\" class=\"wp-image-355\" width=\"195\" height=\"193\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/NeuralNetwork.png 376w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/NeuralNetwork-300x297.png 300w\" sizes=\"auto, (max-width: 195px) 100vw, 195px\" \/><\/figure><\/div>\n\n\n\n<p>In the example you can see a traditional feed-forward neural network. If we are in  a convolutional layer they will be fully connected layers, in this case 3 layers. <\/p>\n\n\n\n<p>However the output layer will give the final prediction of the entire ConvNet and we will treat the data in a different way.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">OUTPUT LAYER<\/h3>\n\n\n\n<p>This layer will be always the final layer of the network and will have a different activation function. This values will be the predictions of the neural network. If we are in image classification they will be the prediction confidence of the network.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/sOFTMAX.png\" alt=\"\" class=\"wp-image-650\" width=\"260\" height=\"345\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/sOFTMAX.png 644w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/sOFTMAX-226x300.png 226w\" sizes=\"auto, (max-width: 260px) 100vw, 260px\" \/><\/figure><\/div>\n\n\n\n<p>This  layer will have an special activation function called <strong>Softmax <\/strong>that will depend on all the output of the final layer.<\/p>\n\n\n\n<p>$$\\sigma(a)_j=\\frac{e^{a_j}}{\\sum_{k}{e^{a_k}}}$$<\/p>\n\n\n\n<p>Where $\\sigma(a)$ is the activation function and  $j$ is the actual neuron we want to activate.  The sum will loop through all the output neuron values with position $k$.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">CONVOLUTIONAL NEURAL NETWORK ARCHITECTURES<\/h2>\n\n\n\n<p> With all of this layers we will can combine these and create different architectures depending on their use we want to give them. <\/p>\n\n\n\n<p>If we want to use it for digit or small RGB images we can use the <strong>Le-NET architecture<\/strong> (1998)<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/LeNet-1024x584.png\" alt=\"\" class=\"wp-image-655\" width=\"661\" height=\"377\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/LeNet-1024x584.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/LeNet-300x171.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/LeNet-768x438.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/LeNet.png 1734w\" sizes=\"auto, (max-width: 661px) 100vw, 661px\" \/><\/figure><\/div>\n\n\n\n<p>Firstly we have too iterations of the <strong>Convolution-ReLU-Pooling<\/strong> and the we flaten the final feature images and assign them to a Feed-Forward Neural Network of 2 hidden layers (fully connected layers). Finally we apply the softmax activation to the output layer.<\/p>\n\n\n\n<p>We can use also this architecture for RGB images so the input image will have 3 channels:<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1024x584.png\" alt=\"\" class=\"wp-image-657\" width=\"658\" height=\"375\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-1024x584.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-300x171.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet-768x438.png 768w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/Rgb-lenet.png 1734w\" sizes=\"auto, (max-width: 658px) 100vw, 658px\" \/><\/figure><\/div>\n\n\n\n<p>There are many architecture that we can use for image classification. We have the <strong>Alex Net<\/strong> (2012), ResNet, DenseNet that are more modern architectures and are used nowadays. To understand deeply these  convolutional neural networks I recommend you to see this link: <a href=\"https:\/\/www.jeremyjordan.me\/convnet-architectures\/\">https:\/\/www.jeremyjordan.me\/convnet-architectures\/<\/a>.<\/p>\n\n\n\n<p>In this presentation you will be able to see the images and the information you must know to understand neural networks.<\/p>\n\n\n\n[embeddoc url=&#8221;https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/02\/Convolutional-Neural-Network-2.pptx&#8221; download=&#8221;all&#8221; viewer=&#8221;microsoft&#8221;]\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Convolutional Neural Networks are widely used for image classification. With some filters we can simplify an colored image with its most important parts. The main [&#8230;]<\/p>\n","protected":false},"author":1,"featured_media":296,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,5,46,6],"tags":[49,47,54,48,51,50,53,52],"class_list":["post-292","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","category-explanation","category-neural-networks","category-tutorial","tag-convolution-operation","tag-convolutional-neural-network","tag-feature-map","tag-filter","tag-fully-connected","tag-pooling-layer","tag-relu","tag-softmax"],"_links":{"self":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts\/292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/comments?post=292"}],"version-history":[{"count":30,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts\/292\/revisions"}],"predecessor-version":[{"id":766,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts\/292\/revisions\/766"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/media\/296"}],"wp:attachment":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/media?parent=292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/categories?post=292"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/tags?post=292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}