Centering and Scaling

NN_Pre

nn

Simple Neuron

Neuron0

The above diagram shows a neuron in NN, it simulates a real neuron:

it has inputs:

it has weights for each inputs: : weight vector

it has bias

it has a threshold for the “activation function”

Review

For the ith sample in the training set, we have the following loss function:

is the score classifying to class j,and is the score classifying correctly(classify to class ), is the th row of .

Problem

Problem 1:

Considering the geometrical meaning of the weight vector , it is easy to find out that is not unique, can change in a small area and result in the same .

Problem 2:

It the values in is scaled, the loss computed will also be scaled by the same ratio. Considering a loss of 15, if we scale all the weights in by 2, the loss will be scaled to 30. But this kind of scaling is meaningless, it doesn’t really represent the loss.

Traditional Thread

C++ 11 supports multi-threading, for which we previously used pthread or boost. Standard C++ 11’s threading is based on boost::thread, now it’s cross platform and no dependency needed.

#include <thread>
#include <iostream>
int testThread()
{
	std::cout << "I am a seperate thread...";
}
void main()
{
	std::thread newThread(testThread);//now it's running
        //do something else .... //
	newThread.join();//before the ending, newThread should join
        return;
}

The code is simple, the thread function will run right after the std::thread is declared.

Function binding

std::bind can create a function which is binded to another one:

void originalFunc(int a, double b, string c)
{
	cout << a << " " << b << " " << c.c_str() << endl;
}//original function, to be binded

void main()
{
  	//Newly binded function, created by bind function
  	auto newlyBindedFunc = bind(originalFunc, 
                                placeholders::_2, 
                                10.5,
                                placeholders::_1);
	newlyBindedFunc("Hello world",2);//call the new func
        originalFunc(2,10.5,"Hello world");//the same result
}

lambda expression can be understood as an anonymous function, it doesn’t need a name, it is created in runtime:

auto lam1 = []()
{
  cout<<"Hello World!";
};
lam1();

The code above shows a simplest lambda expression, when lam1() is called, it will print “Hello World!”.

How is the lambda expression existed in C++?

It is existed a closure class, each lambda is a unique closer class generated by the compiler in runtime. So it is a object and can be copied:

shared_ptr is a pointer allowed to be copied, it is like the normal pointer we use, but it’s smart for its counting function.

A normal pointer is not safe as when we are deleting it, we don’t know if someone else is using it. The shared_ptr count the users of this pointer, when someone release the pointer, its count will decrement, the memory block where it points will release only when the count becomes 0, which means no one is using it.

Initialize a shared_ptr:

shared_ptr<int> s{ new int{3} };

It can also be “made”, even made directly from value: