methods of encoding

1.

Linear Function We choose a weight function, $\overrightarrow{w}$, for which the value function is $V_{w}(s)=\overrightarrow{w}\cdot\overrightarrow{H(s)}=\sum w_{i}u_{i}$. Naturally, we cannot calculate every function this way, but in many cases it gives good results. Another advantage is the simplicity of calculating the derivative $\nabla_{w}V_{w}(s)=\nabla_{w}\overrightarrow{w}\cdot\overrightarrow{u}=\overrightarrow{u}$. The derivative is the encoding of the state, which is very convenient computation-wise.

2.

Neural Networks (figure )

  

Figure: A Neural Network

  

Figure: Calculating The Value Of A Gate

Calculation of a gate (see figure ):
First, we compute $\alpha=\sum w_{i}z_{i}$. Then give $\alpha$ as an argument to a non linear function.

1.

perceptron : $h(\alpha) = \{^{1\ \ ,\alpha > 0} _{0\ \ ,\alpha \leq 0}$
The problem: it isn't a continuous function.

2.

sigmoid function: a continuous perceptron approximation,

$$\sigma(\alpha)=\frac{1}{1+e^{-\alpha}}\ .$$

Note that, $\sigma(0)=\frac{1}{2}$, and when $\alpha \rightarrow \infty$, $\sigma(\alpha)\rightarrow1$, and when $\alpha \rightarrow -\infty$, $\sigma(\alpha)\rightarrow0$.

It is possible to connect a large number of such gates, each gate has its own weight vector w_i. There are simple algorithms for computing the derivative by using the chain rule.

Basically, we are left with a learning problem: finding F(r,s) that corresponds to V^*. We are interested in two things:

• $V^{*} =\ ^{\max}_{\ \pi}\{V^{\pi}\}$ equivalent to $^{\max}_{\ r}\{F(r,s)\}$
• $V^{*} \cong F(r^{*},s)$