Value function iteration

We consider the following maximization problem

subject to

where both R and Q are negative as in 0506.

Here, we have two kinds of costs, x and v. The corresponding Bellman equation is

subject to

We start from V0=0 and v=0.

When j=0

That is

When j=1

FOC w.r.t. v is

That is

Plugging this v to Bellman equation at optimum, we get

That is

We can regard it as

When j=2

FOC w.r.t. v is

That is

Plugging this v to Bellman equation at optimum, we get

That is

We can regard it as

.

.

.

We go on and we find that

when j=n

where

that is not changed at all and it starts from .

All we have to do is find the convergence of in the limit and plug it into policy

function and value function.

Notice that initial value does not have to be R but we could start any point.