__MARKOV
PROCESSES__

Suppose a system has a finite number of states and that the
sysytem undergoes changes from state to state with a probability for each
distinct state transition that depends solely upon the current state. Then, the
process of change is termed a *Markov Chain*
or *Markov Process*.

__Definition__: If a
system featuring "n" distinct states undergoes state changes which
are strictly Markov in nature, then the probability that its current state is
"_{} "

given that its previous state was "_{} "
is the *transition probability*, "_{}".
The nxn matrix "_{}"
whose ij^{th} element is _{} is termed the *transition
matrix of the Markov chain*.

Each column vector _{} of the transition matrix is thus associated
with the preceding state. Since there are a total of "n" unique
transitions from this state, the sum of the components of _{} must add to "1", because it is a
certainty that the new state will be among the "n" distinct states.

__Definition__: The *state vector* for an observation of a Markov
chain featuring "n" distinct states is a column vector, _{}, whose
k^{th} component, _{}, is
the probability that the system is in state "_{} "
at that time.

Page 1 of 11

__Example # 1__: Drexel
finds that 70% of it alumni who contribute to the annual fund will also
contribute the next year and 20% of its alumni who do not contribute one year,
will contribute the next year. Determine the probability that a new graduated
student will be a contributor to the annual fund 10 years after she graduates.

_{}

_{ }

_{}

Now that we have the transition matrix, we need a state vector. In fact, we need a particular state vector, namely the initial state vector. Our newly minted graduate became an alumnus immediately upon graduation. While she was a student, she was not an alumnus and thus did not contribute to the annual fund previously. Her last state vector reflects that.

_{}

Page 2 of 11

_{}

_{}

_{}

_{}

Therefore, 10 years after graduation, only 40% of those
celebrating their 10^{th }reunion are likely to be contributors.

What happens after 20 years?

_{}

We obtain the same result! In other words, the state vector
converged to a steady-state vector. I this case that steady-state vector is _{}.

Page 3 of 11

__Theorem__: The
steady-state vector of the transition matrix "P" is the unique
probability vector that satisfies this equation: _{}.

That is true because, irrespective of the starting state, eventually equilibrium must be achieved. The transient, or sorting-out phase takes a different number of iterations for different transition matrices, but eventually the state vector features components that are precisely what the transition matrix calls for. So, subsequent applications of "P" do not change the matured state vector.

__Theorem__: State
transition matrices all have _{} as an eigenvalue.

We will just consider a 2x2 matrix here, but the result can be extended to an nxn.

_{}

_{}

_{}

Page 4 of 11

_{}

That proves the theorem for the 2x2 case. Now, let's find the other eigenvalue.

_{}

_{}

_{}

_{}

_{}

Since _{} and _{}, _{}.
Therefore, _{} is the dominant eigenvalue. This fact will
manifest itself when we demonstrate that the corresponding eigenvector is
indeed the steady state vector, _{}. Let's find that corresponding eigenvector.

_{}

Page 5 of 11

_{}

_{}

_{}

_{}

But the components of the vector _{} must add to "1". Otherwise, it can
not be a state vector.

_{}

__Example # 2__: Show
that the steady-state vector obtained in __Example
# 1__ is the eigenvector of the transition matrix corresponding to the
eigenvalue _{}.

Page 6 of 11

_{}

_{ }

_{}

If the steady- state vector is the eigenvector corresponding
to _{} and the steady-state vector can also be found
by applying "P" to any initial state vector a sufficiently large
number of times, "m", then _{} must approach a specialized matrix.

__Example # 3__: Find _{} for the matrix _{}, where
"N" is a very large positive integer.

_{}

Page 7 of 11

_{}

_{}

_{}

_{}

_{}

For both equations above to be true for all values of _{}, _{}. Then
we obtain these results.

_{ }

Page 8 of 11

_{}

__Example # 4__: A rental
car agency has three locations. Cars can be picked-up any one of the three
locations and returned to any other location including the location it was
picked-up at. The matrix, "P", below is the transition matrix of this
Markov Process. Determine the eigenvalues and eigenvectors; find the
steady-state vector; and express _{} in terms of the eigenvectors of "P".

_{}

Here the columns correspond to the pick-up locations and the
rows correspond to the return locations. For example, entry _{} is the probability that a car rented at
Location # 2 will be returned to Location # 3.

Page 9 of 11

_{}

Note that rows 1 and 2 are identical in "P".
Therefore, _{}. Also,
the sum of columns 1 and 2 is twice column 3 for _{}.
Finally, Markov processes have _{} The corresponding eigenvectors are found in
the usual way.

_{ }

_{ }

After a sufficient number of iterations, the state vector will nominally equal its steady-state vector. The eigenvector associated with the eigenvalue "1" determines the steady-state vector.

_{}

Page 10 of 11

_{}

_{}

_{}

Page 11 of 11