New principles of computer processing

NEW PRINCIPLES OF COMPUTER PROCESSING

Notion of information
The object having very close relationship to the entropy is information. Notion of information naturally arises in the following context. Suppose we have two experiments and ß which are not independent i.e. probability of outcome AjBk (we denote this probability by p(AjBk)) does not equal to the product of probabilities p(Aj) and p(Bk) any more. Instead, we have the relation

where pAj (Bk) is so-called conditional probability of outcome Bk, which equals to probability of Bk under condition that outcome Aj was realized. If experiments a and ß are independent, the realization of outcome Aj can in no way influence experiment ß and pAj(Bk) = p(Bk).

Now it is natural to ask the question: Is it possible to give a numerical value to the measure of uncertainty of experiment ß under condition of realization of experiment a?. The answer turns out to be positive again; this measure of uncertainty is given by conditional entropy, which may be written in the form

It is a function of probabilities p(Aj) and the matrix of conditional probabilities pAj (Bk).
It turns out to be possible to prove that conditional entropy always satisfies inequality

Therefore, we can define non-negative value

which shows how much the uncertainty of experiment ß decreases if we know the result of the experiment . Value of I(, ß) is called the amount of information about experiment ß contained in experiment .

It is possible to verify the following properties of information defined by (2.8):

Information is symmetric:

i.e. the amount of information about experiment ß carried by experiment a always equals to the amount of information about experiment a carried by experiment ß.
Information is zero: I(, ß) = 0 if and only if experiments and ß are independent.
If experiments a and ß coincide, information reduces to entropy of experiment a:
Therefore, the entropy () of experiment a may be interpreted as the total amount of information which we obtain after carrying out experiment (if the experiment a does not contain any uncertainty i.e. () = 0, we do not gain any information after realization of the experiment, in full accordance with intuitive notion of information).
If , ß, ? are three arbitrary experiments, the following inequality always takes place:

i.e. the information about experiment carried by combined experiment ßy is always greater or equal then the information about experiment a carried by experiment ß alone.

The listed properties make intuitive justification of the use of word "information" for variable I(, ß). An additional, already empirical, justification will be given below in sect. 2.3.

Random Variables »

« Back to Home