2016 Update
This commit is contained in:
Tina Müller
parent
948b86eafa
commit
dcf5d15da3
7965 changed files with 139854 additions and 31002 deletions
|
|
@ -1,16 +1,36 @@
|
|||
Calculate the [[wp:Entropy (information theory)|information entropy]] (Shannon entropy) of a given input string.
|
||||
;Task:
|
||||
Calculate the Shannon entropy H of a given input string.
|
||||
|
||||
Entropy is the [[wp:Expected value|expected value]] of the measure of [[wp:Self-information|information]] content in a system.
|
||||
In general, the Shannon entropy of a variable <math>X</math> is defined as:
|
||||
:<math>H(X) = \sum_{x\in\Omega} P(x) I(x)</math>
|
||||
where the information content <math>I(x) = -\log_{b} P(x)</math>.
|
||||
If the base of the logarithm <math>b = 2</math>, the result is expressed in ''bits'', a [[wp:Units of information|unit of information]].
|
||||
Therefore, given a string <math>S</math> of length <math>n</math> where <math>P(s_i)</math> is the relative frequency of each character, the entropy of a string in bits is:
|
||||
:<math>H(S) = -\sum_{i=0}^n P(s_i) \log_2 (P(s_i))</math>
|
||||
For this task, use "<tt>1223334444</tt>" as an example.
|
||||
The result should be around 1.84644 bits.
|
||||
Given the discreet random variable <math>X</math> that is a string of <math>N</math> "symbols" (total characters) consisting of <math>n</math> different characters (n=2 for binary), the Shannon entropy of X in '''bits/symbol''' is :
|
||||
:<math>H_2(X) = -\sum_{i=1}^n \frac{count_i}{N} \log_2 \left(\frac{count_i}{N}\right)</math>
|
||||
|
||||
Related Tasks:
|
||||
where <math>count_i</math> is the count of character <math>n_i</math>.
|
||||
|
||||
:::* [[Fibonacci_word]]
|
||||
:::* [[Entropy/Narcissist]]
|
||||
For this task, use X="<tt>1223334444</tt>" as an example. The result should be 1.84644... bits/symbol. This assumes X was a random variable, which may not be the case, or it may depend on the observer.
|
||||
|
||||
This coding problem calculates the "specific" or "[[wp:Intensive_and_extensive_properties|intensive]]" entropy that finds its parallel in physics with "specific entropy" S<sup>0</sup> which is entropy per kg or per mole, not like physical entropy S and therefore not the "information" content of a file. It comes from Boltzmann's H-theorem where <math>S=k_B N H</math> where N=number of molecules. Boltzmann's H is the same equation as Shannon's H, and it gives the specific entropy H on a "per molecule" basis.
|
||||
|
||||
The "total", "absolute", or "[[wp:Intensive_and_extensive_properties|extensive]]" information entropy is
|
||||
:<math>S=H_2 N</math> bits
|
||||
This is not the entropy being coded here, but it is the closest to physical entropy and a measure of the information content of a string. But it does not look for any patterns that might be available for compression, so it is a very restricted, basic, and certain measure of "information". Every binary file with an equal number of 1's and 0's will have S=N bits. All hex files with equal symbol frequencies will have <math>S=N \log_2(16)</math> bits of entropy. The total entropy in bits of the example above is S= 10*18.4644 = 18.4644 bits.
|
||||
|
||||
The H function does not look for any patterns in data or check if X was a random variable. For example, X=000000111111 gives the same calculated entropy in all senses as Y=010011100101. For most purposes it is usually more relevant to divide the gzip length by the length of the original data to get an informal measure of how much "order" was in the data.
|
||||
|
||||
Two other "entropies" are useful:
|
||||
|
||||
Normalized specific entropy:
|
||||
:<math>H_n=\frac{H_2 * \log(2)}{\log(n)}</math>
|
||||
which varies from 0 to 1 and it has units of "entropy/symbol" or just 1/symbol. For this example, H<sub>n<\sub>= 0.923.
|
||||
|
||||
Normalized total (extensive) entropy:
|
||||
:<math>S_n = \frac{H_2 N * \log(2)}{\log(n)}</math>
|
||||
which varies from 0 to N and does not have units. It is simply the "entropy", but it needs to be called "total normalized extensive entropy" so that it is not confused with Shannon's (specific) entropy or physical entropy. For this example, S<sub>n<\sub>= 9.23.
|
||||
|
||||
Shannon himself is the reason his "entropy/symbol" H function is very confusingly called "entropy". That's like calling a function that returns a speed a "meter". See section 1.7 of his classic [http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20Theory%20of%20Communication.pdf A Mathematical Theory of Communication] and search on "per symbol" and "units" to see he always stated his entropy H has units of "bits/symbol" or "entropy/symbol" or "information/symbol". So it is legitimate to say entropy NH is "information".
|
||||
|
||||
In keeping with Landauer's limit, the physics entropy generated from erasing N bits is <math>S = H_2 N k_B \ln(2)</math> if the bit storage device is perfectly efficient. This can be solved for H<sub>2</sub>*N to (arguably) get the number of bits of information that a physical entropy represents.
|
||||
|
||||
;Related tasks:
|
||||
:* [[Fibonacci_word]]
|
||||
:* [[Entropy/Narcissist]]
|
||||
<br><br>
|
||||
|
|
|
|||
24
Task/Entropy/APL/entropy.apl
Normal file
24
Task/Entropy/APL/entropy.apl
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
ENTROPY←{-+/R×2⍟R←(+⌿⍵∘.=∪⍵)÷⍴⍵}
|
||||
|
||||
⍝ How it works:
|
||||
⎕←UNIQUE←∪X←'1223334444'
|
||||
1234
|
||||
⎕←TABLE_OF_OCCURENCES←X∘.=UNIQUE
|
||||
1 0 0 0
|
||||
0 1 0 0
|
||||
0 1 0 0
|
||||
0 0 1 0
|
||||
0 0 1 0
|
||||
0 0 1 0
|
||||
0 0 0 1
|
||||
0 0 0 1
|
||||
0 0 0 1
|
||||
0 0 0 1
|
||||
⎕←COUNT←+⌿TABLE_OF_OCCURENCES
|
||||
1 2 3 4
|
||||
⎕←N←⍴X
|
||||
10
|
||||
⎕←RATIO←COUNT÷N
|
||||
0.1 0.2 0.3 0.4
|
||||
-+/RATIO×2⍟RATIO
|
||||
1.846439345
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
defmodule RC do
|
||||
def entropy(str) do
|
||||
leng = String.length(str)
|
||||
String.split(str, "", trim: true)
|
||||
String.graphemes(str)
|
||||
|> Enum.group_by(&(&1))
|
||||
|> Enum.map(fn{_,value} -> length(value) end)
|
||||
|> Enum.reduce(0, fn count, entropy ->
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ import Data.List
|
|||
|
||||
main = print $ entropy "1223334444"
|
||||
|
||||
entropy s =
|
||||
sum . map lg' . fq' . map (fromIntegral.length) . group . sort $ s
|
||||
where lg' c = (c * ) . logBase 2 $ 1.0 / c
|
||||
fq' c = let sc = sum c in map (/ sc) c
|
||||
entropy :: (Ord a, Floating c) => [a] -> c
|
||||
entropy = sum . map lg . fq . map genericLength . group . sort
|
||||
where lg c = -c * logBase 2 c
|
||||
fq c = let sc = sum c in map (/ sc) c
|
||||
|
|
|
|||
19
Task/Entropy/Lua/entropy.lua
Normal file
19
Task/Entropy/Lua/entropy.lua
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
function log2 (x) return math.log(x) / math.log(2) end
|
||||
|
||||
function entropy (X)
|
||||
local N, count, sum, i = X:len(), {}, 0
|
||||
for char = 1, N do
|
||||
i = X:sub(char, char)
|
||||
if count[i] then
|
||||
count[i] = count[i] + 1
|
||||
else
|
||||
count[i] = 1
|
||||
end
|
||||
end
|
||||
for n_i, count_i in pairs(count) do
|
||||
sum = sum + count_i / N * log2(count_i / N)
|
||||
end
|
||||
return -sum
|
||||
end
|
||||
|
||||
print(entropy("1223334444"))
|
||||
|
|
@ -1,30 +1,30 @@
|
|||
/*REXX program calculates the information entropy for a given character string*/
|
||||
numeric digits 50 /*use 50 decimal digits for precision. */
|
||||
parse arg $; if $='' then $=1223334444 /*obtain the optional input from the CL*/
|
||||
#=0; @.=0; L=length($); $$= /*define handy-dandy REXX variables. */
|
||||
/*REXX program calculates the information entropy for a given character string. */
|
||||
numeric digits 50 /*use 50 decimal digits for precision. */
|
||||
parse arg $; if $='' then $=1223334444 /*obtain the optional input from the CL*/
|
||||
#=0; @.=0; L=length($); $$= /*define handy-dandy REXX variables. */
|
||||
|
||||
do j=1 for L; _=substr($,j,1) /*process each character in $ string.*/
|
||||
if @._==0 then do; #=#+1 /*Unique? Yes, bump character counter.*/
|
||||
$$=$$ || _ /*add this character to the $$ list. */
|
||||
do j=1 for L; _=substr($,j,1) /*process each character in $ string.*/
|
||||
if @._==0 then do; #=#+1 /*Unique? Yes, bump character counter.*/
|
||||
$$=$$ || _ /*add this character to the $$ list. */
|
||||
end
|
||||
@._=@._+1 /*keep track of this character's count.*/
|
||||
@._=@._+1 /*keep track of this character's count.*/
|
||||
end /*j*/
|
||||
sum=0 /*calculate info entropy for each char.*/
|
||||
do i=1 for #; _=substr($$,i,1) /*obtain a character from unique list. */
|
||||
sum=sum - @._/L * log2(@._/L) /*add (negatively) the char entropies. */
|
||||
sum=0 /*calculate info entropy for each char.*/
|
||||
do i=1 for #; _=substr($$,i,1) /*obtain a character from unique list. */
|
||||
sum=sum - @._/L * log2(@._/L) /*add (negatively) the char entropies. */
|
||||
end /*i*/
|
||||
|
||||
say ' input string: ' $
|
||||
say 'string length: ' L
|
||||
say ' unique chars: ' # ; say
|
||||
say 'the information entropy of the string ──► ' format(sum,,12) " bits."
|
||||
exit /*stick a fork in it, we're all done. */
|
||||
/*──────────────────────────────────LOG2 subroutine───────────────────────────*/
|
||||
log2: procedure; parse arg x 1 ox; ig= x>1.5; is=1-2*(ig\==1); ii=0
|
||||
numeric digits digits()+5 /* [↓] precision of E must be ≥ digits().*/
|
||||
e=2.7182818284590452353602874713526624977572470936999595749669676277240766303535
|
||||
do while ig & ox>1.5 | \ig&ox<.5; _=e; do k=-1; iz=ox* _**-is
|
||||
if k>=0 & (ig & iz<1 | \ig&iz>.5) then leave; _=_*_; izz=iz; end
|
||||
ox=izz; ii=ii+is*2**k; end; x=x* e** -ii-1; z=0; _=-1; p=z
|
||||
do k=1; _=-_*x; z=z+_/k; if z=p then leave; p=z; end /*k*/
|
||||
r=z+ii; if arg()==2 then return r; return r/log2(2,0)
|
||||
say 'the information entropy of the string ──► ' format(sum,,12) " bits."
|
||||
exit /*stick a fork in it, we're all done. */
|
||||
/*──────────────────────────────────────────────────────────────────────────────────────*/
|
||||
log2: procedure; parse arg x 1 ox; ig= x>1.5; ii=0; is=1 - 2 * (ig\==1)
|
||||
numeric digits digits()+5 /* [↓] precision of E must be≥digits()*/
|
||||
e=2.71828182845904523536028747135266249775724709369995957496696762772407663035354759
|
||||
do while ig & ox>1.5 | \ig&ox<.5; _=e; do j=-1; iz=ox* _**-is
|
||||
if j>=0 & (ig & iz<1 | \ig&iz>.5) then leave; _=_*_; izz=iz; end /*j*/
|
||||
ox=izz; ii=ii+is*2**j; end; x=x* e**-ii-1; z=0; _=-1; p=z
|
||||
do k=1; _=-_*x; z=z+_/k; if z=p then leave; p=z; end /*k*/
|
||||
r=z+ii; if arg()==2 then return r; return r/log2(2,.)
|
||||
|
|
|
|||
26
Task/Entropy/Run-BASIC/entropy.run
Normal file
26
Task/Entropy/Run-BASIC/entropy.run
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
dim chrCnt( 255) ' possible ASCII chars
|
||||
|
||||
source$ = "1223334444"
|
||||
numChar = len(source$)
|
||||
|
||||
for i = 1 to len(source$) ' count which chars are used in source
|
||||
ch$ = mid$(source$,i,1)
|
||||
if not( instr(chrUsed$, ch$)) then chrUsed$ = chrUsed$ + ch$
|
||||
j = instr(chrUsed$, ch$)
|
||||
chrCnt(j) =chrCnt(j) +1
|
||||
next i
|
||||
|
||||
lc = len(chrUsed$)
|
||||
for i = 1 to lc
|
||||
odds = chrCnt(i) /numChar
|
||||
entropy = entropy - (odds * (log(odds) / log(2)))
|
||||
next i
|
||||
|
||||
print " Characters used and times used of each "
|
||||
for i = 1 to lc
|
||||
print " '"; mid$(chrUsed$,i,1); "'";chr$(9);chrCnt(i)
|
||||
next i
|
||||
|
||||
print " Entropy of '"; source$; "' is "; entropy; " bits."
|
||||
|
||||
end
|
||||
12
Task/Entropy/ZX-Spectrum-Basic/entropy.zx
Normal file
12
Task/Entropy/ZX-Spectrum-Basic/entropy.zx
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
10 LET s$="1223334444": LET base=2: LET entropy=0
|
||||
20 LET sourcelen=LEN s$
|
||||
30 DIM t(255)
|
||||
40 FOR i=1 TO sourcelen
|
||||
50 LET number= CODE s$(i)
|
||||
60 LET t(number)=t(number)+1
|
||||
70 NEXT i
|
||||
80 PRINT "Char";TAB (6);"Count"
|
||||
90 FOR i=1 TO 255
|
||||
100 IF t(i)<>0 THEN PRINT CHR$ i;TAB (6);t(i): LET prop=t(i)/sourcelen: LET entropy=entropy-(prop*(LN prop)/(LN base))
|
||||
110 NEXT i
|
||||
120 PRINT '"The Entropy of """;s$;""" is ";entropy
|
||||
Loading…
Add table
Add a link
Reference in a new issue