2016 Update

This commit is contained in:
Tina Müller 2016-12-05 22:15:40 +01:00
parent 948b86eafa
commit dcf5d15da3
7965 changed files with 139854 additions and 31002 deletions

View file

@ -1,38 +1,118 @@
The classic [[wp:Hash Join|hash join]] algorithm for an inner join of two relations has the following steps:
<ul>
<li>Hash phase: Create a hash table for one of the two relations by applying a hash
function to the join attribute of each row. Ideally we should create a hash table for the
smaller relation, thus optimizing for creation time and memory size of the hash table.</li>
<li>Join phase: Scan the larger relation and find the relevant rows by looking in the
hash table created before.</li>
</ul>
An [[wp:Join_(SQL)#Inner_join|inner join]] is an operation that combines two data tables into one table, based on matching column values. The simplest way of implementing this operation is the [[wp:Nested loop join|nested loop join]] algorithm, but a more scalable alternative is the [[wp:hash join|hash join]] algorithm.
The algorithm is as follows:
{{task heading}}
'''for each''' tuple ''s'' '''in''' ''S'' '''do'''
'''let''' ''h'' = hash on join attributes ''s''(b)
'''place''' ''s'' '''in''' hash table ''S<sub>h</sub>'' '''in''' bucket '''keyed by''' hash value ''h''
'''for each''' tuple ''r'' '''in''' ''R'' '''do'''
'''let''' ''h'' = hash on join attributes ''r''(a)
'''if''' ''h'' indicates a nonempty bucket (''B'') of hash table ''S<sub>h</sub>''
'''if''' ''h'' matches any ''s'' in ''B''
'''concatenate''' ''r'' and ''s''
'''place''' relation in ''Q''
Implement the "hash join" algorithm, and demonstrate that it passes the test-case listed below.
'''Task:''' implement the Hash Join algorithm and show the result of joining two tables with it.
You should use your implementation to show the joining of these tables:
<table><tr><td><table border>
<tr><th>Age</th><th>Name</th></tr>
<tr><td>27</td><td>Jonah</td></tr>
<tr><td>18</td><td>Alan</td></tr>
<tr><td>28</td><td>Glory</td></tr>
<tr><td>18</td><td>Popeye</td></tr>
<tr><td>28</td><td>Alan</td></tr>
</table></td><td><table border>
<tr><th>Name</th><th>Nemesis</th></tr>
<tr><td>Jonah</td><td>Whales</td></tr>
<tr><td>Jonah</td><td>Spiders</td></tr>
<tr><td>Alan</td><td>Ghosts</td></tr>
<tr><td>Alan</td><td>Zombies</td></tr>
<tr><td>Glory</td><td>Buffy</td></tr>
</table></td></tr></table>
You should represent the tables as data structures that feel natural in your programming language.
{{task heading|Guidance}}
The "hash join" algorithm consists of two steps:
# '''Hash phase:''' Create a [[wp:Multimap|multimap]] from one of the two tables, mapping from each join column value to all the rows that contain it.<br>
#* The multimap must support hash-based lookup which scales better than a simple linear search, because that's the whole point of this algorithm.
#* Ideally we should create the multimap for the ''smaller'' table, thus minimizing its creation time and memory size.
# '''Join phase:''' Scan the other table, and find matching rows by looking in the multimap created before.
<br>
In pseudo-code, the algorithm could be expressed as follows:
'''let''' ''A'' = the first input table (or ideally, the larger one)
'''let''' ''B'' = the second input table (or ideally, the smaller one)
'''let''' ''j<sub>A</sub>'' = the join column ID of table ''A''
'''let''' ''j<sub>B</sub>'' = the join column ID of table ''B''
'''let''' ''M<sub>B</sub>'' = a multimap for mapping from single values to multiple rows of table ''B'' (starts out empty)
'''let''' ''C'' = the output table (starts out empty)
'''for each''' row ''b'' '''in''' table ''B''''':'''
'''place''' ''b'' '''in''' multimap ''M<sub>B</sub>'' under key ''b''(''j<sub>B</sub>'')
'''for each''' row ''a'' '''in''' table ''A''''':'''
'''for each''' row ''b'' '''in''' multimap ''M<sub>B</sub>'' under key ''a''(''j<sub>A</sub>'')''':'''
'''let''' ''c'' = the concatenation of row ''a'' and row ''b''
'''place''' row ''c'' in table ''C''
{{task heading|Test-case}}
{| class="wikitable"
|-
! Input
! Output
|-
|
{| style="border:none; border-collapse:collapse;"
|-
| style="border:none" | ''A'' =
| style="border:none" |
{| class="wikitable"
|-
! Age !! Name
|-
| 27 || Jonah
|-
| 18 || Alan
|-
| 28 || Glory
|-
| 18 || Popeye
|-
| 28 || Alan
|}
| style="border:none; padding-left:1.5em;" rowspan="2" |
| style="border:none" | ''B'' =
| style="border:none" |
{| class="wikitable"
|-
! Character !! Nemesis
|-
| Jonah || Whales
|-
| Jonah || Spiders
|-
| Alan || Ghosts
|-
| Alan || Zombies
|-
| Glory || Buffy
|}
|-
| style="border:none" | ''j<sub>A</sub>'' =
| style="border:none" | <code>Name</code> (i.e. column 1)
| style="border:none" | ''j<sub>B</sub>'' =
| style="border:none" | <code>Character</code> (i.e. column 0)
|}
|
{| class="wikitable" style="margin-left:1em"
|-
! A.Age !! A.Name !! B.Character !! B.Nemesis
|-
| 27 || Jonah || Jonah || Whales
|-
| 27 || Jonah || Jonah || Spiders
|-
| 18 || Alan || Alan || Ghosts
|-
| 18 || Alan || Alan || Zombies
|-
| 28 || Glory || Glory || Buffy
|-
| 28 || Alan || Alan || Ghosts
|-
| 28 || Alan || Alan || Zombies
|}
|}
The order of the rows in the output table is not significant.<br>
If you're using numerically indexed arrays to represent table rows (rather than referring to columns by name), you could represent the output rows in the form <code style="white-space:nowrap">[[27, "Jonah"], ["Jonah", "Whales"]]</code>.
<br><hr>

View file

@ -0,0 +1,129 @@
use framework "Foundation" -- Yosemite onwards, for record-handling functions
-- hashJoin :: [Record] -> [Record] -> String -> [Record]
on hashJoin(tblA, tblB, strJoin)
set {jA, jB} to splitOn("=", strJoin)
script instanceOfjB
on lambda(a, x)
set strID to keyValue(x, jB)
set maybeInstances to keyValue(a, strID)
if maybeInstances is not missing value then
updatedRecord(a, strID, maybeInstances & {x})
else
updatedRecord(a, strID, [x])
end if
end lambda
end script
set M to foldl(instanceOfjB, {name:"multiMap"}, tblB)
script joins
on lambda(a, x)
set matches to keyValue(M, keyValue(x, jA))
if matches is not missing value then
script concat
on lambda(row)
x & row
end lambda
end script
a & map(concat, matches)
else
a
end if
end lambda
end script
foldl(joins, {}, tblA)
end hashJoin
-- TEST
on run
set lstA to [¬
{age:27, |name|:"Jonah"}, ¬
{age:18, |name|:"Alan"}, ¬
{age:28, |name|:"Glory"}, ¬
{age:18, |name|:"Popeye"}, ¬
{age:28, |name|:"Alan"}]
set lstB to [¬
{|character|:"Jonah", nemesis:"Whales"}, ¬
{|character|:"Jonah", nemesis:"Spiders"}, ¬
{|character|:"Alan", nemesis:"Ghosts"}, ¬
{|character|:"Alan", nemesis:"Zombies"}, ¬
{|character|:"Glory", nemesis:"Buffy"}, ¬
{|character|:"Bob", nemesis:"foo"}]
hashJoin(lstA, lstB, "name=character")
end run
-- RECORD PRIMITIVES
-- keyValue :: String -> Record -> Maybe a
on keyValue(rec, strKey)
set ca to current application
set v to (ca's NSDictionary's dictionaryWithDictionary:rec)'s objectForKey:strKey
if v is not missing value then
item 1 of ((ca's NSArray's arrayWithObject:v) as list)
else
missing value
end if
end keyValue
-- updatedRecord :: Record -> String -> a -> Record
on updatedRecord(rec, strKey, varValue)
set ca to current application
set nsDct to (ca's NSMutableDictionary's dictionaryWithDictionary:rec)
nsDct's setValue:varValue forKey:strKey
item 1 of ((ca's NSArray's arrayWithObject:nsDct) as list)
end updatedRecord
-- GENERIC PRIMITIVES
-- foldl :: (a -> b -> a) -> a -> [b] -> a
on foldl(f, startValue, xs)
tell mReturn(f)
set v to startValue
set lng to length of xs
repeat with i from 1 to lng
set v to lambda(v, item i of xs, i, xs)
end repeat
return v
end tell
end foldl
-- map :: (a -> b) -> [a] -> [b]
on map(f, xs)
tell mReturn(f)
set lng to length of xs
set lst to {}
repeat with i from 1 to lng
set end of lst to lambda(item i of xs, i, xs)
end repeat
return lst
end tell
end map
-- splitOn :: Text -> Text -> [Text]
on splitOn(strDelim, strMain)
set {dlm, my text item delimiters} to {my text item delimiters, strDelim}
set lstParts to text items of strMain
set my text item delimiters to dlm
return lstParts
end splitOn
-- Lift 2nd class handler function into 1st class script wrapper
-- mReturn :: Handler -> Script
on mReturn(f)
if class of f is script then
f
else
script
property lambda : f
end script
end if
end mReturn

View file

@ -0,0 +1,84 @@
include FMS-SI.f
include FMS-SILib.f
\ Since the same join attribute, Name, occurs more than once
\ in both tables for this problem we need a hash table that
\ will accept and retrieve multiple identical keys if we want
\ an efficient solution for large tables. We make use
\ of the hash collision handling feature of class hash-table.
\ Subclass hash-table-m allows multiple entries with the same key.
\ After a get: hit one can inspect for additional entries with
\ the same key by using next: until false is returned.
:class hash-table-m <super hash-table
\ called within insert: method in superclass
:m (do-search): ( node hash -- idx hash false )
swap drop idx @ swap false ;m
:m next: ( -- val true | false )
last-node @ dup
if
begin
( node ) next: dup
while
dup key@: @: key-addr @ key-len @ compare 0=
if dup last-node ! val@: true exit then
repeat
then ;m
;class
\ begin hash phase
: obj ( addr len -- obj )
heap> string+ dup >r !: r> ;
hash-table-m R 1 r init
s" Whales " obj s" Jonah" r insert:
s" Spiders " obj s" Jonah" r insert:
s" Ghosts " obj s" Alan" r insert:
s" Buffy " obj s" Glory" r insert:
s" Zombies " obj s" Alan" r insert:
s" Vampires " obj s" Jonah" r insert:
\ end hash phase
\ create Age Name table S
o{ o{ 27 'Jonah' }
o{ 18 'Alan' }
o{ 28 'Glory' }
o{ 18 'Popeye' }
o{ 28 'Alan' } } value s
\ Q is a place to store the relation
object-list2 Q
\ join phase
: join \ { obj | list -- }
0 locals| list obj |
1 obj at: @: r get: \ hash the join-attribute and search table r
if \ we have a match, so concatenate and save in q
heap> object-list2 to list list q add: \ start a new sub-list in q
0 obj at: copy: list add: \ place age from list s in q
1 obj at: copy: list add: \ place join-attribute (name) from list s in q
( str-obj ) copy: list add: \ place first nemesis in q
begin
r next: \ check for more nemeses
while
( str-obj ) copy: list add: \ place next nemesis in q
repeat
then ;
: probe
begin
s each: \ for each tuple object in s
while
( obj ) join \ pass the object to function join
repeat ;
probe \ execute the probe function
q p: \ print the saved relation
\ free allocated memory
s <free
r free2:
q free:

View file

@ -20,7 +20,7 @@ hashJoin xs fx ys fy = runST $ do
Just v -> modifySTRef' l ((map (x,) v) ++)
readSTRef l
test = mapM_ print $ hashJoin
main = mapM_ print $ hashJoin
[(1, "Jonah"), (2, "Alan"), (3, "Glory"), (4, "Popeye")]
snd
[("Jonah", "Whales"), ("Jonah", "Spiders"),

View file

@ -7,10 +7,10 @@ import Control.Applicative
mapJoin xs fx ys fy = joined
where yMap = foldl' f M.empty ys
f m y = M.insertWith (++) (fy y) [y] m
joined = concat . catMaybes .
map (\x -> map (x,) <$> M.lookup (fx x) yMap) $ xs
joined = concat .
mapMaybe (\x -> map (x,) <$> M.lookup (fx x) yMap) $ xs
test = mapM_ print $ mapJoin
main = mapM_ print $ mapJoin
[(1, "Jonah"), (2, "Alan"), (3, "Glory"), (4, "Popeye")]
snd
[("Jonah", "Whales"), ("Jonah", "Spiders"),

View file

@ -0,0 +1,40 @@
import java.util.*;
public class HashJoin {
public static void main(String[] args) {
String[][] table1 = {{"27", "Jonah"}, {"18", "Alan"}, {"28", "Glory"},
{"18", "Popeye"}, {"28", "Alan"}};
String[][] table2 = {{"Jonah", "Whales"}, {"Jonah", "Spiders"},
{"Alan", "Ghosts"}, {"Alan", "Zombies"}, {"Glory", "Buffy"},
{"Bob", "foo"}};
hashJoin(table1, 1, table2, 0).stream()
.forEach(r -> System.out.println(Arrays.deepToString(r)));
}
static List<String[][]> hashJoin(String[][] records1, int idx1,
String[][] records2, int idx2) {
List<String[][]> result = new ArrayList<>();
Map<String, List<String[]>> map = new HashMap<>();
for (String[] record : records1) {
List<String[]> v = map.getOrDefault(record[idx1], new ArrayList<>());
v.add(record);
map.put(record[idx1], v);
}
for (String[] record : records2) {
List<String[]> lst = map.get(record[idx2]);
if (lst != null) {
lst.stream().forEach(r -> {
result.add(new String[][]{r, record});
});
}
}
return result;
}
}

View file

@ -0,0 +1,55 @@
(() => {
'use strict';
// hashJoin :: [Dict] -> [Dict] -> String -> [Dict]
let hashJoin = (tblA, tblB, strJoin) => {
let [jA, jB] = strJoin.split('='),
M = tblB.reduce((a, x) => {
let id = x[jB];
return (
a[id] ? a[id].push(x) : a[id] = [x],
a
);
}, {});
return tblA.reduce((a, x) => {
let match = M[x[jA]];
return match ? (
a.concat(match.map(row => dictConcat(x, row)))
) : a;
}, []);
},
// dictConcat :: Dict -> Dict -> Dict
dictConcat = (dctA, dctB) => {
let ok = Object.keys;
return ok(dctB).reduce(
(a, k) => (a['B_' + k] = dctB[k]) && a,
ok(dctA).reduce(
(a, k) => (a['A_' + k] = dctA[k]) && a, {}
)
);
};
// TEST
let lstA = [
{ age: 27, name: 'Jonah' },
{ age: 18, name: 'Alan' },
{ age: 28, name: 'Glory' },
{ age: 18, name: 'Popeye' },
{ age: 28, name: 'Alan' }
],
lstB = [
{ character: 'Jonah', nemesis: 'Whales' },
{ character: 'Jonah', nemesis: 'Spiders' },
{ character: 'Alan', nemesis: 'Ghosts' },
{ character:'Alan', nemesis: 'Zombies' },
{ character: 'Glory', nemesis: 'Buffy' },
{ character: 'Bob', nemesis: 'foo' }
];
return hashJoin(lstA, lstB, 'name=character');
})();

View file

@ -0,0 +1,7 @@
sub hash-join(@a, &a, @b, &b) {
my %hash := @b.classify(&b);
@a.map: -> $a {
|(%hash{a $a} // next).map: -> $b { [$a, $b] }
}
}

View file

@ -0,0 +1,17 @@
my @A =
[27, "Jonah"],
[18, "Alan"],
[28, "Glory"],
[18, "Popeye"],
[28, "Alan"],
;
my @B =
["Jonah", "Whales"],
["Jonah", "Spiders"],
["Alan", "Ghosts"],
["Alan", "Zombies"],
["Glory", "Buffy"],
;
.say for hash-join @A, *[1], @B, *[0];

View file

@ -1,18 +0,0 @@
my @A = [1, "Jonah"],
[2, "Alan"],
[3, "Glory"],
[4, "Popeye"];
my @B = ["Jonah", "Whales"],
["Jonah", "Spiders"],
["Alan", "Ghosts"],
["Alan", "Zombies"],
["Glory", "Buffy"];
sub hash-join(@a, &a, @b, &b) {
my %hash{Any};
%hash{.&a} = $_ for @a;
([%hash{.&b} // next, $_] for @b);
}
.perl.say for hash-join @A, *.[1], @B, *.[0];

View file

@ -0,0 +1,24 @@
(de A
(27 . Jonah)
(18 . Alan)
(28 . Glory)
(18 . Popeye)
(28 . Alan) )
(de B
(Jonah . Whales)
(Jonah . Spiders)
(Alan . Ghosts)
(Alan . Zombies)
(Glory . Buffy) )
(for X B
(let K (cons (char (hash (car X))) (car X))
(if (idx 'M K T)
(push (caar @) (cdr X))
(set (car K) (list (cdr X))) ) ) )
(for X A
(let? Y (car (idx 'M (cons (char (hash (cdr X))) (cdr X))))
(for Z (caar Y)
(println (car X) (cdr X) (cdr Y) Z) ) ) )

View file

@ -1,32 +1,31 @@
/*REXX program demonstrates the classic hash join algorithm for two relations.*/
S. = ; R. =
S.1 = 27 'Jonah' ; R.1 = 'Jonah Whales'
S.2 = 18 'Alan' ; R.2 = 'Jonah Spiders'
S.3 = 28 'Glory' ; R.3 = 'Alan Ghosts'
S.4 = 18 'Popeye' ; R.4 = 'Alan Zombies'
S.5 = 28 'Alan' ; R.5 = 'Glory Buffy'
hash.= /*initialize the hash table (array). */
do #=1 while S.#\==''; parse var S.# age name /*extract information*/
hash.name=hash.name # /*build a hash table entry with its idx*/
end /*#*/ /* [↑] REXX does the heavy work here. */
#=#-1 /*adjust for the DO loop (#) overage.*/
do j=1 while R.j\=='' /*process a nemesis for a name element.*/
parse var R.j x nemesis /*extract the name and its nemesis. */
if hash.x=='' then do; #=#+1 /*Not in hash? Then a new name; bump #*/
S.#=',' x /*add a new name to the S table. */
hash.x=# /* " " " " " " hash " */
end /* [↑] this DO isn't used today. */
do k=1 for words(hash.x); _=word(hash.x,k) /*get the pointer.*/
S._=S._ nemesis /*add the nemesis ──► applicable hash. */
end /*k*/
end /*j*/
_='' /*the character used for the separator.*/
pad=left('',6-2) /*spacing used in header and the output*/
say pad center('age',3) pad center('name',20 } pad center('nemesis',30 )
say pad center('',3) pad center('' ,20,_) pad center('' ,30,_)
/*REXX program demonstrates the classic hash join algorithm for two relations. */
S. = ; R. =
S.1 = 27 'Jonah' ; R.1 = "Jonah Whales"
S.2 = 18 'Alan' ; R.2 = "Jonah Spiders"
S.3 = 28 'Glory' ; R.3 = "Alan Ghosts"
S.4 = 18 'Popeye' ; R.4 = "Alan Zombies"
S.5 = 28 'Alan' ; R.5 = "Glory Buffy"
hash.= /*initialize the hash table (array). */
do #=1 while S.#\==''; parse var S.# age name /*extract information*/
hash.name=hash.name # /*build a hash table entry with its idx*/
end /*#*/ /* [↑] REXX does the heavy work here. */
#=#-1 /*adjust for the DO loop (#) overage.*/
do j=1 while R.j\=='' /*process a nemesis for a name element.*/
parse var R.j x nemesis /*extract the name and its nemesis. */
if hash.x=='' then do; #=# + 1 /*Not in hash? Then a new name; bump #*/
S.#=',' x /*add a new name to the S table. */
hash.x=# /* " " " " " " hash " */
end /* [↑] this DO isn't used today. */
do k=1 for words(hash.x); _=word(hash.x, k) /*obtain the pointer.*/
S._=S._ nemesis /*add the nemesis ──► applicable hash. */
end /*k*/
end /*j*/
_='' /*the character used for the separator.*/
pad=left('', 4) /*spacing used in header and the output*/
say pad center('age', 3) pad center("name", 20 ) pad center('nemesis', 30 )
say pad center('', 3) pad center("" , 20, _) pad center('' , 30, _)
do n=1 for #; parse var S.n age name nems /*get information.*/
if nems=='' then iterate /*No nemesis? Skip*/
say pad right(age,3) pad center(name,20) pad nems /*display an S. */
end /*n*/
/*stick a fork in it, we're all done. */
do n=1 for #; parse var S.n age name nems /*obtain information.*/
if nems=='' then iterate /*No nemesis? Skip. */
say pad right(age,3) pad center(name,20) pad center(nems,30) /*display an "S". */
end /*n*/ /*stick a fork in it, we're all done. */

View file

@ -0,0 +1,25 @@
sqliteconnect #mem, ":memory:"
#mem execute("CREATE TABLE t_age(age,name)")
#mem execute("CREATE TABLE t_name(name,nemesis)")
#mem execute("INSERT INTO t_age VALUES(27,'Jonah')")
#mem execute("INSERT INTO t_age VALUES(18,'Alan')")
#mem execute("INSERT INTO t_age VALUES(28,'Glory')")
#mem execute("INSERT INTO t_age VALUES(18,'Popeye')")
#mem execute("INSERT INTO t_age VALUES(28,'Alan')")
#mem execute("INSERT INTO t_name VALUES('Jonah','Whales')")
#mem execute("INSERT INTO t_name VALUES('Jonah','Spiders')")
#mem execute("INSERT INTO t_name VALUES('Alan','Ghosts')")
#mem execute("INSERT INTO t_name VALUES('Alan','Zombies')")
#mem execute("INSERT INTO t_name VALUES('Glory','Buffy')")
#mem execute("SELECT *,t_age.name FROM t_age LEFT JOIN t_name ON t_name.name = t_age.name")
WHILE #mem hasanswer()
#row = #mem #nextrow()
age = #row age()
name$ = #row name$()
nemesis$ = #row nemesis$()
print age;" ";name$;" ";nemesis$
WEND