A C C E P T E D
F A C U L T Y OF G R AD U A T E S T U D I E S
DATE.
INDEXICAL PARALLEL PROGRAMMING
by W eichang D u
B.Sc., Beijing Polytechnical U niversity, 1983 M .Sc., U niversity of V ictoria, 1987
A D issertation S ubm itted in P a rtia l Fulfillm ent of th e R equirem ents for th e Degree of
D O C T O R O F PH IL O SO PH Y in th e D ep artm en t of C om puter Science
W e accept this thesis as conform ing to the required sta n d a rd
D r. W . W . Wiadge, Supervisor (D ep artm en t of C om puter Science) —
Dr. G. C. Shoja, D e p a rtm en ta l M em ber (D e p a rtm e n t of C om p u ter Science)
Dt.M /. IW fey y , D ep artm en ta l M em ber (D ep artm en t of C o m p u ter Science)
DEAN
-Dr^i'G. G . Millef. O utside M em ber (D e p a rtm e n t of M athem atics)
Dr! C. G. M organ, O utside M eidbeW D epartm ent of Philosophy)
D r. R. J a ^ n n a t h a n , E x tern al E xam iner (S R I In te rn atio n a l)
© W E IC H A N G D U , 1991 U niversity of V ictoria
All rights reserved. Thesis m ay n o t be reproduced in whole o r in p a rt, by m im eograph or o th er m eans, w ithout th e perm ission of th e author.
Supervisor: Dr. William W. W adge
A b stract
Indexical program m ing means program m ing languages a n d /o r com putational m odels based on indexical logic and possible world sem antics. Indexical languages can be considered as the result of enriching conventional languages by allowing constructs to vary according to an im plicit context or index. Program s w ritten in an indexical language define th e way in which objects vary from context to context, using context sw itching or indexical op erato rs to combine m eanings of objects from different contexts.
Based on indexical sem antics, in indexical program s, c o n t e x t p a r a l le l is m m eans th a t com putations of objects a t different contexts can be perform ed in parallel, and i n d e x ic a l c o m m u n ic a t io n m eans th a t parallel com putation tasks a t different contexts com m uni cate w ith each other through indexical operators provided by the indexical language.
T he dissertation defines th e indexical functional language m Lucid - a m ultidim ensional extension of th e program m ing language Lucid proposed by A shcroft and W adge. T he ianguage enriches the functional language ISW IM by in co rp o ratin g functional sem antics w ith indexical sem antics. T he indexical sem antics of m Lucid is based on the context space consisting of points in an a rb itra ry n-dim ensional integer space. T he m eanings of o bjects, called intensions, in m Lucid are functions from these contexts to d a ta values. T he language'provides five prim itive indexical operators, origin, next, prev, fby and before to sw itch context along a designated dimension.
T he dim ensionality of an intension in the indexical sem antics of m Lucid is defined as the set of dimensions th a t determ ines the range of th e context space in w hich the intension varies. An a b stra ct in terp retatio n are defined th a t m aps m Lucid expressions to approxim ations of dim ensionalities. C ontext parallelism an d indexical com m unication in m Lucid program s are defined by a sem antics-based dependency relation betw een the values of variables a t different contexts.
sion and space dimensions. T h e time dimension can be used to specify tim e steps in synchronous com putations, or to specify indices of d a ta stream s in asynchronous com p u tatio n s. T he space dimensions can be used to specify process-to-processor m appings, T h e dissertation shows th a t mLucid supports several parallel program m ing models, in cluding systolic program m ing, m ultidim ensional dataflow program m ing, and d a ta parallel program m ing.
Exam iners:
D r. W. W . W adge, Supervisor (D ep artm en t of C om puter Science)
D r. G. C. Shoja, D ep artm en tal M em ber (D ep artm en t of C o m p u ter Science)
D r. A /p '^ ^ ^ rjrJ jD e p a rtm e n ta l M em ber (D e p a rtm e n t of C o m p u ter Science)
7—...—... -... ... ... ...
D r.,G . G. M iller, O utside M em ber (D ep artm en t of M athem atics)
D r. C. G. M organ, O u tsid e M em ber (l)e]p&rtment of Philosophy)
A ck n ow led gem ents
F irst and forem ost, I would like to express my deep g ra titu d e to my supervisor Bill W adge who, th ro u g h his insights and careful guidance, has m ade this dissertation possible. He has provided me with timely encouragem ent and advice as well as an outstan d in g am ount of freedom to pursue research issues I considered im p o rta n t while at the same time c rit ically appraising my work. M ost im portantly, Bill has th a t invaluable asset of the good supervisor, an ever open door.
I would like to th an k my oth er com m ittee m em bers, Dr. G. Shoja, Dr. M. Levy, D r. G. M iller and Dr. C. M organ, for th eir conscientious reading of the dissertation and suggestions.
Special th an k s go to Dr. Bill McColl for suggesting th e topic of this dissertation and discussions in the early stages of this research.
I would also like to th an k all m em bers of the D ep artm en t of C om puter Science at U niversity of V ictoria for the friendly and pleasant w orking environm ent, especially m em bers of the functional-program m ing group in the d ep a rtm e n t with whom I have had very stim ulating discussions.
I was sup p o rted financially at U niversity of V ictoria by a University of V ictoria Fel lowship an d a BC Advanced System s G ra d u a te Scholarship.
vi
C ontents
Abstract
ii
Acknowledgements
v
Contents
vi
List of Figures
vi
List of Tables
vii
1 INTRODUCTION
1
1.1 W h y Parallel P r o g r a m m i n g ... 1
1.2 Issues of P arallel P rogram m ing . . ... 2
1.3 T h e Conflicting Goals of Parallel P r o g r a m m in g ... 3
1.4 Problem s of E xisting Parallel Program m ing P a r a d ig m s ... 3
1.5 A Solution from Indexical P r o g r a m m in g ... 6
1.6 Synopsis ... 8
2 BACKGROUND AND RELATED WORK
11
2.1 Indexical Logic and Indexical P r o g r a m m in g ... 11C O N T E N T S vii
2.2 T he Dataflow Program m ing Language Lucid ... 12
2.3 Recent Development of L u c i d ... 18
2.4 Systolic A r r a y s ...20
2.5 T he Functional Language C r y s ta l ... 23
2.6 Parallel Functional P ro g ra m m in g ... 26
2.7 Systolic Logic P ro g ra m m in g ... 28
3 THE INDEXICAL FUNCTIONAL LANGUAGE mLucid
32
3.1 Syntax and Sem antics of m L u c i d ...323.1.1 S y n ta x ... 32
3.1.2 Indexical Sem antics of E x p r e s s io n s ...34
3.1.3 Pointw ise Extension of ISW IM O p e r a t o r s ...35
3.1.4 Pointwise Extension of C o n s ta n ts ...37
3.1.5 Functional Sem antics of mLucid ...38
3.2 Sem antics of Prim itive Indexical O p e r a t o r s ... 41
4 DIMENSIONALITY ANALYSIS
45
4.1 D e f in itio n ...454.2 Dim ensionalities of Prim itive O p e r a tio n s ... 51
4.3 D e c l a r a ti o n ... 56
4.4 A n a ly s is ... 60
5 CONTEXT PARALLELISM AND INDEXICAL COMMUNICATION 70
5.1 D irect V ariable-V alue D e p e n d e n c e ... 70C O N T E N T S viii
5.3 C ontext Dependence G raph and C ontext P a r a lle lis m ...84
5.4 Indexical C o m m u n ic a tio n ...90
0
SPECIFICATION OF SYSTOLIC ARRAYS
94
6..1 A M athem atical Model of Systolic A r r a y s ...946.2 D e s c r i p t io n ... 97
6.3 E x a m p l e s ... ...106
7
MULTIDIMENSIONAL DATAFLOW PROGRAM M ING
116
7.1 Overview . ...1167.2 D e s c r i p t io n ... 118
7.3 E x a m p l e s ... 119
8
DATA PARALLEL PROGRAMMING
133
8.1 O v e r v i e w ...1338.2 D e s c r i p t io n ...134
8.3 E x a m p l e s ... 139
9
CONCLUSION AND FUTURE WORK
146
9.1 S u m m a r y ... 146 9.2 Conclusions ... 148 9.3 L i m i t a t i o n s ... 149 9.4 C o n tr ib u tio n s ... ... ... 150 9.5 F u tu re Work ... 150 9.5.1 Parallel Functional P ro g ra m m in g ...150C O N T E N T S ix
9.5.2 D istributed P r o g r a m m in g ... 152
9.5.3 Parallel Im plem entation S t r a t e g i e s ...153
9.5.4 O t h e r s ...155
List o f Figures
2.1 A dataflow n e tw o rk ... ... 17
2.2 A m atrix m ultiplication systolic a r r a y ... 22
5.1 T he variable-value dependence graph of a m atrix m ultiplication program . 81 5.2 T he fine context dependence g raph of a m atrix m ultiplication program . . . 87
5.3 T h e context dependence graph of a m atrix m ultiplication p r o g r a m ...88
5.4 T h e coarse context dependence graph of a m atrix m ultiplication program . 88 5.5 T h e indexical com m unication of a m atrix m ultiplication p r o g r a m ...93
6.1 A systolic array for m atrix m ultiplication ... 96
6.2 T h e context dependence graph of a systolic m atrix m ultiplication program . 105 6.3 Systolic array B1 for c o n v o lu tio n ... 107
6.4 Systolic array B2 for c o n v o lu tio n ... 108
6.5 Systolic array III for c o n v o lu tio n ... 110
6.6 Systolic array R2 for c o n v o lu tio n ...112
6.7 Systolic array W 1 for c o n v o lu tio n ... 113
6.8 Systolic array W 2 for c o n v o lu tio n ... 114
L I S T OF F I GU R E S xi
7.1 T he initializer module of a m ultidim ensional dataflow network for prime num ber g e n e r a t i o n ... 120
7.2 T he behavior of whenever filter . . . ... 122
7.3 T he generator module of a m ultidim ensional dataflow netw ork for prim e num ber g e n e r a t i o n ...122
7.4 A m ultidim ensional dataflow netw ork for generating prim e num bers . . . . 123
7.5 T he dataflow direction of (a) th e elim ination and (b) th e back-substitution phases of a m ultidim ensional dataflow network for Gauss elim ination . . . . 124
7.6 T he functionality of the (a) pivot and (b) cell processors In the elimination phase ...124
7.7 T he functionality of the (a) pivot a,nd (b) cell processors in the back- su b stitu tio n p h a s e ... 1.25
7.8 T h e pivot dataflow m odule for G auss e lim in a tio n ... 127
7.9 T he cell dataflow m odule for Gauss e lim in a tio n ... 129
8.1 C om puting p artial sums of an array of 16 e le m e n t s ... 1.41
List of Tables
4.1 A dim ensionality exam ple
C h a p te r 1
IN T R O D U C T IO N
1.1
W h y P arallel P rogram m ing
As th e use of com puters affects increasingly broader segm ents of the world, m any of th e problem s to which people apply com puters grow continually larger and more complex. D em ands for faster and larger com puter systems increase steadily. C om puter architectures have followed two general approaches in response to these dem ands. T he first approach uses novel technology in conventional serial com puter architectures. In the past decade, c om puter technology has advanced rapidly and the speed of serial com puters has increased h u n dreds of tim es, b u t new problem s th a t need even faster com puters to solve are still em erging. T h e dem and for faster and larger com puter system s seems to outpace the technology. Also in th e conceivable future, technologies for developing com puter hardw are will a p p ro ach the physical lim it, th e speed of light, for tran sm ittin g electronic signals. T h is raises th e question, to satisfy these demands, “can we still m ake a great leap in p erform ance while th e rate of technology im provem ent rem ains relatively co n sta n t? ”
T h e second approach exploits the parallelism inherent in m any problem s. It allows m any p a rts of an application program to be com puted on different com puting elem ents sim ultaneously and to combine th e results of the subcom putations as the final solution, T h e parallel approach seems to offer the best long-term strategy because as the problem s grow , m ore a n d m ore opportunities arise to exploit the parallelism inherent in the problem s
C H A P T E R 1. I N T R O D U C T I O N 2
them selves.
Parallel com puters are becoming increasingly w idespread. T he question is w hat to do with them . Parallelism is only useful in solving problem s when it can be expressed in program s of a program m ing language, and the expression of parallelism can also be identified by the compilers of th e language and im plem ented on p a ra le l com puters. T he task of parallel program m ing research is to make the power of parallelism accessible to program m ers and detectable by compilers.
1.2
Issu es o f P arallel Program m ing
T h e re are th re e issues in parallel program m ing distinct from sequential program m ing.
• Expression of parallelism in a program of a program m ing language by identifying pieces of the program , i.e. com putation tasks, th a t can be executed in parallel.
• Expression of com m unication and synchronization betw een com putation tasks in parallel execution of the program .
• Given a parallel com puter architecture, expression of a m apping from com putation tasks o n to com puting elem ents or processors of the architecture.
Parallel program m ing can be classified into two categories: explicit parallel program m ing and im plicit parallel program m ing. In explicit parallel program m ing, program s are w ritte n w ith explicit parallelism in so-called parallel program m ing languages. A p a ra l lel program m ing language provides special constructs for program m ers to express one or m ore of th e above issues explicitly in th e ir program s. T h e m eanings of th e constructs are usually in te rp re te d by the operational sem antics of the language, instead of its m a th em atical sem antics. In c o n trast, in im plicit parallel program m ing, program s are w ritten in conventional program m ing languages w ithout explicit constructs for expressing any of th e above issues. Parallelism a n d com m unication in program s of a language w ith o u t ex plicit parallel constructs are autom atically detected by the language’s compilers for targ et parallel m achines.
C H A P T E R 1. I N T R O D U C T I O N 3
1.3
T h e C onflicting G oals o f P arallel Program m ing
T here are two m ain goals th a t should be achieved by a program m ing language. On the ap plication side, program m ers need a program m ing language th a t is a b stra ct enough to allow them to concentrate on solving th eir problem s w ith o u t bothering about im plem entation details. On the im plem entation side, efficient com pilation of program s in a program m ing language require t h a t inform ation on th e relationships betw een program structures and targ e t m achine architectures can be easily exploited by compilers.
These two goals conflict because the form er needs languages to be independent of un derlying com puter architectures, whereas th e la tte r needs languages’ operational semantics to be close to m achine architectures on which th eir program s will be executed. T he task of any program m ing language design is to harm onize the conflicting goals.
In parallel program m ing, expression of parallelism affects both sides of the problem. On the application side, th e stru c tu re of a program m ing language m ust help program m ers to write program s w ith more parallelism . On the im plem entation side, it should be possible and easy for compilers to d etect parallelism in program s. Expression of com m unication affects im plem entations of parallel program s in th e way th a t it m ust provide compilers w ith enough and clear inform ation ab o u t how parallel tasks com m unicate. T h e conflicting goals appearing on th e two sides of th e problem causes a dilem m a in the choice of languages for parallel program m ing. For application program m ers, im plicit parallel program m ing is preferred, w hereas for efficient parallel im plem entations, explicit parallel program m ing is needed.
1.4
P rob lem s o f E xistin g P a ra llel P rogram m ing Paradigm s
For im plicit parallel program m ing, there are m ainly two classes of program m ing languages: im perative languages and declarative languages. Im plicit parallel program m ing in conven tional languages like F o rtran , Pascal and C is a ttra c tiv e to a certain ex ten t. It absolves program m ers, who are fam iliar w ith sequential program m ing, from thinking ab o u t p a r allelism, and allows them to ru n th eir old program s on new parallel com puters without,
C H A P T E R 1. I N T R O D U C T I O N 4
rew riting. However, this approach has problem s when high perform ance of sequential pro gram s on parallel com puters is considered, as sequentialism is im plicit in the operational sem antics o f the b n g u ag e. A parallel compiler m ay not be able to discover the artificial sequentialization in a sequential program im posed by th e program m er, and as a result, parallelism in the program can n o t be fully exploited by analyzing th e d a ta dependency. Also, as sequential program m ers are encouraged to w rite program s th a t are efficient in th e sense of sequential im plem entation instead of parallel im plem entation, there m ay not be too m uch inherent parallelism in th eir program s.
Im plicit parallel program m ing in declarative languages is prom ising, because declara tive program s have im plicit parallelism . For exam ple, in functional program s, expression parallelism occurs when argum ents of a function application a n d /o r th e argum ents and th e function itself can be evaluated in parallel. In logic program s, A N D /O R parallelism occurs when two subgoals of a clause or two clauses can be com puted in parallel. Implicit parallelism in declarative languages m akes it easy for compilers to identify p o ten tia l p ar allelism in declarative program s. However, there are still problem s in im plem enting these languages efficiently on parallel com puters. C om m unication betw een parallel com putation tasks in m ost declarative languages is expressed im plicitly through shared variables, which results in com plexity in analysis of com m unication p a tte rn s in the program s an d makes it difficult for the compilers to m ap parallel processes onto processors in parallel systems. Also, since th e expression parallelism and A N D /O R parallelism are usually fine-grain, granularity control is needed to compile the program s to parallel com puters w ith coarse granularity. On the o th er h a n d , for parallel com puters w ith massively parallel com puta tion p o ten tial, the expression parallelism and th e A N D /O R parallelism are not sufficient to encourage program m ers to w rite program s w ith m assive parallelism .
For explicit parallel program m ing, existing parallel program m ing languages can roughly be classified into th ree categories: extensions of conventional languages such as Parallel Pascal [Ree84] and C* [Thi86]; specially designed im perative parallel languages such as C SP [Hoa85] and L inda [ACG86]; and extensions of declarative languages such as parallel functional languages [Hud88] a n d parallel logic languages [Sha86].
C H A P T E R 1. I N T R O D U C T I O N
In parallel program m ing languages based on th e im perative program m ing principle, explicit parallel constructs provide program m ers w ith facilities to express parallelism and com m unication in th e ir program s. In parallel program m ing languages based on the declar a tiv e program m ing principle, explicit an notations are needed for program m ers to express control inform ation on parallelism , com m unication and m apping, so th a t efficient imple m en tatio n s on p a rticu la r parallel architectures can be achieved.
T h e comm on problem for explicit parallel, program m ing is th a t it obviously diverges from the goal of freeing program m ers from im plem entation details. Explicit parallelism p u ts a heavy burden on the program m ers’ shoulders. Explicitly expressing parallelism, com m unication and even process-to-processor m apping are irrelevant to the program m ers’ task s for finding solutions for their problem s. To make their program s run efficiently on parallel m achines, they h ave to u nderstand the operational m eanings of the explicit co n stru cts, an d th en use th em to express com plex parallel com putation activities in the program s, a t the sam e tim e co n cen tratin g on solving their problems. However, depending on th e c u rre n t developm ent of compiler techniques, explicit parallel program m ing is still a n effective approach to m aking im perative program s parallelized and allowing declarative program s to be im plem ented efficiently on parallel com puters.
T he problem s o f th e existing parallel program m ing paradigm s discussed above have raised a question. Is it possible to have a kind of program m ing language th a t is a better c a n d id a te for harm onizing th e conflicting goals? W ithout altering or extending its se m an tics, th is language could be used for both im plicit and explicit parallel program m ing depending on program m ers’ desires. For im plicit parallel program m ing, the language encourages program m ers to w rite program s w ith large scale parallelism and regular com m u n icatio n . For explicit parallel program m ing, the language provides program m ers with m eans to express precisely in their program s p artitions of parallel com putation tasks, com m unication and synchronization betw een th e tasks and m apping from the tasks to processors. For efficient parallel im plem entations, the language provides compilers with easily detectable inform ation on parallelism and com m unication in program s, so th a t the com pilers can easily tailo r needed inform ation for efficient parallel im plem entations. Also,
C H A P T E R 1. I N T R O D U C T I O N 6
to make the language application-oriented, precise and rigorous in specifying parallel com p u tatio n s, th e language’s expressive power for parallelism and com m unication and the ease of parallel im plem entation are implied in its m athem atical sem antics, w ithout depending on some kind of irrelevant parallel operational m eaning.
1.5
A S olu tion from In dexical Program m ing
T h is dissertation proposes a solution to harm onizing the conflicting goals of parallel pro gram m ing based on th e indexical program m ing paradigm .
Indexical program m ing m eans program m ing languages a n d /o r co m putational models based on indexical logic and possible world sem antics. Indexical program m ing languages can be considered as th e result of enriching conventional program m ing languages in the functional, logic, or im perative paradigm s. An indexical program m ing language is ob tained by allowing co n structs (expressions, predicates, com m ands) in th e base language to vary according to an im plicit context or index. T he im plicit context could be a m om ent in tim e, a p o in t in a space, a node in a tree, a word in a language - all depending on the ta rg e t application area. Program s w ritten in an indexical program m ing language define th e way in which o b jects vary from context to context, using context sw itching operators (such as ’’n e x t” , ” u p ” , or ’’p a re n t” ) to combine m eanings from different contexts.
Indexical sem antics of indexical languages can be used to in te rp re t parallelism and com m unication, called c o n t e x t p a r a lle lis m and in d e x ic a l c o m m u n ic a tio n , in index ical program s. C ontext parallelism in an indexical program means t h a t com putations of objects a t different contexts can be perform ed in parallel. Indexical com m unication m eans th a t parallel com p u tatio n tasks a t different contexts com m unicate w ith each o th er through co n text-sw itching or indexical o perators provided by the indexical language. C ontext p a r allelism and indexical com m unication are based on indexical operational Semantics of the language, and are distin ct from o th er kinds of parallelism and com m unication in the con ventional languages.
C H A P T E R 1. I N T R O D U C T I O N 7
for the meanings of objects at different contexts and indexical com m unication is specified through the use of indexical operators in the definitions. Indexical sem antics, as a com ponent of the m athem atical sem antics of an indexical program , is a p a rt of the solution to the problem th a t th e program solves. T he indexical sem antics can also be interpreted as the expression of parallelism and com m unication in th e program . This duality of indexical sem antics m eans two things. (1) On th e application side, it im plicitly forces program m ers to w rite program s w ith context parallelism , when they are solving their problem s. (2) On the im plem entation side, it provides compilers with a global view about where and when parallelism and com m unication occur in program s.
For exam ple, given a vector A, to com pute a vector B whose elem ent a t position p is the average of th e elem ents of A a t positions p - 1 and p + 1, w e1 can define B in an indexical language as follows, where th e im plicit context of th e language is a point in one dim ensional integer space:
B = ( l e f t A + r i g h t A ) /2 .
In the definition, left and right are indexical operators; a t a point in space, they switch context from the c u rre n t point to its neighbors on th e left and rig h t, respectively. T h e context parallelism specified in the definition is th a t the value of B a t each point in the space can be com puted in parallel. T h e indexical com m unication specified by the indexical operators in the definition is th a t to com pute th e value of B a t a point p in space, p has to com m unicate w ith th e points p - 1 and p -f 1, so th a t A ’s elem ents a t these points can be fetched.
In explicit parallel program m ing using indexical languages, a parallel com putation task can be expressed as a subcom putation perform ed a t a context. All the subcom putations a t a context represent the com putation activity of a processor. T h e parallelism am ong com putations a t different processors is expressed by context parallelism . T h e com m uni cation betw een the processors is expressed by indexical com m unication. By the rigorous m athem atical m eaning of indexical sem antics, using context parallelism and indexical com
p i l e uses o f “we” in this dissertation are ju st a sty le o f w riting. In m ost o f places it m eans the author
C H A P T E R 1. I N T R O D U C T I O N 8
m unication, the program m er can precisely specify and form ally verify parallel com putation and communication behaviors and process-processor m appings in indexical program s.
For example, given lists A and B of the sam e len g th , consider the following parallel com putation. F irst let processor 0 sort A and processor 1 sort B, th e n let processor 0 sum th e sorted A and B pairwise. T h e following definitions in an indexical language specify th e parallel com putation, whciv. the im plicit c o n te x t of th e language is also a point in one dimensional integer space and th e processor 0 an d 1 correspond to th e contexts 0 and 1, respectively.
SA = s o r t _ l i s t ( A ) a t 0; SB = s o r t _ l i s t ( B ) a t 1;
SUM = s u m _ l i s t ( S A , r i g h t SB) a t 0;
In th e definitions, at is an indexical operator; a t a point p in th e space, x a t q switches context from point p to point q and accesses x ’s value at q. T he process-processor m apping specified by th e indexical operator at is th a t SA and SUM are com puted a t context 0 corresponding to processor 0 and SB is com puted a t context 1 corresponding to processor 1. T he parallelism specified in the definitions is th a t A and B can be sorted at contexts 0 a n d 1 in parallel. T h e com m unication specified by th e indexical o p e ra to r right in th e definitions is th a t to com pute the value of SUM a t context 0, th e co n tex t 0 has to com m unicate w ith context 1 so th a t SB can be fetched from context 1.
1.6
Synopsis
In C hapter 2, we introduce some background a n d survey related work. T h e background includes the concept of indexical logic a n d indexical program m ing, th e dataflow program m ing language Lucid and its recent developm ent and systolic array s. T h e related work reviews the functional systolic program m ing language CRYSTAL, th e parallel functional program m ing paradigm and the systolic logic program m ing paradigm .
In C hapter 3, we define the sy n ta x and sem antics of m L ucid, including the form al sem antics of m L ucid’s prim itive indexical o p e ra to rs which serve as context sw itching op
C H A P T E R 1. I N T R O D U C T I O N 9
e ra to rs to combine values a t different contexts.
In C h a p te r 4, we define th e notion of dimensionalities of mLucid objects. The di m ensionality of an mLucid object is a set of dimensions in the context space in which th e value of the object varies. We also study the relationship between m Lucid’s indejrical sem antics and dim ensionalities of its objects. We provide a formal approach to analyzing dim ensionalities of various objects in mLucid program s based on an a b stra ct interpretation technique.
In C h a p te r 5, we define th e concept of context parallelism and indexical com m unication in an m Lucid program by th e dependency relation among values of variables a t various co n tex ts. We first define w hat is called direct dependency between two values of variables a t two contexts. Based on the direct dependency relation, a d a ta dependency graph, called variable-value dependency graph, of the program is defined. T he cot text dependency g rap h of th e program is then defined as a contraction of the variable-value dependency g rap h , in which only the dependency betw een values of variables a t different contexts is considered. Based on the context dependency graph, context parallelism and indexical com m unication in th e program are formally defined by the connectivity of th e graph.
In C h a p te r 6, we describe specifications of systolic arrays in mLucid program s. We choose systolic arrays as an application of mLucid, in order to show th e expressive power of th e language for parallel program m ing, because systolic arrays as a ty p e of special-purpose com puters have b o th the tem poral and spatial architecture-level parallelism . Moreover, th eir tem p o ral regularity in term s of com putations a t processors and spatial regularity in term s of com m unication am ong the processors, are best suited to be specified by context parallelism and indexical com m unication of mLucid program s. We also describe a class of m Lucid program s called SysLucid and show th a t the systolic array models and SysLucid program s are sem antically equivalent. We then describe how parallel com putation and com m unication activities in systolic arrays can be specified by the context parallelism and indexical com m unications of SysLucid program s.
In C h a p te r 7, we describe a m ultidim ensional dataflow program m ing paradigm in rnLu- cid. We show how to w rite mLucid program s w ith m ultidim ensional pipeline parallelism
C H A P T E R 1. I N T R O D U C T I O N 10
on a m ultidim ensional a b stract dataflow machine.
In C hap ter 8, we study th e relationship between context parallelism and d a ta paral lelism in mLucid program s. We describe how th e common themes of d a ta parallel pro gram m ing, i.e. elem entwise parallelism , parallel conditional, replication, reduction and p e rm u tatio n , are supported by mLucid constructs.
In the la st ch ap ter, we sum m arize th e contributions of the dissertation, and discuss fu ture work.
11
C h a p te r 2
B A C K G R O U N D A N D
RELATED W O RK
2.1
In d exical Logic and In d exical Program m ing
Indexical logic if a subset of intensional logic th a t originated in the study of ‘intensioiial’ phenom ena in hum an reasoning, such as m odality, knowledge, or (low of tim e. These all require a richer sem antic picture th a n sta n d a rd tru th values in one sta tic environm ent. Such a picture is provided by so-called possible worlds sem antics [vB8S].
In indexical logic, th e m eaning of a n a tu ra l language expression depends on an implicit co n tex t, called a possible world. For exam ple, th e m eaning of the expression today’s
tem perature depends on a possible world - th e day on which th e expression is uttered. The
essential id ea of indexical logic is based on th e distinction betw een extension and intension of an expression. T h e extension of an expression is the m eaning of the expression in a given possible world. For instance, the tem p e ra tu re on July 1, 1991 in the above example is 20 c. T h e intension of an expression captures the way !n which its extensions correspond to possible worlds; it is a function m apping possible worlds to extensions. For the above exam ple, the intension of the expression is th e set of pairs consisting of dates and degrees o f tem p eratu re.
C H A P T E R 2, B A C K G R O U N D A N D R E L A T E D W O R K 12
possible worlds, indexical logic also provides such operators, called indexical operators, th a t serve to switch contexts. Using indexical operators, th e m eaning of an expression in a possible world can be expressed as a com bination of possibly the m eaning of this expression and other expressions in other possible worlds. For exam ple, the m eaning of the expression
Today’s tem perature is five degree less than yesterday’s tem perature can be expressed as a
form ula yesterday(t) - 5. In th e form ula, t represents today 's tem p eratu re, and yesterday is an indexical o p erato r th a t switches context from a possible w orld, namely today, to another possible world, nam ely yesterday. T he result of th e application yesterday(t) on a given date is the tem p e ra tu re of the day before th e given d ate.
Indexical program m ing, originally proposed by Faustini and W adge [FW86b], means program m ing in a language th a t at th e sam e tim e is a form al system w ith indexical sem antics. In an indexical program m ing language, the value of a sy n tactic object, such as an expression, a predicate or a com m and, depends on an im plicit context; it is an intension ( a m ap) from a set of possible worlds, called the universe of possible worlds, to a base value dom ain. A possible world can be broadly defined, such as a m om ent in tim e, a node in a tree, an d a p o in t in a space, depending on targ et applications of the language. A program in th e language as an object also has indexical m eaning, t h a t is, it is a function m apping intensions of in p u t variables to intension(s) as o u tp u t. A,n indexical language provides users w ith a set of indexical o p e ra to rs whose definitions are based on th e intended m eaning of possible worlds and targ et applications. Using indexical operators, users can define the value of an object in a possible world as a com bination of values of o th er objects a n d /o r itself in o th er possible worlds.
2.2
T h e D ataflow P rogram m in g Language Lucid
Lucid is th e first indexical program m ing language, though th e concept of indexical pro gram m ing was proposed 10 years after th e language was originally introduced [AW76a] [AW76b] [A.W79]. Lucid is an enrichm ent of L a n d in ’s functional program m ing language ISW IM [Lan66],
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K L3
An ISW IM program is simply an expression called the defining expression. T he output, of th e program is th e value of the defining expression. An expression in ISW IM can be p a rt of a where clause. The expression is then called the subject of th e where clause. A
where clause also has a body which consists of a set of equations, called the body of the
clause. T h e left hand side of an equation in the body is a variable sym bol w ith certain arity ; and th e right hand side of the equation is an expression, which m ay contain other
where clauses. The body of a where clause defines a local environm ent for the value of its
su b ject. For exam ple, the following is an ISW IM program th a t com putes the factorial of 5.
f a c ( 5 ) w here
f a c ( n ) = i f n eq 1 t h e n 1 e l s e n * f a c ( n - i ) f i ; e n d .
ISW IM is not a single language; it is actually a param eterized family of languages, each m em ber of which is determ ined by a choice of underlying d a ta stru c tu re s. T h e different m em bers of this fam ily look very similar; they are identical in their o uter syntax, i.e. their
where clause structures. A m em ber of the ISW IM family is determ ined by an algebra.
Given a continuous algebra A, the m em ber of th e family determ ined by A is ISW IM (A ). T h e language ISW IM (A), its syntax and sem antics, is entirely determ ined by th e algebra A . Given an algebra A, its signature S determ ines th e sy n tax of ISW IM (A ), where a signature is a collection of individual and operation constants to g eth er with an arity for each operation constant. A n ISW IM (A) program is one in which the expressions are built up from nullary variables and from individual constants in S. Expressions are built up by applying operation symbols in S to appropriate num ber of operands, by applying function variables to argum ent lists and by form ing where clauses. T he individual and function variables defined in a where clause are called th e locals of the clause, where each local of a clause has only one definition in the clause.
Given an algebra A w ith signature S, the sem antics of a program in ISW IM (A ) con sisting of expression E w ith respect to an in te rp re ta tio n I is defined as follows. Here an
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 14
in te rp re ta tio n is a m athem atical object which extends A by giving m eanings to individual an d function variables as well as to th e constants. If E consists of an individual variable o r constant c, th en the value of E with respect to I is th e value which I assigns to c. If E consists of an n-ary operation constant or function variable G together w ith actual param eters E \ , E2, - - - , E n , then th e value of E w ith respect to I is g { v i , V2, • • • ,u „ ), where g is the operation over the universe of A which I assigns to G and each v{ is the m eaning
of E{ w ith respect to I. If E consists of a where clause then th e value of E w ith respect to I is the value of th e subject of E w ith respect to th e least in te rp re ta tio n I ’ such th a t (a) each definition in th e clause is tru e w ith respect to I ’ and (b) I’ differs from I a t m ost in th e values it assigns to variables having definitions in the clause.
Lucid inherited ISW IM ’s sy n tax and extended its sem antics. In Lucid th e value of an expression is no longer a single datu m . In stead , it is a stream of d a ta item s. Each d a tu m in th e stream corresponds to a tim e point indexed by a nonnegative integer. T he first d atu m in the stream corresponds to tim e 0. In term s of indexical program m ing, in Lucid’s indexical sem antics the context space, or th e universe of possible w orlds, consists of all tim e points, t h a t is, all nonnegative integers. T he value of an expression in a Lucid program , hence th e m eaning of th e entire program , depends on th e im plicit context - a m om ent in time.
The indexical language mLucid defined in this dissertation is a m ultidim ensional ex tension of Lucid, so th a t Lucid is co n sid e ^ d as a special case of m Lucid. T h e form al sem antics of m Lucid is defined in th e next chapter.
W hen program m ing in Lucid, program m ers th in k in term s of stream s an d filters. A sim ple Lucid program corresponds to a filter w ith n in p u t variables and one o u tp u t. T he following is an exam ple of a trivial Lucid program :
a + b w here
a = 1 .1 ; b = 3 .8 9 ; end
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 15
Note th a t th e sy n tactic object 1.1 denotes the stream or sequence < 1.1, l.'l, 1.1, • • • > . T hus the definition a = 1.1 defines th e variable a to be an infinite sequence of 1.1’s. In general, th e o u tp u t expression on the rig h t hand side can contain references to variables th a t have no definition. These free variables are th e program ’s input stream s. For example, the program
a + b
has two in p u t stream s and perform s th e pointw ise addition of the stream s. Lot the input stream s for a and b be < 2 ,4 ,6 ,8 ,1 0 ,• • • > and < 1 ,3 ,5 ,7 ,9 ,-• • > . Then the stream corresponding to a + b is < 3 ,7 ,1 1 ,1 5 ,1 9 , • • • >
Lucid provides three prim itive indexical operators for context switching: fb y , n e x t and f ir s t. T hese operators p erm it stream s to vary with the stream index.
T he two argum ents of th e binary filter fby (read as ’’followed by” ) are combined by taking th e first com ponent of the first argum ent and appending to it the stream corre sponding to th e second argum ent. S yntactically, fby is an infix operator, and thus the program
x w here
x = 1 fb y 2; end
defines x to be th e stream < 1 ,2 ,2 ,2 ,2 , - • • > . N ote th at no p art of the second argum ent is lost; its com ponents are in some sense pushed back by 1 in index ordering. Using fby an infinite stream can be defined recursively, e.g.
x w here
x = 1 fb y x + 1; e n d .
T his program defines x (the o u tp u t) to be the stream < 1 ,2 ,3 ,4 ,5 ,- • • > , an infinitely varying sequence. T h e first argum ent of fby is th e initial value of the stream based on
C H A P T E R 2, B A C K G R O U N D A N D R E L A T E D W O R K 16
which successive future values to be generated. N ote th a t since Lucid, as a functional language, is referentially tra n sp a re n t, the variable x denotes th e sam e stream everyw here in the program . T h u s, in th e definition x = 1 fb y x + 1, the x on the right hand side of the fby is the sam e as the one being defined which begins w ith 1. Since x is defined to be 1 at index 0 and 1 is defined to be 1 at index 0, x ’s value a t index 1, which is 2, can be produced according to the definition (from x+1). Obviously this argum ent can be applied ad infinitum .
U nary filter n ext removes the index 0 com ponent of its argum ent stream . In the following program , the term next x defines th e stream < 2 ,3 ,4 ,5 ,6 , • > :
n e x t x w here
x = 1 fb y x + 1 ; end
T he following program defines the Fibonacci sequence < 1 ,1 ,2 ,3 ,5 ,8 ,1 3 , • • • > :
f i b w here
f i b = 1 fb y 1 fb y f i b + n e x t f i b ; end
U nary filter fir s t simply takes th e index 0 com ponent of its argum ent stream and produces a c o n sta n t stream . In th e following program , the term first x defines the stream < 1,1,1,1, • • • > :
f i r s t x w here
x = 1 fb y x + 1 ; end
T h e following p rogram defines the factorial of an in p u t num ber n:
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 17
where
f a c = i f x < 1 t h e n 1 e l s e x * n e x t f a c f i ; x = n f b y x - 1;
end
A Lucid program is a set of equations th a t is really a tex tu al form of a directed graph. T h e nodes in the graph are the operations in th e te x t (for exam ple, + and f b y ) and arcs a re variables such as x or expressions such as x -f 1. For exam ple, th e following program
x where
x = 1 f b y x + 1 ; end
is the textual form of th e net in Figure 2.1. T h u s, all Lucid program s have a unique 1
f b y
Figure 2.1: A dataflow netw ork
representation as a netw ork of operators. [A J8 6] has term ed such dataflow networks corresponding to Lucid operators operator nets.
The evaluation of Lucid program s is based on a com putation m odel called e d u c tio n o r ta g g e d d e m a n d d r iv e n d a ta f lo w [FW 86a][AFII85]. Unlike trad itio n al dataflow [Den80][AK81], eduction can be th o ught of as a form of dataflow with tw o-w ay traffic in com m unication lines. D a ta flows in one direction from producers to consum ers in th e usual way. In the o th er direction, dem ands are sent from consum ers upstream to producers. B oth dem ands sent upstream and d a ta sent dow nstream are tagged w ith the stream index.
The com putation of a Lucid program is dem and driven in th a t it com putes the value of the program ’s o u tp u t, or the value of th e defining expression, a t the stream indices th at
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 18
th e user dem ands. T he com putation of an y p articular stream index leads to th e dem ands o f values of various program variables. T hese variables m ay be defined in term s of context switching operators th a t can cause dem ands for variables a t stream indices different from th e index associated w ith the original dem and.
2.3
R ecent D evelopm en t o f Lucid
T w o views of Lucid exist in the com puter science com m unity. T h e oldest view is th a t Lucid is a language for writing and proving properties ab o u t program s. T his view is still valid even though the language has evolved considerably since th e earliest publication [AW76a]. T h e second and more recent view is th a t Lucid is a stre a m -o rie n te d (dataflow ) language. T his view is largely based on [WA85]. This second view has been tak e n by m any as a final definition of the language. This is certainly not th e case.
From a historical perspective th e recent developm ent of Lucid can be tra c ed back to th e chapter entitled ’’beyond Lucid” in [WA85] and to [Ash85] on m assive parallelism in Lucid, b u t it differs from the original versions in these early articles. T he developm ent m ainly consists of two independent but closely related p rojects a t "University of V ictoria [DW88][DW90][Du91][Wad91b], some of whose recent results c o n stitu te this dissertation, a n d at A rizona State University an d SRI in te rn atio n a l [AF89][AJ89][Ash90][Ash91][FJ91] [JF91J.
T he early and recent work b o th concentrate on m ultidim ensional extensions of Lucid. In original Lucid the im plicit index was sim ply tim e or ite ra tio n index. Consequently, a m ultidim ensional problem would have to b e solved using tim e and som e form of m onolithic d a ta stru c tu re such as list or string [WA85]. M ultidim ensional problem s can be expressed directly in m ultidim ensional extensions of Lucid w ithout th e need for m onolithic d a ta structures. T his has im p o rta n t im plications for parallel and d istrib u ted im plem entations as described in this dissertation.
Originally, m ultidim ensional Lucid o b jects were defined and m an ip u la te d using the operators initial, rest and cby [WA85][Ash85]. T his was elegant since it com plem ented
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 19
the first, next and fby of stream oriented Lucid. Moreover, the proposed operators were inform ation preserving. In other w ords, a m ultidim ensional object could be defined and m anipulated w ith o u t any of the com ponents being lost.
U nfortunately, using this approach to m ultidim ensionality, program m ers found diffi culty to w rite program s, as they had to think of m ultidim ensional object monolithically. T h e op erato rs preserved inform ation by m anipulating d a ta in a nested hyperspace. For exam ple, th e first argum ent of the op erato r cby m ight itself be a m ultidim ensional object th a t would be “tw isted” by the o p e ra to r into a higher dimension to preserve it. T he op e ra to r initial w ould th en perform an 'untw isting” operation to ex tract the d a ta into its original “shape” .
T h e work described in this dissertation and th a t in [FJ91] are based on a distributed view of m ultidim ensionality. In this view, a m ultidim ensional space is composed of or thogonal dim ensions, and each dimension has associated w ith its operators for defining d a ta on a p e r dim ension basis.
Unlike m Lucid (described in C h a p te r 3), which uses an absolute approach to extendin ; Lucid, th e language described in [FJ91] (called Indexical Lucid) extends the m ultidim en sionality of Lucid using a relative approach. T heir approach is to allow the user to extend th e dim ensions explicitly by giving ap p ro p riate names to the new dimensions as they are needed. T h e language extends L ucid’s where clause to so-called indexed where clause. A n indexed where clause is one th a t contains declarations of indices for new dimensions th ro u g h us<; of an index declaration (or dimension declaration). T he scope of the indices in troduced by th e index declaration is the sam e as the scope of the variables defined in an ordinary where clause. For exam ple the following indexed where clause introduces two new indices or dim ensions height and width.
a + b where
i n d e x h e i g h t , w i d t h ; a = . . .
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 20
end
In the above program , the variables a and b m ay be defined to vary over two new dim en sions height and width, in addition to tim e dimension th a t is always implicit in Indexical Lucid.
All of th e tem poral operators in Lucid are extended in Indexical Lucid to any u s e r- declared dim ension h as follows:
Conventional Conventional
Extended
Lucid
Lucid Operators Indexical Lucid
Operators
expanded
Operators for Dimension h
first first.tim e first.h n e x t next.tim e next.h
fby fby.time fby.h
As in m L ucid, each of the above extended Lucid operators works on its p a rtic u la r dim en sion in th e sam e w ay th at th e ordinary Lucid operators worked on th e tim e dim ension.
In Indexical Lucid, index variables th a t returns the indices of corresponding dim ensions a t a point in the m ultidim ensional space have the sam e nam e as those used for dim ension nam es. All index variables vary over the n a tu ra l num bers. T h e following is an exam ple of sim ple Indexical Lucid th a t defines a unit m atrix:
where
i nd e x i
,
j ;u n i t M a t r i x = i f i eq j t h e n 1 e l s e 0 f i ; e n d.
2.4
S y sto lic Arrays
An application of th e indexical language mLucid for parallel program m ing is to specify systolic a rra y s. Systolic arrays, initially proposed by H .T .K ung [KL78][Kun82], are a
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 21
kind of high-perform ance, special-purpose com puter system s for solving com puting-bound problem s in m any areas, such as signal/im age processing and scientific com puting. Systolic arrays are becom ing increasingly a ttra c tiv e because of th e continuous advances in VLSI technology.
A systolic array consists of a netw ork of processing elem ents (or P E s) connected to gether by a regular interconnection w ith d a ta stream s flowing in to and o ut of the array, much like th e flow of blood to and from th e h eart. D a ta stream s flow into the array through PE s a t th e boundaries of th e netw ork. Each P E in th e array processes every d atu m in one or m ore stream s and passes th e result to its neighboring PEs. T h e final results produced by th e array either stay in th e local memory of the PE s or, as d a ta stream s, flow o ut of th e network through th e bo u n d ary PEs. T h e com putation in the array is synchronized; the d a ta is rhythm ically com puted (tim ed by a global clock) and passed through th e netw ork.
F unctionalities of P E s and the interconnection topology of a systolic array are specified by an algorithm , called a systolic algorithm, which the th e systolic array design is based on. In this dissertatio n , we only consider systolic arrays a t the level of their h ardw are- independent co m p u tatio n and com m unication behaviors. In this sense, we use the term s,
systolic arrays a n d systolic algorithms, synonym ously in late r discussions.
Systolic array s have the following essential characteristics. P E s in a systolic array have (1) functional m odularity in term s of th a t they usually perform homogeneous com pu tatio n s, a n d (2) regular com m unication in term s of th a t they are usually connected via some regular netw ork topology, such as one-dim ensional linear array, ring, mesh, tw o- dim ensional hexagonal array, tria n g u la r array, torus or binary tree. T here is no global storage in th e array; each P E has access to its local storage. P E s can com m unicate through the netw ork w ith some sm all num ber of neighboring P E s via message passing.
For exam ple, Figure 2.2 shows a systolic array for com puting m atrix m ultiplication. T he systolic algorithm th a t th e array is based on is described as follows.
Algorithm :
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 22
a33 a32 a31 a30 0 0 0
a23 a22 a21 a20 0 0
al3 a l2 a ll alO 0
a03 a02 aOl aOO
bOO 0 0 0 blO bOl 0 0 b20 b ll b02 0 b30 b21 b 12 b03 b31 b22 bl3 b32 b23 b33
Figure 2.2: A m atrix m ultiplication systolic array
Output
: T h e product C = A X B of th e in p u t m atrices A a n d B. T he o u tp u t is found in the array when the com putation term inates; each P E a t position (i,j) holds an elem ent C i j of the product.Method
: E ach Cij a t PE{j is initially0.
A t each tim e step, P E { j does th e following:1. in p u ts two d a ta item s from A { j-1 a t th e P E to its left and jB,_i j from the P E below, or from the in p u t m atrices, respectively;
2. m ultiplies the two in p u t d a ta item s a n d adds the result to th e value stored in
C i j;
3. passes th e in p u t d a ta item s to the P E s to its rig h t a n d above along th e original dim ensions in which they cam e, so th a t th ey can be in p u ts for the neighboring P E s at the next tim e step;
4. for each C i j , after n + i + j — 2 tim e steps th e result Cij is held by P E j . □
One of problem s in systolic array designs concerns th e specification and verification of systolic arrays. For purposes of im plem entation and proof, rigorous n o tatio n other
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 23
th a n inform al pictures or “snapshot” for specifying systolic arrays is desirable. T he search for program m ing languages th a t p e rm it complete specifications of the com putation and com m unication activities on the systolic arrays represents a very challenging field of pro gram m ing language research [Kun82][Kun87]. This dissertation takes up this challenge.
2.5
T h e Functional L anguage C rystal
In term s of expressing systolic algorithm s, some features of the indexical language mLucid defined in this dissertation are sim ilar to the functional language C rystal [Che8 6c][CheS6a] [Che8 6b].
In C rystal, a program consists of a system of recursion equations sim ilar to other functional languages, but it is in te rp re te d differently from the traditional functional in ter p reta tio n . A C rystal program has th e following general form:
Fi = M Fi ( Tn ( v ) ) , F 2(Ti2( v ) ) , - - - , F n(Tin(v))), F2 = <j>2(F l( r2l ( v ) ) , F 2(T2 2( v ) ) , - - - , F n(T2n(v))),
Fn = <j>n(Fl(Tnl ( v ) ) , F 2(Tn2( v ) ) , - - - , F n(Tnn(v))).
where
• v — [ui, v2, • • •, un] € A, w ith A a C artesian product A \ x A 2 X • • • X A n where eacli A; is a subset of the set of integers. T h e set A, considered as a discrete stru c tu re , is sim ilar to th e context space in m Lucid’s indexical sem antics.
• Functions r ; j , called com m unication functions, m ap from A to A. They are sim ilar to indexical operators in m Lucid.
• F unctional variables F \ , F2, • • •, F n , range over continuous functions from (A, C ) to (jD ,C ), where D is some value dom ain and (A, C ) and ( D , C ) are flat dom ains. Fi is called a multiple data stream and is sim ilar to a variable in mLucid whose value depends on a context - an elem ent in A.
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 24
• Functions • • • A m are continuous functions over the value dom ain ( D C ). <£,• called processing function is sim ilar to a pointw ise function in m Lucid.
T hus the above system of recursion equations is a system of lixed-point equations, and on its right hand side,
A(F\,
F2,• • •,
Fn) . \ v . [ M '• 0.
W '••) , ••• ,
M- • •)]
is a continuous function from E n to E n , where E = [(A, C) —► (£>,□)], th e dom ain of continuous functions from (/I, C ) to (D , C).
For exam ple, the following is a C rystal program for integer partitio n . T h e nu m b er of p artitio n s of an integer k into integers less than or equal to m can be obtained by applying the following recursively defined function C to th e pair of integers(m ,k):
= 1 - ^ 1 i > 1 i > j C ( i - l , j ) i = j -+■ C ( i - 1, j ) + 1 i < j -*■ C ( i - 1 J ) + C ( i , j - i) where
• v € A = N 2, where N is th e set of positive integers.
• D epending on v, r ( i , j ) — (i - 1 , j ) or (i, j - 1).
• T h e functional variable C is a continuous function from ( N 2, C) to (JV, C).
• D epending on u, the function <j> is th e addition or id en tity function.
In C rystal, an operational in te rp re ta tio n is given to the equations, a n d parallelism sim ilar to context parallelism in m Lucid is introduced. Instead of being considered as actual p aram eters of a function application like C(i,j), th e pair (i,j) is considered as an index p a ir for a process in an ensem ble of parallel processes. E ach pair (i,j) in th e set
A corresponds to a process, a n d A is called a process stru c tu re . The process s tru c tu re
C H A P T E R 2. B A C K G R O U N D A N D R E L A T E D W O R K 25
of nodes, where each node corresponds to an index pair (i,j) in the set A. Each directed edge comes o ut of a node whose index pair appears on the right hand side of an equation in the system , and goes into a node whose index pair appears on the left hand side of the sam e equation. T h e directed edges of th e DAG define the d a ta dependency relation of the algorithm expressed by the program . T h e dependency relation (u -< v) betw een two nodes
u and v in the DAG holds, if there is a directed edge from u to v. T h e transitive, closure
of the relation on all such pairs is a p artial order. T he com putation of a C rystal program s ta rts from executions of the processes corresponding to the source nodes of the partial order, and is followed by other processes after their predecessors have been executed.
A m ajor difference betw een C rystal and m Lucid is their m athem atical sem antics. T he sem antics of C ry stal is e x te n s io n a l, while th a t of mLucid is in te n s io n a l. In C rystal, since indices of dimensions appear explicitly as param eters in function definitions, func tions have extensionality, th a t is, th e value of a function application corresponds to an explicit context. In other words, a function application with p articu lar indexical actual param eters is considered as an individual o b ject, whose value is a d a tu m in th e given d a ta dom ain. By c o n tra st, in m Lucid, since context is im plicit, functions have intensionality, th a t is, the value of a function application as an intension depends on the im plicit con tex t w ithout referring to explicit indices. T h e sem antic difference of the tw o languages results in different program m ing styles. In C rystal program s, indices are explicitly and directly m an ipulated; context sw itching is expressed by changing indices. T h is is conve nient when com m unication p a tte rn s in the program s are irregular. In m Lucid program s, indices are im plicitly and indirectly m anipulated by indexical operators; co n tex t sw itch ing is expressed by applications of indexical operators. W hen com m unication pattern s in the program s are regular, in term s of parallel processing, the intensional approach re flects clearer global structures of th e program s th an the extensional one. Also, a C rystal program m ust be rew ritten if it is to be used on objects w ith different dim ensionalities th an the targeted objects. In co n trast, an mLucid program can be used on objects with various dim ensionalities w ithout changing the program , because different dim ensionalities of the objects only affect the program ’s im plicit context b u t not th e program ’s tex t itself.
C H A P T E R 2, B A C K G R O U N D A N D R E L A T E D W O R K 26
Furtherm ore, using indexical operators is more expressive pow er in term s of specifying com m unications in parallel program s th an explicitly m anipulating indices. T his is shown in the indexical parallel functional program m ing technique described in C h a p te r 9.
2.6
Parallel F unctional P rogram m ing
T h e indexical parallel functional program m ing described in C h ap ter 9 is inspired by th e parallel functional program m ing paradigm proposed by [HS86b][Hud8 6a][Hud8 6b] [Ilud88][IIud89].
One advantage of functional languages is th a t parallelism in a functional program is im plicit, which is called expression parallelism; it is m anifested solely th ro u g h d a ta depen dencies and the sem antics of prim itive operators. Expression parallelism in a functional language has two forms [G0I88]:
• Horizontal parallelism occurs when two or m ore argum ents of a function application are evaluated in parallel.
• Vertical parallelism occurs when an argum ent is evaluated in parallel w ith th e func tion application to which, it is being passed.
In most im plem entations of functional languages, parallelism in functional program s is detected by compilers and processes are assigned to parallel processors autom atically. A lthough in certain constrained classes of functional languages like C ry stal th e m apping of processes to processors can be determ ined optim ally, in th e general case th e optim al strategy is undecidable. B ut w hat if a program m er knows a good m apping stra te g y for a program executing on a particu lar m achine, b u t th e com piler is n o t sm a rt enough to determ ine it? To m eet this need, some researchers have designed extensions of functional languages, called parafunctional programming /anguages[HS8 6b][Hud8 6a][Hud8 6b]. T he extensions am ount to a m etalanguage (e.g. an n o tations) for program m ers to express control inform ation on process-to-processor m appings..
T he fundam ental idea behind process-to-processor m apping is as follows. Consider th e expression e\ + e^. T he strict sem antics of + allows th e subexpressions e \ and e2 to
