Journal of Information
`Science
`
`http://jis.sagepub.com/
`
`
`A clustering method using the strength of citation
`Tatsuki Saito
`Journal of Information Science
` 1990 16: 175
`DOI: 10.1177/016555159001600305
`
`The online version of this article can be found at:
`
` http://jis.sagepub.com/content/16/3/175
`
`Published by:
`
`
`http://www.sagepublications.com
`
`On behalf of:
`
`
`Chartered Institute of Library and Information Professionals
`
`
`
`Additional services and information for can be found at:Journal of Information Science
`
`  
`    
`
`
`
`Email Alerts:
`
`
`http://jis.sagepub.com/cgi/alerts
`
`
`
`http://jis.sagepub.com/subscriptionsSubscriptions:
`
`
`
`Reprints:
`
`
`http://www.sagepub.com/journalsReprints.nav
`
`
`
`http://www.sagepub.com/journalsPermissions.navPermissions:
`
`
`
`Citations:
`
`http://jis.sagepub.com/content/16/3/175.refs.html
`
`
`
`>>
`
`Version of Record
`
`- Jan 1, 1990
`
`What is This?
`
`
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`001
`
`Facebook Inc. Ex. 1212
`
` 
`  
` 
` 
` 
` 
` 
` 
`

`

`A clustering method using the strength
`of citation
`
`175
`
`Tatsuki Saito
`Department of Precision Engineering, Faculty of Engineering,
`Hokkaido University, North 13 West 8, Kita-ku, Sapporo, 060
`Japan
`
`Received 20 September 1989
`Revised 11 December 1989
`
`A new method for modelling and clustering a relational
`graph produced from the citation relation among scientific
`articles is discussed. One article corresponds to a point and a
`citation link corresponds to a directed walk in the graph. This
`graph is a direct-citation graph and a total-citation graph is
`derived from it. There exist two types of directed citation
`graph; i.e., citing directed-graph and cited directed-graph. The
`former is considered in this paper. These graphs are repre-
`sented in the form of similarity matnces which are asymmetric.
`The characteristics of these graphs are analyzed by clustering.
`For this study, a research database was designed and produced
`to acquire bibliographic information and obtain relations be-
`tween articles in it. A modelling methodology of its relational
`structure is described in this article. Combinatonal clustering
`methods have been examined and a clustering method for an
`asymmetric similarity matrix is also proposed.
`
`1
`
`Introduction
`
`Various kinds of important information exist in
`scientific articles. Relations between articles are
`useful for the study of the properties of research.
`While marketing bibliographic databases bring us
`bibliographic information comprehensively, they
`sometimes contain inaccurate information [ 1 ]. The
`research database named &dquo;ANGEL&dquo; used to pro-
`cess detailed information, especially on citation
`relations, has been produced for the reason that
`direct acquisition from original articles is nieces-
`sary [2,3]. There have been some studies on how to
`develop a methodology enabling improved re-
`trieval efficiency by using citation relations [2-15]
`and to evaluate journals or evaluate the dynamic
`influence of scientific activity [2-6,16-21]. The
`aim of this study is not only to do this, but also to
`
`develop a method to analyze the structure of rela-
`tions between articles and clarify the properties of
`the research. A methodology for modelling the
`relational structure among scientific articles is
`treated and clustering methods, which include a
`present technique, are examined in this article.
`Ordinarily, the graph used for modelling is a non-
`directed graph by reason of its ease of processing
`[22.23]. However, scientific article information has
`directionality in the form of the relation between a
`citing article and a cited article. There are two
`types of citing directed-graph and cited directed-
`graph. Though the former is described in this
`article, when a direct-citation relation matrix is
`generated, it is necessary to exchange &dquo;cite&dquo; for
`&dquo;cited by&dquo; if a cited relation graph is required. We
`discuss representation by a directed-valued graph
`which constructs a more precise model in order to
`deal with the strength of the relation. That is, this
`is a discussion about the validity of a method to
`represent the relations among scientific articles by
`asymmetric similarity matrices. Cluster analyses
`have been applied because the processing model is
`represented in the form of a similarity matrix in
`its final form. In cases where a processing object is
`expressed by a symmetric matrix, combinatorial
`cluster analyses are suitable [24-30]. However, in
`the case of an object expressed by an asymmetric
`matrix, it is necessary to transform the matrix to a
`symmetric matrix. Then the problem of distortion
`of the original structure appears. This fact has
`been confirmed by our experiment. A new cluster
`analysis method which makes it possible to em-
`ploy an asymmetric similarity matrix is proposed
`in order to avoid such distortion.
`
`2 Representation of citation relation
`
`2.1 I Representation of relational strength lp’ a i>al-
`ued gruph
`
`The graph is made up of the non-empty finite
`set P that has p points, and the specified set L
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`002
`
`Facebook Inc. Ex. 1212
`
`

`

`176
`
`that has unordered q pairs belonging to P. Pair
`I = ( i, j ) of point i and points belongs to L, and
`is called a line of the graph. A graph that has p
`points and g lines is called a ( p, q) graph or
`graph G = (P ; L). It can be used to model struc-
`tural objects; that is, the graph is a relational
`graph in which a point corresponds to an objective
`article and a line corresponds to a relation be-
`tween two articles. There are two types of graphs,
`non-directed and directed. A non-directed graph
`is thought to be a special directed graph that
`always accompanies a directed line of reverse di-
`rection. In a directed graph, the finite point-set P,
`which is not empty, and the specified set L, which
`has the ordered pairs of two different points, are
`dealt with simultaneously. In the author relation
`graph, articles by the same author are connected
`by lines. Though the author relation has no direc-
`tion, the citation relation has direction. Therefore,
`the citation relation is represented by a directed
`graph. While a directed graph can express the
`presence or absence of a relation between articles,
`it cannot express the strength of the relation.
`Consequently, a valued graph is introduced to
`enable the expression of the strength of the rela-
`tion. A valued graph ( P ; L ; Y) is a graph in which
`a line of set L is accompanied by the value r,
`where r is mapped onto the real number set Y. To
`express r( I, .1), r is called a value of a line. The
`production of the valued graph and its relational
`similarity matrix are described in Section 2.2.
`While the value is 0 or 1 in a direct-citation
`relation graph, it assumes various values in a
`total-citation relation graph as shown in the next
`procedure.
`
`2.2 Modellitia procedure
`
`The modelling is processed as follows. To begin
`with, considering the binary citation relation be-
`tween article i and article j, it is represented as a
`relational graph which is expressed by point i,
`point j and line ( r, ~ ). Then, after generating a
`directed graph that corresponds to citation rela-
`tions between articles, the graph is represented as
`a similarity matrix. The generation of a direct-ci-
`tation relation graph and the production of its
`similarity matrix are described below.
`
`Direct-CitatIOn relatioll f1l{J[r/X
`The direct-citation relation matrix is defined as
`
`A = [ll’J]’ This is a so-called adjacency matrix,
`where
`
`a = 1 ; when article i cites article j.
`= 0; otherwise.
`
`If a cited relation graph is needed, it is neces-
`sary to exchange &dquo;cites&dquo; for &dquo;is cited by.&dquo;
`
`Total-citation relation rnatri.a
`Considering the total-citation relation that in-
`volves indirect citation, the total-citation relation
`matrix is defined as S = [s,,]. where
`
`Here, k implies the length of a directed walk, and
`/., a’l means the number of directed walks whose
`length between article i and article is k. The
`upper boundary n does not exceed max( k ) (the
`maximum length of the directed walk), and >5,~
`is a
`weight that takes a value such as 1/k or 1/k2.
`The h a II can be obtained by the k th power of
`matrix A by resetting the diagonal elements to 0.
`An efficient algorithm that calculates only the
`non-zero elements of matrix A has been devel-
`oped.
`
`3 Cluster analysis of the similarity matrix
`
`3.l I
`
`CIWrl1ctenstlcs nf the (-itatioti relatlOll mutrru
`
`Methods of clustering that analyze the rela-
`tional graph represented by the similarity matrix
`are discussed here. Hierarchical techniques of the
`combinatorial cluster analysis are discussed in
`Section 3.2 and the present method is described in
`Section 3.3. The matrix to be analyzed is given by
`
`,
`
`where p is an enhancing exponent.
`This matrix has the following characteristics.
`First, each element of the matrix has a positive
`real quantity. Element s&dquo; is similarity as a corre-
`lation-like measure and element /- has a positive
`real 4quantity. Second, the matrix is asymmetric.
`The similarity matrix of the citation relation is
`asymmetric because there is a time sequence in the
`citation relation between a citing article and a
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`003
`
`Facebook Inc. Ex. 1212
`
`

`

`cited article. In consequence, matrix R becomes
`asymmetric. When the methods of the combina-
`torial cluster analyses discussed in Section 3.2 are
`applied, it is necessary be change it to a symmetric
`matrix. However, this is not desirable due to the
`occurrence of the distortion of the original matrix.
`We propose a new clustering method for the
`asymmetric matrix in Section 3.3.
`
`3.2 Application of combinatorial clustering meth-
`ods
`
`Combinatorial cluster analysis is also called the
`method of hierarchical cluster analysis for the
`reason that it constructs a tree. All computer
`programs using the following combinatorial meth-
`ods have been implemented and applied for the
`analysis of the relational graph model. It should
`be noted that the original similarity matrix of the
`relational graph model is asymmetric but that it
`must be changed to a symmetric matrix by the
`method described below because the combina-
`torial methods are applicable only to symmetric
`matrices. Seven hierarchical clustering methods
`have been examined. The nearest neighbour
`method is known as single linkage because clusters
`are joined at each stage by the single shortest or
`strongest link between individuals. The furthest
`neighbour method is also called the complete-link-
`age method because all individuals in a cluster are
`linked to each other by some max-min similarity.
`The median method adopts the middle value of
`the nearest neighbour value and the furthest
`neighbour value. The method of average linkage
`within the new group is not influenced by extreme
`values for defining clusters so it cannot make any
`statements about the minimum or maximum simi-
`larity within a cluster. Average linkage between a
`merged group, also called the group-average
`method, evaluates the potential merger of clusters
`i and j in terms of the average similarity between
`the two clusters. The difference between the latter
`and the former is whether the sums of within-group
`similarities are ignored or not. The centroid
`method uses both the mean value of similarities
`and the number of individuals for the merger. The
`minimum variance method (the Ward method),
`used to find at each stage those two clusters whose
`merger gives the minimum increase in the total
`within-group error-sum of squares, is generally
`reasonable even if it is not optimal.
`
`177
`
`When these combinatorial methods were ap-
`plied, the asymmetric similarity matrix R was
`changed to a symmetric matrix by r, ~ =
`(max(s,,, (i >.l ).
`
`3.3 Cluster analv.ci.c method for the asymmetric
`matri
`
`Though combinatorial clustering methods are
`versatile, the matrix must be symmetrized when
`applied to an asymmetric similarity matrix. Con-
`sequently, the initial information space to be clus-
`tered is distorted, so that the precision of analysis
`often decreases. We propose a new method of
`clustering for the asymmetric similarity matrix. It
`is called &dquo; total-relationship cluster analysis&dquo; and
`classifies objects by the total relationship between
`articles. To use this method, let od( i ) denote the
`outdegree at article i, which is obtained by
`E~=~a,~, where t is the total number of articles.
`If combinatorial analysis of the total-citation
`relation is required, lett be as ’’,.=~,. other-
`wise, let a matrix which is represented by
`
`be introduced, where a’¡, is a weight (for example,
`wx = 1, Ilk or l/k2). The quantity LV&dquo;
`is an
`( i, j ) element which is a powered matrix [ y, ~ J k
`times and e is an enhancing exponent. The quan-
`tity ~,~ is construed as the influence of article i <>n
`article j.
`
`Total- relationship cluster atiali-sis (TRCA)
`Let value z. , designate the total relationship
`between article j and all other articles. It is intro-
`duced by
`
`To find the best u seeds through j. if the
`number of goal clusters is u, let them be denoted
`as 7<(7t ~7t- 72&dquo;-&dquo; j~, ), where u is the number of
`goal clusters. Article i is clustered into article jo’
`,~( j~ =1= j) and j, ju 6/J. The proce-
`where ~,~~ >_
`dure of total-relationship cluster analysts is as
`follows.
`Procedure I : Find the best u seeds through =. ~,
`and denote them y,( 7, = jB, 7:’ - - -. j,, ),
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`004
`
`Facebook Inc. Ex. 1212
`
`

`

`178
`
`Procedure 2: Article i is clustered into article j,
`where =’} has the maximum value on the same
`i-row in Z ( ~ =.1~,, jn Ejç)’
`One of the important properties of total-rela-
`tionship cluster analysis is that the number of goal
`clusters can be controlled and no hierarchical
`structure arises between articles, except at the core
`level.
`
`4 Exploratory result
`
`In order to test the present method, scientific
`articles for two different research fields were in-
`vestigated. One group included 231 articles con-
`cerning CAD/CAM and the other 140 articles
`addressed the two or three bodies problem in
`nuclear physics. All of them had a citation relation
`between articles in each field. The former con-
`tained mainly articles about computational geom-
`etry and several articles relevant to artificial intel-
`
`ligence (AI). In the following comparison with
`manual classification by experts, the conformance
`percentage is the separation rate between the AI-
`cluster and the proper CAD/CAM cluster. Be-
`cause its research field is obviously different from
`the research field of CAD/CAM it was hoped
`that this would suppress the occurrence of error
`due to the analyst’s subjectivity. On the other
`hand, the latter contained three groups which use
`a different approach or methodology. That is, it is
`the first group of articles using the direct solution
`(DIS), the second group of articles using the varia-
`tion solution (VAR) and the third group of articles
`focusing on the realistic interaction (REAL). The
`clustering methods used were the seven combina-
`torial methods and the present method. Computer
`programs by Anderberg were utilized for combi-
`natorial clustering [31].
`The average linkage within the new group
`method was the best among hierarchical methods.
`Other combinatorial methods are not useful be-
`
`Fig. 1. The result clustered by TRCA for 140 articled in the field of nuclear physics.
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`005
`
`Facebook Inc. Ex. 1212
`
`

`

`Table 1
`The conformance ratio with experts of nuclear physics: the
`result is clustered by average linkage within the new group
`method for 140 articles in the field of nuclear physics
`
`Table 2
`Effect of the directed-walk’s length for the citation relation of
`231 CAD/CAM articles
`
`179
`
`citation relation. Though 72.2% at k = 4 is little
`different from 7l.l~o at k = 1. its value is reliable
`because the same values are obtained at k = 4 to
`k = 13. In contrast, Table 3 shows the results for
`the 140 nuclear physics articles, where the maxi-
`mum value of 65.9% was obtained at k = 1 and
`the minimum value of 49.9% was obtained at
`k = 4 in the case of the citations. From these
`results, while citation relation covers a wide range
`it is thus necessary to consider it until directed-
`walk length four in the research field of
`CAD/CAM. In contrast, the relation in the field
`of nuclear physics is closed and accordingly it
`does not need to be considered to a distance
`greater than length two. While an optimal value
`
`Table 3
`Effect of the directed-walk’s length for the citation relation of
`140 nuclear physics articles
`
`cause these clusters never or rarely agglomerate
`until the last stage. This tendency appeared to be
`common to both processed groups. Accordingly,
`these methods are not adequate for analyzing the
`relations structure of scientific articles.
`For the 140 nuclear physics articles, the cluster-
`ing result by TRCA is shown in Fig. 1. The result
`classified by twelve physics experts is surrounded
`by the thin line, and the result by TRCA is
`surrounded by the thick line. Superscripts
`(D, V, R) at the upper right of an article’s number
`indicate the initial letters of proper clusters. In
`comparison with the human classification by
`physicists, the conformance percentage by aver-
`age-linkage within the new group of hierarchical
`clustering methods is similarly calculated and dis-
`played in each cluster for these 140 articles in
`Table 1. In order to compare TRCA with the
`interpretative structural model (ISM) [32], combi-
`natorial analyses were examined in which the en-
`hancing factor p was set to 0 in R.
`The effect of the direct walk’s length is shown
`in Tables 2 and 3. Concerning Table 2, for the 231
`CAD/CAM articles, the maximum value of 72.2%
`was obtained at over k = 4 in the case of the
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`006
`
`Facebook Inc. Ex. 1212
`
`

`

`180
`
`72.2% was obtained at greater than k = 4 in the
`case of the total-citation relation by the hierarchi-
`cal clustering method for the 231 CAD/CAM
`articles, the optimal value was at most 65.9% by
`the combinatorial method in the case of the
`direct-citation relation for the 140 nuclear physics
`articles. In consequence, a more appropriate
`method for clustering was required. This require-
`ment led to the present clustering method (TRCA)
`and the conformance percentage attained by it
`was 75.1%. It also became clear that the exponen-
`tial coefficient e in YI} = r, ~/( od( i ))‘’, should be
`between 1.0 and 3.0. Furthermore, when the
`total-citation relation was utilized, to adopt a suit-
`able weighting coefficient (Wk = 1/A&dquo;~) of combi-
`nation, it was possible to extract key articles
`(3, 40, 44) as marked by double circles in Fig. 1.
`The exploratory results can be summarized as
`follows. Because the present method had better
`results than ISM, this modelling method expressed
`the original information space more exactly. The
`present TRCA clustering method attains good
`conformance with the research experts in compari-
`son with combinatorial cluster analyses. Thus it is
`considered that the present method is more suit-
`able for cluster analysis owing to the proper clus-
`tering of the relation matrix without distortion. It
`was clarified that different citation relations should
`be considered for different research fields; i.e., a
`deep citation relation for a field like CAD/CAM
`or a shallow citation relation in a field such as the
`two or three bodies problem of nuclear physics.
`
`5 Conclusion
`
`It t has been confirmed by investigation of
`articles in different research fields that the present
`model with a relational graph is more suitable for
`the cluster analysis of scientific articles than com-
`binatorial cluster analyses or ISM. The citation
`relation should be considered at most until k = 1
`in the field of nuclear physics, but it should be
`considered until max( k ) in the CAD/CAM field.
`Therefore, we must choose the range of the cita-
`tion link according to the research field. The pre-
`sent method is also applicable for small-sized rela-
`tion networks of citations. However, it can be
`expected that this method will extend its ability
`more fully with large-sized complicated-relation
`networks. The final result does not necessarily
`
`coincide with manual classification completely, but
`we believe that this methodology provides signifi-
`cant information which it is difficult to find by
`human classification. It is considered that the pre-
`sent method of modelling and clustering is appli-
`cable for not only a scientific article-citation net-
`work but also for an engineering relation network.
`
`Acknowledgements
`
`The author would like to express his sincere
`gratitude to Professor Chooichiro Asano of
`Kyushu University for his valuable suggestions
`and continuous encouragement. Thanks are also
`owed to Professor Hajime Tanaka (Sapporo
`Gakuin University) and Professor Yoshinori
`Akaishi of Hokkaido University for their signifi-
`cant advice.
`
`References
`
`[1] H.F. Moed and M. Vriens, Possible inaccuracies occurring
`in citation analysis, Journal of Information Science 15
`(1989) 95-107.
`[2] T. Saito, S. Tejima, N. Kawai and N. Okino, Study on
`development of methodology for grasping research trend
`and efficiency of the scientific information retrieval by it.
`In: Formation Process of Information Systems and Organi-
`zation of Scientific Information &mdash; Report (Grant-in-Aid of
`the Ministry of Education of Japan, 1977) (in Japanese).
`[3] T. Saito, An intelligent input system for scientific article
`information data base system, Bulletin of the Faculty of
`Engineering. Hokkaido University 140 (in Japanese) (1988)
`101-108.
`[4] T. Saito, N. Okino, C. Asano and H. Tanaka, A modeling
`and clustering method by relational graph model and an
`exploratory application for cluster analysis of scientific
`article space, Proceedings IFAC 81 XII (1982) 62-67.
`[5] T. Saito, Cluster analysis for scientific articles by optimal
`relationship model, Bulletin of the Faculty of Engineering,
`Hokkaido University 138 (in Japanese) (1988) 16-22.
`[6] T. Saito and C. Asano, A new method of modeling and
`clustering of citation relation, Engineering Scrences Re-
`ports, Kvushu University 11 (1989) 327-337.
`[7] L.J. Cummings and D.A. Fox, Some mathematical prop-
`erties of cycling strategies using citation indexes, Informa-
`tion Storage and Retrieval 9 (1973) 713-719.
`[8] J. Bichteler and R.G. Parsons, Document retrieval by
`means of an automatic classification algonthm for cita-
`tions, Information Storage and Retrieval 10 (1974) 267-
`278.
`[9] N.L. Geller, On the citation influence methodology of
`Pinski and Narin, Information Processing and Manage-
`ment 13 (1978) 277-288.
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`007
`
`Facebook Inc. Ex. 1212
`
`

`

`[10] H. Small, The relationship of information science to the
`social sciences: a co-citation analysis, Information
`Processing and Management 17 (1981) 39-50.
`[11] T.R. Kochtanek, Bibliographic compilation using refer-
`ence and citation links, Information Processing and
`Management 18 (1) (1982) 33-39.
`[12] E. Noma, Untangling citation networks, Information
`Processing and Management 18 (2) (1982) 43-53.
`[13] C. Tomer, A statistical assessment of two measures of
`citation: the impact factor and the immediacy index,
`Information Procesing and Management 22 (2) (1986) 251-
`258.
`[14] C.D. Hurt, Conceptual citation differences in science
`technology and social science literature, Information
`Processing and Management 23 (1) (1987) 1-6.
`[15] R. Todorov and W. Glanzel, Journal citation measures&mdash;a
`concise review, Journal of Information Science 14 (1988)
`47-56.
`[16] P. Zunde, Structural models of complex information
`sources, Information Storage and Retrieval 7 (1971) 1-18.
`[17] P. Zunde and V. Slameck, Predictive models of scientific
`progress, Information Storage and Retrieval 7 (1971) 103-
`109.
`[18] G. Borenius and S. Schwarz, Remarks on the use of
`citation data in predictive models of scientific activity,
`Information Storage and Retrieval 8 (1972) 171-175.
`[19] W.D. Garvey, N. Lin and K. Tomita, Research studies on
`pattern of scientific communication: III. Information-ex-
`change processes associated with the production of jour-
`nal articles, Information Storage and Retrieval 8 (1972)
`207-221.
`[20] W.D. Garvey, N. Lin and K. Tomita, Research studies in
`scientific commumcation: IV. The continuity of dissemi-
`nation of information by "productive scientists," Informa-
`tion Storage and Retrieval 8 (1972) 265-276.
`[21] A.E. Cawkell, Evaluating scientific journals with Journal
`
`181
`
`Citation Reports&mdash;a case study in acoustics, Journal of
`the American Society of Information Science 29 (1978)
`41-46.
`[22] J.M. Culnan, An analysis of the information usage pat-
`terns of academics and practitioners in the computer
`field: a citation analysis of a national conference. Infor-
`mation Processing and Management 14 (1978) 395-404.
`[23] G. Loizou, Graph model of complex information sources,
`Information Processing and Management 15 (1979) 127-
`131.
`[24] K. Garland, An experiment in automatic hierarchical
`document classification, Information Processing and
`Management 19 (3) (1983) 113-120.
`[25] N. Jardine and C.J.V. Rijsbergen, The use of hierarchic
`clustenng in information retrieval, Information Storage
`and Retrieval 7 (1971) 217-240.
`[26] C.J.V. Rijsbergen, Further experiments with hierarchic
`clustering in document retrieval. Information Storage and
`Retrieval 10 (1974) 1-14.
`[27] H.G. Small and M.E.D. Koenig, Journal clustering using
`a bibliographic coupling method, Information Processing
`and Management 13 (1977) 277-288.
`[28] L.E. Stanfel, Applications of clustering to information
`system design. Information Processing and Management 19
`(1) (1983) 37-50.
`[29] E.M. Voorhees, Implementing agglomerative hierarchic
`clustering algorithms for use in document retrieval. Infor-
`mation Processing and Management 22 (6) (1986) 465-476.
`[30] D. Lucarella. A document retneval system based on
`nearest neighbour sercrung. Journal of Information Scrence
`14 (1988) 25-33.
`[31] M.R. Anderberg, Cluster Analysis for Applications
`(Academic Press. New York, 1973).
`&mdash; Planning, Policy and
`[32] J.N. Warfield, Societal Systems
`Complexity (Wiley, New York, 1976).
`
`Downloaded from
`
`jis.sagepub.com
`
` by guest on February 21, 2013
`
`008
`
`Facebook Inc. Ex. 1212
`
`