0022-301819211801-1723$03.00/0
`THE JOURNAL OF NERVOUS AND MENTAL DISEASE
`Copyright © 1992 by WiUiams & Wilkins
`
`Vol. 180, No. 11
`Printed in U.S.A.
`
`The Positive and Negative Syndrome Scale
`and the Brief Psychiatric Rating Scale
`
`Reliability, Comparability, and Predictive Validity
`
`MORRIS BELL, PH.D./·2 ROBERT MILSTEIN, M.D., PH.D} JOSEPH BEAM-GOULET, M.S./·2 PAUL LYSAKER, PH.D., 1•2
`AND DOMENIC CICCHETTI, PH.D.1.2
`
`In a psychiatric rehabilitation study, 154 concurrent ratings were performed using the 30-
`item Positive and Negative Syndrome Scale (PANSS) and the 18-item Brief Psychiatric Rating
`Scale (BPRS). Although both instruments had excellent interrater reliability, the PANSS was
`consistently better: on the 18 symptom items the two instruments share, the PANSS had
`higher intraclass r's on 14; for the syndromes, the PANSS was higher than the BPRS on
`positive, negative, and total. Weighted Kappas comparing shared items revealed that most
`were not interchangeable, with only three coefficients in the excellent range. However,
`syndrome scale scores were very highly correlated and resulted in similar classification for
`negative schizophrenia. Ten of the 12 items of the P ANSS not included in the BPRS had low
`zero-order correlations with BPRS items, which suggests that they measure symptoms distinct
`from those measured by the BPRS and should add to clinical predictive power. This proved
`true in our study of rehabilitation of patients with schizophrenia. PANSS symptom ratings
`explained up to 55% of the variance on seven measures of work performance, whereas the
`BPRS had lower predictive power on six of the seven measures. We concluded that the
`PANSS may be superior to the BPRS in clinical research on schizophrenia and that most
`BPRS items are not interchangeable with identically named PANSS items.
`J Nerv Ment Dis 180:723-728, 1992
`-
`
`The Positive and Negative Syndrome Scale (PANSS)
`is a rating scale designed to provide a "trustworthy"
`instrument for the study of symptom phenomenology
`in schizophrenia (Kay et al., 1987). It is composed of
`the 18-item Brief Psychiatric Rating Scale (BPRS; Over(cid:173)
`all and Gorham, 1962) and 12 additional items from the
`Psychopathology Rating Scale (Singh and Kay, 1975).
`The authors believed this expansion of the BPRS would
`capture clinical phenomena not accounted for by the
`BPRS, including several negative symptoms and anum(cid:173)
`ber of general symptoms.
`Other changes included more extensive anchor de(cid:173)
`scriptors and identification of the first two points on
`the 7-point scale (absence, minimal) as being below
`clinical significance. The BPRS break point for the pres(cid:173)
`ence of a clinical entity is between the first and second
`points (not present, very mild) of the 7-point scale.
`In a series of papers, the P ANSS demonstrated strong
`psychometric properties, including excellent interrater
`reliability (Kay et al., 1989), good internal consistency,
`
`'West Haven V.A. Medical Center, West Haven, Connecticut. Send
`reprint requests to Dr. Bell at 116-B, D.V.A. Medical Center, West
`Haven, Connecticut 06516.
`2Department of Psychiatry, Yale University School of Medicine,
`New Haven, Connecticut.
`Research was funded by the Department of Veteran Affairs, Reha(cid:173)
`bilitation, Research, and Development Service.
`
`and good to fair test-retest reliability (Kay et al., 1987).
`Validity of the PANSS has been supported in studies
`of psychopharmacological treatment (Kay and Opler,
`1985; Singh et al., 1987), cognitive functioning (Kay,
`1990), discriminant validity, and course of illness (Kay
`et al., 1987; Lindenmayer et al., 1986).
`The P ANSS authors did not, however, investigate the
`relationship between their instrument and the BPRS.
`The BPRS is widely used in psychiatric research and
`has been employed in investigations of course of posi(cid:173)
`tive and negative symptoms in schizophrenia (Kane
`et al., 1988). In considering whether to employ the
`PANSS, investigators want to know whether it will sur(cid:173)
`pass the BPRS in its power to describe schizophrenic
`phenomenology.
`A similar question was asked comparing the BPRS
`with the Scale for the Assessment of Negative Symp(cid:173)
`toms (SAN; Andreasen, 1982). The authors (Thiemann
`et al., 1987) determined that the BPRS withdrawal(cid:173)
`retardation subscale was highly correlated with the
`SANS summary score and that SANS interrater reliabili(cid:173)
`ties compared unfavorably with the BPRS interrater
`reliability. They concluded that there was little to be
`gained by using the SANS rather than the BPRS.
`Given the robustness of the BPRS, it is desirable to
`learn whether, and in what ways, the additional effort
`
`723
`
`Par Pharm., Inc.
`Exhibit 1016
`Page 001
`
`

`

`724
`
`BELL et al.
`
`involved in using the PANSS would be justified. The
`present study compares the PANSS to the BPRS by
`examining interrater reliabilities and internal consis(cid:173)
`tencies of the syndrome scales; it investigates whether
`the additional P ANSS items capture clinical phenom(cid:173)
`ena distinct from the BPRS items, and, if so, whether
`the PANSS has greater power to predict measures of
`work performance in a psychosocial treatment study.
`In addition, Kay3 suggested that we examine the com(cid:173)
`parability of the 18 items shared by the PANSS and the
`BPRS. If the modifications the PANSS authors have
`made in descriptors for the BPRS symptom items of
`the same name did not substantially alter their ratings,
`then it might be possible to convert BPRS scores into
`PANSS ratings. This would make the vast psychiatric
`literature using the BPRS available for comparison with
`P ANSS studies.
`
`Method
`
`Subjects
`
`A total of 154 concurrent symptom ratings were ob(cid:173)
`tained from 56 subjects with DSM-III-R diagnoses of
`schizophrenia or schizoaffective disorder. Assessments
`of work performance were obtained from a subset of
`30 of the initial 56 subjects who accepted work place(cid:173)
`ments. Subjects were recruited from the Psychiatry Ser(cid:173)
`vice of a Department of Veterans Affairs Medical Center
`for a study of the rehabilitative effects of productive
`activity on schizophrenia. Subject demographics and
`mean subscale values are presented in Table 1.
`
`Instruments
`
`Positive and Negative Syndrome Scale. The PANSS
`is a 30-item rating scale, completed by clinically trained
`research staff at the conclusion of chart review and a
`semistructured interview. The PANSS ite~s are
`grouped into three rationally derived categories: posi(cid:173)
`tive symptoms, or symptoms that should not be present
`in a normal mental status (e.g., hallucinations and delu(cid:173)
`sions); negative symptoms, symptoms that represent
`behaviors that should be present in a normal mental
`status but are not (e.g., blunted affect and emotional
`withdrawal); and general symptoms (e.g., anxiety and
`depression). The list of PANSS items is found in Table
`2. Scores for positive, negative, and general scales are
`obtained by summing scores within each category. An
`overall total is obtained by summing all items.
`The Brief Psychiatric Rating Scale. The BPRS (Over(cid:173)
`all and Gorham, 1962) is an 18-item rating scale filled
`out by clinically trained research staff. The BPRS is
`completed after chart review and a semistructured in(cid:173)
`terview. Kane et al. (1988) have grouped the items of
`
`3S. R. Kay (personal communication, 1989).
`
`TABLE 1
`Demographics and Mean Subscale Values
`Group Frequency
`Reliability
`(N = 56)
`
`Predictive
`(N = 30)
`
`Gender
`Female
`Male
`Race
`Black
`Hispanic
`Caucasian
`Primary Diagnosis
`Disorganized
`Paranoid
`Residual
`Schizoaffective
`Undifferentiated
`Course of Illness
`Episodic
`Undetermined
`Continuous
`
`Age
`Slosson IQ score
`Years of education
`Age at first hospitalization
`Total no. (lifetime) of
`hospitalizations
`Duration of illness (yrs)
`No. of mos. in hospital, last
`3 yrs
`Years of work
`BPRS negative
`BPRS positive
`BPRS general
`BPRS total
`PANSS negative
`PANSS positive
`PANSS general
`PANSS total
`GAF
`X± SD.
`
`4
`52
`
`6
`2
`48
`
`2
`30
`1
`16
`7
`
`11
`8
`37
`40.2 ± 8.6
`101.8 ± 14.8
`12.5 ± 2.1
`23.3 ± 5.7
`
`10.0 ± 7.8
`16.9 ± 7.8
`
`7.1 ± 6.8
`3.6 ± 3.8
`4.3 ± 3.1
`12.3 ± 6.2
`8.4 ± 3.6
`25.1 :!: 8.8
`17.3 :!: 4.5
`19.0:!: 5.9
`38.3 ± 7.1
`74.7 ± 13.2
`35.6 + 5.7
`
`2
`28
`
`2
`1
`27
`
`2
`16
`0
`7
`5
`
`4
`4
`22
`
`38.0 ± 8.4
`103.4 ± 12.9
`12.4 ± 2.3
`23.6 ± 6.3
`
`10.8 ± 9.1
`15.2 ± 9.2
`
`7.8 ± 7.2
`3.5 ± 4.2
`4.4 ± 3.4
`12.3 ± 5.9
`8.3 ± 3.6
`25.1 ± 8.9
`17.6:!: 4.7
`19.2 :!: 5.5
`39.1 ± 7.2
`75.9 ± 12.6
`35.6 + 4.2
`
`the BPRS into categories paralleling those in the
`PANSS: positive, negative, and general; they are pre(cid:173)
`sented in Table 2.
`Work Personality Profile. The Work Personality Pro(cid:173)
`file (WPP; Bolton and Roessler, 1986) is a 58-item work
`performance inventory completed after direct observa(cid:173)
`tion of a subject's work behavior and supervisor inter(cid:173)
`view. From these ratings, five factor scores are gener(cid:173)
`ated: work skills, social skills, work motivation, self(cid:173)
`control and judgment, and personal presentation.
`These scores are compared to norms from a rehabilita(cid:173)
`tion population.
`Minnesota Satisfactoriness Scale. The Minnesota
`Satisfactoriness Scale (MSS; Gibson et al., 1970) is a
`work performance inventory rated by supervisors. Four
`scores are derived: work performance, work confor-
`
`Par Pharm., Inc.
`Exhibit 1016
`Page 002
`
`

`

`Delusions
`Conceptual
`disorganization
`Hallucinations
`Excitement
`Grandiosity
`Suspiciousness
`Hostility
`Positive total
`
`.86
`.93
`.81
`.92
`.88
`.82
`.93
`
`PANSS/BPRS
`
`725
`
`.82
`
`.75
`.92
`.85
`.86
`.84
`.69
`
`.52
`.87
`
`.71
`
`.52
`.61
`.74
`.87
`.90
`
`.84
`.70
`.82
`.67
`.86
`.89
`
`.87
`
`agnoses were determined using the Structured Clinical
`Interview for DSM-III-R (Spitzer et al., 1989).
`Mter the PANSS and BPRS interviews, subjects were
`offered job placement such as work in the hospital
`escort service, pharmacy, or blind center. Subjects'
`work performance was assessed weekly using the WPP
`and MSS.
`Analysis
`Agreement between scores of similarly named items
`on the PANSS and BPRS was calculated using Kappa
`(Cohen, 1968; Fleiss, 1981), by applying a weighing sys(cid:173)
`tem recommended by Cicchetti (1976) and Cicchetti
`and Fleiss (1977). Concurrent symptom ratings were
`compared and resulted in 154 pairs of observations.
`Interrater reliabilities within scales were assessed
`using the intraclass correlation coefficient (Bartko,
`1966). Calculations were performed using a computer(cid:173)
`ized scoring routine (Cicchetti and Showalter, 1988)
`and implementing an algorithm that takes into account
`both the number and specific set of judges that vary
`from subject to subject (i.e., of three raters, all three
`or any combination of two). There were 55 ratings.
`Cronbach's alpha (Anastasi, 1988) was calculated
`item-by-item and overall for the three subscales of the
`BPRS and PANSS.
`Symptom ratings from the initial interview were used
`to predict the first-week work performance measures
`in a series of multiple regression procedures (optimal
`subsets). First, symptom items were used to predict
`each of seven measures of work performance. The com(cid:173)
`bination of symptoms with the highest R2 were retained .
`The adjusted R2's in Table 4 are the values found using
`the best combination off our symptom variables. Differ(cid:173)
`ent combinations of symptoms predict each perform(cid:173)
`ance measure. For example, the four PANSS negative
`symptoms best predicting work skills resulted in an
`adjusted R2 of .35.
`
`Results
`The P ANSS and BPRS interrater reliabilities are pre(cid:173)
`sented in Table 2. Both instruments had high interrater
`reliability for each of the syndrome scales and the
`symptom total, although the PANSS reliability coeffi(cid:173)
`cients were higher on 14 of the 18 items the instruments
`share. Uncooperativeness and poor attention on the
`P ANSS and emotional withdrawal and mannerisms on
`the BPRS were the only items below a good (.60 to . 74
`[Fleiss, 1981]) range of interrater reliability.
`Internal consistency measures on PANSS positive
`and negative syndrome scales (alpha = .74, .69) were
`similar to those of BPRS syndrome scales (alpha = .69,
`.68) although the PANSS general scale was consider(cid:173)
`ably more homogenous (alpha = .64) than the BPRS
`general scale (alpha = .46).
`
`TABLE 2
`Interrater Reliabilities for the Positive and Negative Syndrome
`Scale and the Brief Psychiatric Rating Scale, for Three Raters
`Intraclass r
`Intraclass r
`PANSS Score
`BPRS Score
`Positive Scales
`.93
`Unusual thoughts
`Conceptual
`disorganization
`Hallucinations
`Excitement
`Grandiosity
`Suspiciousness
`Hostility
`Mannerisms and
`posturing
`Positive total
`Negative Scales
`.88
`Blunted affect
`.84
`Emotional
`. 72
`withdrawal
`. 79
`Uncooperativeness
`Motor retardation
`Disorientation
`Negative total
`
`.90
`.80
`.63
`.94
`General Scales
`.92
`Somatic concern
`.77
`Anxiety
`.84
`Guilt feelings
`.62
`Tension
`Depressive mood
`General total
`
`.84
`.80
`.80
`.57
`. 82
`.76
`.54
`
`Blunted affect
`Emotional withdrawal
`Poor rapport
`Passive withdrawal
`Difficulty in abstract
`thinking
`Lack of spontaneity
`Stereotyped thinking
`Negative Total
`
`Somatic concern
`Anxiety
`Guilt feelings
`Tension
`Mannerisms and
`posturing
`Depression
`Motor retardation
`Uncooperativeness
`Unusual thoughts
`Disorientation
`Poor attention
`Lack of insight and
`judgment
`Disturbance of volition
`Impulse control
`Preoccupation
`Active social avoidance
`General total
`
`PANSS total
`
`.74
`.73
`.64
`.65
`.74
`.84
`Overall Scale Totals
`BPRS total
`.91
`
`mance, dependability, and personal adjustment. Scores
`are compared with norms for general workers.
`Procedures
`Subjects were concurrently rated on the PANSS and
`BPRS as part of an initial screening battery or a weekly
`interview for a study on rehabilitation. Raters were
`a Ph.D. clinical psychologist and four master's level
`assistants. All had used PANSS training tapes prepared
`by Kay and participated in practice interviews before
`their ratings were included. Before the interviews, dem(cid:173)
`ographic information was collected. The DSM-III-R di-
`
`Par Pharm., Inc.
`Exhibit 1016
`Page 003
`
`

`

`726
`
`BELL et al.
`
`TABLE 3
`Pearson Correlation Coefficients and Weighted Kappa Coefficients (K)for Items Shared by the PANSS and BPRS
`r
`BPRS Item
`
`PANSS Item
`
`K"'
`
`.72
`.88
`Unusual Thoughts
`Delusions
`.86
`.93
`Hallucinations
`Hallucinations
`.65
`.84
`Conceptual disorganization
`Conceptual disorganization
`.70
`Excitement
`.56
`Excitement
`.94
`.82
`Grandiosity
`Grandiosity
`.53
`.72
`Hostility
`Hostility
`Suspiciousness
`.70
`.86
`Suspiciousness
`.89
`.86
`Blunted affect
`Blunted affect
`Emotional withdrawal
`Emotional withdrawal
`.43
`.24
`.24
`.47
`Emotional withdrawal
`Passive withdrawal
`.71
`.84
`Somatic concern
`Somatic concern
`Anxiety
`.54
`.57
`Anxiety
`.51
`.75
`Tension
`Tension
`Guilt feelings
`.72
`.89
`Guilt feelings
`.82
`Depressive feelings
`.63
`Depression
`.74
`.78
`Motor retardation
`Motor retardation
`Mannerisms and posturing
`.46
`.68
`Mannerisms and posturing
`.51
`Uncooperativenss
`.38
`Uncooperativeness
`Disorientation
`.76
`Disorientation
`.. 60
`.63
`.81
`Unusual thoughts
`Unusual thoughts
`.11
`.20
`Emotional withdrawal
`Active social avoidance
`'Qualitative descriptors of the clinical or practical meaning of weighted Kappa or intraclass r coefficients (Cicchetti and Sparrow, 1981): <:
`.75, excellent; .60 to .74, good; .40 to .59, fair/moderate; ::; .40, poor.
`
`The P ANSS items originating from the BPRS were
`compared with 18 BPRS items in Table 3. Twenty-one
`comparisons were made because emotional withdrawal
`on the BPRS contains descriptors appropriate to
`P ANSS active and passive social avoidance as well as
`emotional withdrawal, and unusual thoughts from the
`BPRS contains descriptors that apply to the delusions
`and unusual thoughts items on the PANSS. Weighted
`Kappa is the appropriate procedure for determining
`comparability; Pearson coefficients r (which can inflate
`reliability estimates) allow reference to previous litera(cid:173)
`ture on the BPRS. Results indicated. that only three
`items (blunted affect, hallucinations, and grandiosity)
`had kappas in the excellent range (:2! . 75), which demon(cid:173)
`strates that modifications have altered the meaning of
`the items so that they are not interchangeable with
`BPRS items of the same designation. Items with the
`poorest Kappas had the poorest interrater reliability,
`which suggests that error of measurement compounds
`discrepancies between instruments.
`Pearson correlation coefficients between the 12
`added symptom items of the P ANSS and the 18 BPRS
`items revealed only modest relationships. Of the 216
`correlations, only two were above .60 (passive with(cid:173)
`drawal with blunted affect, r = .61; active social avoid(cid:173)
`ance with suspiciousness, r = .65, p < .0001). Lack
`of spontaneity was moderately correlated with BPRS
`negative symptoms (motor retardation, r = .52; unco(cid:173)
`operativeness, r = .55; and blunted affect, r = .55, p <
`.0001) and impulse control correlated with BPRS hostil(cid:173)
`ity (.55).
`
`Correlations between PANSS and BPRS syndrome
`scales revealed a very close association. Negative syn(cid:173)
`dromes were correlated .82 and positive syndromes .92.
`The general scales were moderately correlated at .61,
`whereas the scale totals were .84.
`To demonstrate the equivalence of the negative syn(cid:173)
`drome scores on the two instruments, a typology was
`created of high negative and low negative subjects us(cid:173)
`ing the median as the cutoff score on each instrument
`(PANSS > 17; BPRS > 3). Agreement was 85.72%, with
`48 subjects falling in the same category, three subjects
`rated high on the BPRS and low on the P ANSS, and
`five subjects rated high on the PANSS and low on the
`BPRS (Kappa= .72, p < .001).
`Table 4 compares the predictive power of the two
`instruments with measures of work performance. A
`maximum of four symptom items was used to predict
`each work score. The PANSS negative symptom items
`had significant predictive power for all seven work
`scores; the BPRS negative symptom items had signifi(cid:173)
`cant predictive power for five . The PANSS positive
`symptom items predicted three work scores; the BPRS
`positive symptom items predicted four. The PANSS
`general symptom items proved significant predictors of
`six work scores; the BPRS general symptoms predicted
`one.
`
`Discussion
`
`Have the P ANSS authors succeeded in developing an
`instrument that may surpass the BPRS in studies of
`
`Par Pharm., Inc.
`Exhibit 1016
`Page 004
`
`

`

`PANSS/BPRS
`
`TABLE 4
`Prediction of Work Performance Ratings from Subsets of PANSS and BPRS Positive, Negative, and
`General Symptom Ratings Using Multiple Regression
`Adjusted R2
`
`Negative
`subsets
`
`.35**
`.54***
`.24*
`.15*
`.23**
`
`PANSS•
`Positive
`subsets
`
`NS
`NS
`NS
`.16*
`NS
`
`General
`subsets
`
`Negative
`subsets
`
`.38**
`.35***
`.31**
`.44***
`.24**
`
`.27**
`.39***
`NS
`NS
`.18**
`
`Performance Variables
`WPP Ratings
`Work skills
`Social skills
`Work motivation
`Work conformance
`Personal presentation
`MSS Ratings
`.20*
`.21**
`.41**
`.33**
`.55***
`Work quality
`NS
`.17*
`NS
`.21*
`.19*
`Response to rules and authority
`"Each subset contained a maximum of four symptoms that best predicted each work performance rating (Cf Metlwd: Analysis).
`*p < .05; ** p < .01; ***p < .001.
`
`727
`
`General
`subsets
`
`NS
`NS
`NS
`.24**
`NS
`
`NS
`NS
`
`BPRS•
`Positive
`subsets
`
`.18*
`NS
`NS
`.31**
`.14*
`
`schizophrenia? The results of this study suggest they
`have. Although both instruments have high interrater
`reliability, we found superior reliability for the PANSS
`on the shared symptom items and the additional P ANSS
`items. This fmding suggests the symptom descriptors
`of the P ANSS were superior to those of the BPRS. One
`advantage of the P ANSS is that it distinguishes between
`severity and frequency of occurrence of a symptom,
`elements that are combined in a single anchor on the
`BPRS (e.g., anxiety: mild/infrequent, mild/transient, up(cid:173)
`set most of the time and/or attacks of acute anxiety,
`continuously shaken). The PANSS interrater reliabili(cid:173)
`ties may also have been enhanced by using videotapes
`as part of the training procedure. Our reliabilities for
`the BPRS were higher than those usually reported
`(Hedlund and Viewig, 1980) and may indicate that our
`reliabilities on the P ANSS are unusually high as well.
`More experience by a variety of investigators will be
`needed before reaching a conclusion about the reliabil(cid:173)
`ity of the P ANSS.
`The internal consistency of the PANSS syndrome
`scales, the general scale, and the instrument as a whole
`was greater than that of the BPRS. This suggests that
`the additional items on the PANSS and the modifica(cid:173)
`tions of shared items increased the homogeneity of the
`syndrome scales. The greatest improvement was on the
`general scale, which had been increased to 16 symptom
`items. These findings suggest that the P ANSS rational
`categorization is psychometrically valid and yields
`more homogeneous syndromes than the BPRS categori(cid:173)
`zations used by Kane et al. (1988).
`An exploration of the comparability of the shared
`items demonstrated that modifications had sufficiently
`altered the meaning of those items so as to make most
`of them noninterchangeable. A BPRS finding could be
`substituted for a PANSS fmding for only three symp(cid:173)
`toms. However, the negative and positive syndrome
`
`scores were a different matter. The close association
`between the two instruments approached theoretical
`limits based on reliability error variance. Our correla(cid:173)
`tion for negative symptom syndrome scores of .82 actu(cid:173)
`ally exceeds the . 70 between the BPRS and SANS as
`reported by Thiemann et al. (1987). The high agreement
`using high and low negative symptom categories sup(cid:173)
`ports the use of either instrument in dichotomization.
`Investigators can reasonably conclude that studies us(cid:173)
`ing BPRS negative syndrome scores can be compared
`with studies using the PANSS negative syndrome
`scores.
`The extra effort in rating the 12 additional items of
`the PANSS is justified by their generally low correla(cid:173)
`tions with BPRS items. The additional items also in(cid:173)
`creased predictive power. Although positive symptoms
`for both instruments were not especially good pre(cid:173)
`dictors of work variables, PANSS negative and espe(cid:173)
`cially general symptoms proved much more useful than
`the BPRS in explaining variance in work behavior. The
`PANSS, although it required additional effort, resulted
`in more powerful predictions of work performance.
`Important findings may be lost by employing an instru(cid:173)
`ment less sensitive to schizophrenic phenomenology
`than the P ANSS.
`Taken together, these results support continued use
`of the P ANSS in schizophrenia research. At this point,
`the P ANSS provides an instrument with considerable
`psychometric stability and predictive power, and we
`encourage investigators to consider its use.
`
`References
`Anastasi A (Ed) (1988) Psyclwlogical testing. New York: Macmillan.
`Andreasen NC (1982) Negative symptoms in schizophrenia: Defini(cid:173)
`tion and reliability. Arch Gen Psychiatry 39:784-788.
`Bartko JJ (1966) The intraclass correlation coefficient as a measure
`of reliability. Psychol Rep 19:3-11.
`Bolton B, Roessler R (1986) Manual for the Work Personality Profile.
`
`Par Pharm., Inc.
`Exhibit 1016
`Page 005
`
`

`

`728
`
`BELL et al.
`
`Fayetteville, AR: Arkansas Research and Training Center in Voca(cid:173)
`tional Research.
`Cicchetti DV (1976) Assessing interrater reliability for rating scales:
`Resolving some basic issues. Br J Psychiatry 129:452-456.
`Cicchetti DV, Fleiss JL (1977) Comparison of the null distributions
`of weighted kappa and the C ordinal statistic. Appl Psychol Meas
`1:195-201.
`Cicchetti DV, Showalter D (1988) A computer program for determin(cid:173)
`ing the reliability of dimensionally scaled data when the numbers
`of specific sets of examiner may vary at each assessment. Educ
`PsycJwl Meas 48:717-720.
`Cicchetti DV, Sparrow SS (1981) Developing criteria for establishing
`interrater reliability on specific items. Am J Ment Defic 86:127-137.
`Cohen JA (1968) Weighted kappa: Nominal scale agreement with
`provision for scaled disagreement or partial credit. Psychol Bull
`70:213-220.
`Fliess JL (1981) Statistics for rates and proportions. New York:
`Wiley.
`Gibson D, Weiss D, Dawis P (1970) Manual for the Minnesota Satis(cid:173)
`factoriness Scales. Minneapolis: Vocational Psychology Research,
`University of Minnesota.
`Hedlund JL, Vieweg MS (1980) The Brief Psychiatric Rating Scale
`(BPRS): A comprehensive review. J Oper Psychiatry 11:48-65.
`Kane J, Honigfeld G, Singer J, Meltzer H (1988) Clozapine for the
`treatment-resistant schizophrenic. Arch Gen Psychiatry 45:789-
`796.
`Kay SR (1990) Significance of the positive-negative distinction in
`schizophrenia. Schizophr Bull 16:635-652.
`
`Kay SR, Fiszbein A, Opler LA (1987) The positive and negative syn(cid:173)
`drome scale (PANSS) for schizophrenia. Schizophr Bull 13:261-
`276.
`Kay SR, Opler LA (1985) L-DOPA in the treatment of negative schizo(cid:173)
`phrenic symptoms: A single-subject experimental study. Int J Psy(cid:173)
`chiatry Med 15:293-295.
`Kay SR, Opler LA, Lindemeyer JP (1989) The positive and negative
`syndrome scale (P ANSS): Rationale and standardization. Br J Psy(cid:173)
`chiatry 155(Suppl 7): 59-65.
`Lindenmayer JP, Kay SR, Freidman C (1986) Negative and positive
`syndromes after the acute phase: A prospective follow-up. Compr
`Psychiatry 27:276-286.
`Overall JE, Gorham DR (1962) Brief Psychiatric Rating Scale. Psychol
`Rep 10:799-812.
`Singh MM, Kay SR (1975) A comparative study of haloperidol and
`chlorpromazine in terms of clinical effects and therapeutic reversal
`with benztropine in schizophrenia: Theoretical implications for
`potency differences among neuroleptics. Psychopharmacologia
`43:103-113.
`Singh MM, Kay SR, Opler LA (1987) Anticholinergic-neuroleptic an(cid:173)
`tagonism in terms of positive and negative symptoms of schizo(cid:173)
`phrenia: Implications for psychobiological subtyping. Psychol Med
`17:39-48.
`Spitzer RL, Williams JB, Gibbon M, First M (1989) Structured Clinical
`Interview for DSM-III-R. New York: Biometrics Research Depart(cid:173)
`ment.
`Thiemann S, Csemansky JG, Berger PA (1987) Rating scales in re(cid:173)
`search. The case of negative symptoms. Psychiatry Res 20:47-55.
`
`Par Pharm., Inc.
`Exhibit 1016
`Page 006
`
`