Use of Peer Ratings to Assess Sociability Among Inpatients With Severe Psychiatric Disorders

Persons with schizophrenia and other severe psychiatric disorders often have impairments in social functioning that may be independent of positive and negative symptoms (1). These impairments have been shown to predict relapse rate and community functioning among outpatients (2,3,4). Given the prognostic importance of social functioning among patients with severe psychiatric disorders, procedures for identifying persons at risk for social failure and rehospitalization are needed. Once these individuals are identified, they may be able to benefit from family interventions and social skills training focused on teaching the skills necessary for maintaining community tenure and improving quality of life (5,6).

An important aspect of inpatient treatment planning is evaluation of patients' social skills and deficits. Numerous methods for evaluating social competence of inpatients with severe psychiatric disorders exist, including role play, semi-structured interviews, self-report measures, and naturalistic observation (7,8). These techniques have different strengths and weaknesses that must be considered in selecting an assessment tool. For example, self-report measures are generally less time intensive for both staff and patients, but their psychometric properties may be questionable, especially when they are used with inpatients with schizophrenia (7). Conversely, role play allows direct assessment of overt behavior and has sound psychometric properties (2), but considerable staff training may be needed to administer the intervention and to code the results. Furthermore, if the role play does not match significant problem areas in the patient's life, it may be viewed as contrived or not externally valid.

Naturalistic observation affords advantages not provided by role play or self-report. As described by Mueser and Sayers (7), naturalistic observation tends to be an ecologically valid procedure for evaluating a wide range of social behaviors. Furthermore, information may be obtained about how a number of others, not just a single individual in a role play, respond to the target patient. Therefore, the reciprocal relationship between the person and the social environment may be assessed. Some examples of naturalistic rating scales for use with inpatients include the Time-Sample Behavioral Checklist (9), the Nurse's Observation Scale for Inpatient Evaluation (NOSIE-30) (10), the rehabilitation evaluation scale of Baker and Hall (REHAB) (11), and the recently developed Social-Adaptive Functioning Evaluation (12).

Staff-rated observation of patients' naturalistic behavior has several disadvantages. First, the person's behavior may be more reflective than naturalistic because the process of being observed by staff could produce behaviors that are not typically demonstrated by the person in most contexts. For example, a male patient who tends to act sexually inappropriately with female patients may act appropriately in the presence of staff. Second, a certain amount of resources must be devoted to initiating and maintaining a staff-rated observation system. Such resources include staff training and periodic evaluation of observer reliability. Unfortunately, some treatment facilities may not have the financial support or the commitment from upper-level administration to implement a staff-rated observation system.

Peer assessment is an alternative procedure for evaluating naturalistic behavior that circumvents many of the problems listed above. Peer assessment refers to a set of techniques in which the peer group is the primary source of information about the group members' behavior. The two most popular peer assessment techniques are peer nominations and peer ratings, which have been used to measure the social competence of children and adolescents (13,14,15). Peer nomination procedures ask children to identify, or nominate, a certain number of peers who meet some interpersonal standard. For example, a child might be asked to nominate his or her three best friends (positive nominations) and three least liked friends (negative nominations). Peer ratings typically require the children to rate all other children in a group using Likert-type scales. For example, a child may be asked to rate how much he or she likes to play with each of the other children in a group on a scale from "very much" to "not at all."

Both techniques have good psychometric properties, although peer ratings tend to have better reliability than nomination procedures (14,15). Whether these two techniques measure similar constructs is controversial (13). However, it appears that peer nominations assess the level of popularity, and peer ratings measure overall acceptability among the peer group. Bukowski and Hoza (13) have argued that peer ratings, relative to nomination procedures, represent a composite measure of popularity, by virtue of incorporating information from both the "the acceptance and rejection dimensions of the popularity construct." Therefore, peer rating strategies may provide the most comprehensive data concerning an individual's popularity within his or her peer group.

This paper reports the results of an evaluation of peer ratings as a method for assessing the social functioning of inpatients with chronic schizophrenia and other severe psychiatric disorders. The findings focus on two issues. First, the reliability of peer ratings for use in this clinical population was investigated. Second, the construct validity of peer ratings was evaluated by assessing their association with staff ratings on a standard observational measure, the NOSIE-30. We hypothesized that peer ratings would be a reliable method for assessing the sociability of persons with severe psychiatric disorders and that they would be positively associated with adaptive behavior on the unit and negatively associated with maladaptive behavior.

Methods

Participants

The study participants were 32 patients treated on an inpatient psychiatric unit at the community transition program of the Lincoln Regional Center, a state facility in Lincoln, Nebraska, during the period from June 1992 to July 1992. The sample included 19 male patients and 13 female patients. Twenty-eight patients were Caucasian, three were African American, and one was Hispanic. Their mean±SD age was 34.9±7.8 years.

The majority of the patients had an axis I diagnosis of major psychiatric disorder made by a research psychiatrist. Twenty-six patients had a diagnosis of schizophrenia, four had schizoaffective disorder, one had delusional disorder, and one had personality disorder not otherwise specified. A more detailed discussion of the diagnostic procedures is presented elsewhere (16). The subjects had spent an average of 18±25.8 months on the unit. Their average daily dosage of medication was 1,780± 1,894.6 mg of chlorpromazine equivalents.

Patients in the sample had given their consent to participate in the study as part of a larger research project on the cognitive rehabilitation of patients with schizophrenia. Patients were informed that the peer ratings were optional and that refusal to participate in that aspect of the study would not affect their treatment on the unit.

Measures

Staff ratings.

The study participants' behavior on the unit was rated by staff members using the NOSIE-30 (10). The NOSIE-30 is checklist of 30 behavioral items completed by staff based on observation of patients over the previous 72 hours. The frequency of each behavior is rated on a 5-point Likert-type scale from "never" to "always."

In this study, NOSIE-30 ratings were made by staff psychiatric technicians who were blind to the study hypotheses. The ratings were collected routinely on the unit as part of the rehabilitation assessment and treatment battery. To control for minor fluctuations in patients' functioning, the weekly NOSIE-30 ratings were averaged over a four-week period.

A total of 14 staff members—eight men and four women, all of whom were Caucasian—were involved in rating patients' behavior. In general, each patient was rated for at least three of the four weeks by the psychiatric technician who was assigned to provide routine care for the patient. The second rater came from the remaining pool of day or evening shift psychiatric technicians.

NOSIE-30 data are summarized by six subscales—social competence, social interest, personal neatness, irritability, psychoticism, and psychomotor retardation. Periodic analyses of interrater reliability revealed Pearson correlations of .68 to .72 for all scales.

Peer popularity ratings.

Peer popularity ratings were obtained through a survey that listed the names of all patients who remained on the unit for at least two months. Patients were instructed to rate how much they enjoyed visiting with each patient using a 7-point Likert-type scale anchored by the statements "not at all" and "very much." Patients were instructed to leave items blank for any patients they did not know.

Thirty-six staff members, including psychiatric technicians, nurses, social workers, therapists, and other support staff in daily contact with the patients, also rated each patient's popularity using the 7-point Likert-type scale. Twenty female staff members and 16 male staff members, all of whom were Caucasian, completed the ratings.

The two indexes of patients' popularity—peer popularity and staff popularity—were derived by summing the ratings across raters and then dividing by the total number of raters. This procedure resulted in a fixed range of ratings from 1 to 7.

Using another 7-point scale, patients rated how much they believed the staff members and other patients enjoyed visiting with them. These ratings constituted patients' prediction of their own popularity as rated by staff members and other patients. Finally, a measure of each patient's overall interest in visiting with others was derived by averaging the patient's ratings of how much he or she enjoyed visiting each of the other patients on the unit.

To determine reliability, patients were asked to complete the peer ratings a second time one week after the first administration. Test-retest reliabilities were obtained by calculating Pearson r correlations for the patients' ratings at time 1 and time 2. A second administration of the survey was not conducted with the 36 staff members. After test-retest reliability was established, the patient's ratings were calculated as an average of the two scores. All analyses were conducted using peer and behavior ratings that were done during the same month.

Results

Reliability analyses and descriptive statistics

Analyses of test-retest reliability of patients' ratings showed that all ratings were highly reliable. Correlation coefficients were .93 for patients' ratings of the popularity of their peers, .79 for their predictions of staff members' rating of their popularity, .92 for their predictions of other patients' ratings of their popularity, .91 for their level of interest in others, and .95 for items left blank, indicating that the respondent did not know the patient listed on the survey form. All correlations were significant (p<.001).

Mean ratings of patients' popularity by peers and staff members and patients' mean scores on the NOSIE-30 staff-rated measure of behavior are summarized in Table 1. In general, patients were rated by peers and staff in the average range of popularity. Furthermore, staff ratings of patients' behavior were above average, compared with other long-term-stay patients with severe psychiatric disorders (17).

Correlational analyses

Peer and staff popularity ratings were significantly correlated (r=.75, p< .001), indicating that patients and staff members tended to perceive popularity similarly. Table 2 summarizes the relationships between peer and staff ratings of popularity with NOSIE-30 ratings of behavior on the unit. Most of the peer and staff popularity ratings were significantly associated with staff ratings of patients' behavior. After the Bonferroni correction was applied to reduce type I error, peer and staff popularity ratings were significantly associated with ratings on the NOSIE-30 subscales measuring social interest and neatness, and staff popularity ratings were negatively associated with the NOSIE-30 subscale measuring irritability.

Correlates of patients' popularity

Two factors that could influence patients' popularity are the number of patients not known by the respondent and the length of time an individual has been on the unit. Correlational analyses revealed a significant negative association between the number of patients the respondent did not know and popularity (r=-.39, p<.05); greater popularity was associated with knowing more patients by name. However, the relationship between patients' popularity and time on the unit was not significant.

Finally, correlational analyses were conducted to determine whether patients' popularity was associated with their overall interest in visiting with others and their ability to predict others' interest in visiting with them. The only significant correlation was between patients' popularity and the prediction of patients' ratings (r=.39, p<.05); greater popularity was associated with higher expectations that other patients would want to spend time with the respondent. The association between popularity and interest in visiting with others was not significant.

Discussion

The study had two major findings. First, popularity of inpatients with severe psychiatric disorders can be reliably assessed using a survey form. Second, peer popularity ratings were highly correlated with staff popularity ratings, as well as with staff members' independent evaluations of patients' behavior on the unit. These findings suggest that peer ratings of popularity may be a useful measure of a component of social functioning among inpatients with severe psychiatric disorders.

The consistently high correlations between peer popularity ratings and staff ratings of behavior suggest that staff ratings may partly reflect how much staff like the patients. This finding may have clinical significance, as previous research has demonstrated that the "likability" of patients is associated with outcome for individuals with psychiatric disorders (18). However, the role of likability in staff ratings of patients' behavior should not be overstated, as peer popularity ratings accounted for no more than 31 percent of the variance in any one staff-rated index of behavior. The remainder of the variance in behavior is likely to be accounted for by factors such as patients' specific behaviors, for example, prosocial behaviors, and by perceived physical attractiveness, which is related to social skill in this population (19).

What are the clinical implications of the findings presented in this paper? From an economic perspective, the use of peer ratings appears to be both cost- and time-effective. Patients completed the survey in a timely manner with little burden on staff resources. The patients did not object to completing the survey, nor were they overly concerned about how others would rate them.

A second implication for clinical work, which has been discussed by Hansen and colleagues (20), is that peer ratings may be useful for determining which types of behaviors are associated with popularity among particular patient subgroups—those identified by diagnostic group or gender, for example—and thus peer ratings could be used as outcome measures for social skills interventions. However, Hansen and associates (20) have shown that peer ratings, at the nomothetic or group level, may be too stable to detect subtle changes in social skills. For example, a summary score based on peer ratings of numerous individuals may not be sensitive to a change in a few ratings as a function of improved social skills. Thus an idiographic approach should be adopted in addition to a nomothetic approach when using peer ratings as an outcome measure for a psychosocial treatment.

The utility of an idiographic approach to peer ratings for treatment planning may be illustrated by the following case vignette. After patients completed the peer ratings, the researchers noticed that one of the respondents knew almost none of the other patients by name, despite having been on the unit for more than a year. This finding led to some concern that the patient's social isolation would interfere with psychiatric rehabilitation.

One of the authors (DR) developed an intervention to help this patient learn the names of others on the unit. Photographs of patients and direct-care staff members were taken with their permission, and a set of five photographs, each accompanied by the person's name, were presented to the patient once a week. After three months, the patient learned the names of all the patients and staff members on the unit, as well as the roles of various staff members. After this intervention, the patient began to greet other patients and staff members by name and to show signs of knowing which staff members to approach for particular kinds of assistance or information. The patient's tendency to become isolated from others may never have been brought to staff attention if not for the peer ratings.

Two unexpected findings emerged from the study. First, the association between peer ratings and social competence was significant only at the conventional significance level, which was somewhat surprising because peer ratings were considered a component of social functioning. However, the items that make up the NOSIE-30 social competence subscale—for example, "has to be reminded what to do," "has to be told to follow hospital routine"—suggest that this scale may more readily measure ability to independently follow a schedule rather than social performance, per se.

Second, the association between patients' interest in visiting with others and popularity ratings by peers was not statistically significant. This finding suggests that patients who state that they enjoy spending time with others may not necessarily visit with others frequently on the unit. These individuals may particularly benefit from social skills training, which could teach them methods to convert their interest in others to actual behavioral contacts.

The study had a number of limitations. First, although the NOSIE-30, the measure of behavior on the unit, has established validity and reliability in use with persons with severe mental illness, it provides a better assessment of general social behavior than of specific social skills. Therefore, the construct validity of peer ratings may be enhanced by including assessments of actual social skills through role plays or other formats and comparing the results to ratings by peers. Second, although the study provides evidence of a cross-sectional association between peer ratings and behavior on the unit, it does not speak to the potential predictive validity of peer ratings. Future research should examine whether peer ratings obtained during an inpatient stay predict patients' subsequent relapse rate. Finally, the generalizability of the findings to other treatment settings, such as a day program, is unknown. However, participants who have sustained contact with one another should be able to reliably complete peer ratings.

Conclusions

The study results suggest that peer ratings may provide reliable, and perhaps clinically significant, information about the social functioning of inpatients with severe psychiatric disorders. In addition to providing an alternative method for assessing a component of social behavior—sociability—the findings indicate an additional method for involving persons with severe mental illness in the treatment process. Finally, as illustrated by our case vignette, peer ratings may be useful in identifying gaps in service delivery. The ratings may help staff discern individual patients' sense of social isolation, which is often difficult to detect by traditional assessment procedures and which may not be addressed by routine interventions.

Acknowledgment

The authors thank Diane Quartarolo.

Dr. Penn is assistant professor in the department of psychology at Louisiana State University, 236 Audubon Hall, Baton Rouge, Louisiana 70803-5501 (e-mail, [email protected]). Dr. Reed and Ms. Sullivan are affiliated with the community transition program of Lincoln Regional Center in Lincoln, Nebraska. Dr. Spaulding is with the department of psychology at the University of Nebraska in Lincoln.

Enlarge table

Enlarge table