Student Rating of Teaching Effectiveness (SRTE) Procedures

Appendix B

SENATE COMMITTEE ON FACULTY AFFAIRS

Student Rating of Teaching Effectiveness (SRTE) Procedures

(Advisory and Consultative)

Background

Penn State has been using the SRTE to provide student input into the faculty evaluation process for about 15 years. While there has been an internal study of its reliability (“An Analysis of the Penn State Student Rating of Teaching Effectiveness: A Report Presented to the University Faculty Senate,” September 9, 1997), the SRTE instrument needs to be continually scrutinized because it remains a significant part of our faculty evaluation process. At the request of the Senate Committee on Faculty Affairs, Vice Provost Robert Secor conducted a survey of SRTE use. The survey was University-wide and was conducted in the summer of 2002. The results of the survey are summarized in Appendix A. These results suggest the need for some clarification of procedures as well as some stricter oversight. In addition, the results suggest the need for some changes in procedures for the use of the SRTE in the faculty evaluation process.

Discussion

The survey results indicate the following areas need to be addressed:

(1)    Frequency of reviews by the SRTE

The original Senate legislation allowed academic units to determine the frequency of SRTE use for their faculties, but cautioned against using the SRTE for every course every time it is taught. Research at the time the SRTE was created suggested that such frequent evaluation made students take student evaluations less seriously. Later research indicates that this regular evaluation does not have this effect and is advisable if the results are to be used for performance evaluations (as the SRTE is). The Secor survey reveals that most units are administering SRTEs at every opportunity, in apparent violation of the policy as it is currently written.

(2)    Determining which SRTE results are to included in dossiers

There are indications that some units allow individual faculty members to decide which of their SRTE results are included in their dossiers. In addition, some units allow faculty members to decide which student comments are included. Clearly, if we are to have any confidence in SRTE results, all results must be included in candidates’ dossiers, not just self-selected results.

(3)    Student Comments

Most units include summaries of student comments in tenure and promotion dossiers, and in the majority of cases the academic unit head or his or her delegate summarizes the comments. However, in some units, candidates themselves prepare these summaries, and in a few units the candidates make the decision about whether to include student comments in the dossier.

(4)    Procedural Irregularities

The survey revealed that a few units blatantly violate policy, e.g., by allowing faculty members to distribute or collect their own SRTE forms. Stricter oversight is necessary to assure adherence to mandated procedures.

Recommendations

1.    Frequency of Reviews

Because most units do use the SRTE to evaluate each course every time it is taught and since research now suggests doing so is useful when student evaluations are used for performance reviews, we are recommending the following changes in the “frequency of review” section of the Administrative Guidelines for HR23.

The current version of this section reads as follows:

11.    Frequency of reviews
a.    The specific procedures for determining the frequency of reviews for the faculty members within a unit shall be determined by the unit, based on guidelines developed by the coordinating office. Surveys shall also be conducted when requested by the faculty member. The following principles about the frequency of reviews apply:

(1)    Where possible, evaluations should be conducted over a period of years and in a variety of courses.
(2)    Units should refrain from having every course evaluated every time it is taught.

Recommendation 1.We recommend that the section on frequency of reviews be changed to read as follows:

11.    Frequency of reviews
a.    The specific procedures for determining the frequency of reviews for the faculty members within a college shall be determined by the college. These procedures must be developed in consultation with the faculty of the college. The following principles about the frequency of reviews apply:

(1)    Where possible, evaluations should be conducted over a period of years and in a variety of courses.
(2)    For provisional faculty (i.e., tenure track faculty who do not yet have tenure), all sections of all courses shall be evaluated by the SRTE every time it is taught. The results from each of these evaluations must be included in in tenure or promotion dossiers. Such agreements should be in writing.)
(3)    For all other faculty, each college must develop clear and specific guidelines to accomplish the purpose outlined in the principles stated in section 11a of the Administrative Guidelines. Since students now expect to have the opportunity to evaluate their instructors and their courses and since such evaluations continue to have value for many purposes, it is recommended that all sections of all courses shall be evaluated. College Guidelines will be reviewed by the Office of the Provost to ensure that they are consistent with these principles.
(4)    Faculty being reviewed for promotion, even when it is not coupled with a tenure review, should be able to demonstrate their teaching achievements in part through student evaluations that have been done over time and in a variety of courses.

2.    Inclusion of Results

Recommendation 2.    We recommend that the procedures require that the results from all SRTEs that have been conducted during the appropriate period be included in tenure and promotion dossiers. There shall be no selecting of which results should be included and which excluded.

3.    Student Comments

The current statement about summarizing student comments says, “If student comments from such sources as student evaluations, formal interviews or exit surveys are reviewed, the findings should be presented by a summary statement that conveys the students’ sense of strengths and weaknesses.”
Recommendation 3.    We recommend adding the following statement to these guidelines:

The summary shall be prepared by the department head, division head, director of academic affairs, or the department head’s administrative or faculty delegate, perhaps in consultation with the chair of the unit’s tenure and promotion committee. Under no circumstances shall the candidate be involved in preparing the summary of student comments.

4.    Procedural oversight

Recommendation 4.    We recommend that academic units, under the guidance of the Office of the Provost, take responsibility for seeing that proper procedures are followed in the use of SRTEs. Under no circumstances shall a faculty member be present during the administration of the SRTE for the evaluation of his or her teaching.

Additional Recommendations

Discussions of the SRTE suggest that some further refinements of the SRTE could enhance its usefulness. Among these changes is the selection of items for the B (discipline-specific) section.

5.    Selection of Items for the discipline-specific (B) section

Items for section B are chosen by academic units so that unit-specific concerns and behaviors can be addressed and evaluated. The original legislation required that academic units (departments or department-like units such as divisions) make these selections from a large pool of items.

Two developments in the last decade suggest that it may be time to make some changes in the selection of items for the B-section. First, the academic structure of the University has been radically changed. When the original legislation was implemented, all faculty at Commonwealth Education System campuses were members of University-wide departments. Now faculty at those locations are members of new academic units (generally divisions of new colleges), although some have retained their connections to their UP-based departments. It makes sense to allow the faculty of these new units to make their own selection of items from the pool for inclusion in the B-section.

The second development that suggests that it is time to reconsider which items should be in the B-section is the change in the way classes are taught. In the last 15 years, there has been a significant shift away from traditional lectures toward more active and collaborative classrooms. Thus questions that were once appropriate for evaluating instructors may not be as relevant today; and new questions that were not important may be crucial for evaluating instructors in this new era of instruction.

Recommendation 5.    Therefore we recommend that the items for the B-section be chosen by current academic units. For faculty in the campus colleges, that will mean the academic divisions in their colleges. For faculty at University Park, the unit will remain the department or department-like entity.

Recommendation 6.    We further recommend that every academic unit review the questions it has chosen for inclusion in the B-section of the SRTE in light of its current academic practices and concerns.

As units review their selections, they should keep in mind that different courses are often taught in different ways and thus may require different sets of questions for appropriate evaluation. This point is most obvious for multi-discipline divisions; the questions that are best for one discipline may not be best for another within the same division. However, it may also be true even for a single-discipline unit. For example, the questions appropriate for an English composition course may not be the best questions for evaluating a literature course, even though both are taught by the English department. Similarly, science departments may want to use a different form for evaluating laboratory courses than they use for regular classroom teaching. Therefore units may want to have different forms for different courses. Some departments have already been doing this but it is not clear that everyone realizes that they may do so.

Conclusion

We believe that the changes we are proposing will enhance the usefulness of the SRTE for evaluating the quality of teaching. However, our evaluation of the SRTE, indeed of all our methods of evaluating teaching, is an ongoing process. As new methods of evaluation are developed or new evidence arises, we must look anew at our practices to make sure that we are using the best practices available. One issue under current study is the order of the A and B sections of the SRTE. The questions in section A are the most general and play the most central role in promotions and tenure. They include the student’s overall evaluation of the course and of the instructor. There is some evidence that more meaningful results will be obtained if students are asked about specific instructor attributes first (such as those in section B) and then make a general judgment about the instructor. This simply requires that we reverse the order of sections A and B on the SRTE form. We are currently researching this question and will return with a specific recommendation if the results indicate that a change is warranted.


SENATE COMMITTEE ON FACULTY AFFAIRS 2002-2003

Susan M. Abmayr
Mohamad A. Ansari
Judd Arnold
Kultegin Aydin
Thomas W. Benson
Leonard J. Berkowitz
Clay Calvert
Michael J. Cardamone
Richard A. Carlson
Debora Cheney
Roy B. Clariana
Elizabeth J. Corwin
Robert P. Crum
Dwight Davis
Mary I. Frecker
Margaret B. Goldman
David J. Green
Amir Khalilollahi
Sallie M. McCorkle, Vice-Chair
Arthur C. Miller
Jamie M. Myers
Katherine C. Pearson
Robert Secor
Kim C. Steiner, Chair
Mila C. Su
Joan S. Thomson
Tramble T. Turner