Initial Neglect of Evaluation

Despite this mandate, implementation of evaluation came slow. Three years into the program, an Education Week article based on a 1993 NCVRE study noted that Tech Prep programs were "still in an embryonic phase of development" and pointed out that there was "little consistency in purpose, design, or curriculum across states and of states" (Sommerfeld, 1993, p. 12). Two years later, Bragg (1995) concluded that "evaluation of any kind has been one of the most neglected components of Tech Prep" (p. 23).

There are many reasons for this neglect. Among them are a lack of resources and expertise to perform valid evaluations; a focus on program planning, development and implementation rather than evaluation; the slow start-up of many programs; fear that Tech Prep funding would cease; and failure by state agencies to enforce the systematic collection of evaluation information.

Nevertheless, by 1997 Custer, Ruhland, and Steward (1997) were able to report that nationwide 77% of consortia had evaluation programs in place. When looking at the content of these evaluation systems, however, it becomes apparent that most focused on Tech Prep program features. Minnesota’s evaluation model, for instance, centered on four Tech Prep systems: Curriculum and Instruction, Marketing, Student Assessment, and Support Services and Counseling and seven systems activities: Overall Planning, Staff Development, Special Populations, Curriculum Integration, Articulation, Partnerships, and Evaluation (Pucel, Brown, Kuchinke, 1996). This systemic focus can be explained by the fact that most consortia spent the majority of the early funding years on planning and development. Tech Prep, after all, required restructuring and fundamental change in the way education was conceived, planned, and delivered. In this context, process evaluation was important, timely, and meaningful (Custer, Ruhland, Steward, 1997).

A final reason for the slow adoption of Tech Prep evaluation, especially related to student outcomes, is the program implementation pattern. Many consortia spent the first several years with planning and developing the institutional infrastructure for Tech Prep. In the spring of 1992, for example, close to three-quarter of all consortia (72%) were still involved in the planning phase and had not yet begun enrolling secondary students (Office of Educational Research and Improvement, 1994). In those early years, then, planning, not evaluation was the priority. Where evaluation was conducted, it focused on process issues, such as articulation, marketing, or staff development. Enrollments increased over the following years as more programs "went live." In Illinois, for example, the number of secondary Tech Prep students for fiscal years 1993 and 1994 were small and constant (6,526 and 6,736 respectively), but quadrupled in 1995 to 24,598, rising to 44,474 in 1996, 52,620 in 1997, and 75,062 in 1998. (Source: Illinois State Board of Education). Only in recent years are there sufficient numbers of Tech Prep students and graduates to make outcome-focused evaluation feasible and meaningful.

Increasing Need for Evaluation

With rising enrollments and the conviction among administrators that Tech Prep was "here to stay," scholars began to take note of the lack of reliable program outcome information. Especially lacking was information related to actual student benefits. Bragg (1997) concluded that "the number of local consortia that were able to provide . . . data in the area of participation and completion was so limited as to make most of the estimates meaningless" (p. 10).

The focus on student outcomes has become particularly salient with the passage of the 1998 Carl Perkins Vocational-Technical Education Act (Perkins III) that mandates stringent accountability measures. The Act requires states to monitor performance using four core indicators: student attainment of academic and technical skill proficiencies, student attainment of high school and post-secondary graduation, job placement and retention, and student participation and completion leading to nontraditional employment (see Hoachlander and Klein, pp. 2-4 in this issue). Tech Prep evaluation under Perkins III places an explicit focus on clear, valid, and reliable information, collected at the building, consortium, and state levels, with clear lines of accountability and funding consequences in case of under-performance.

Evaluation and Perkins III

What then are some of the attributes of evaluation systems that can meet the requirements of Perkins III? First, successful state evaluation systems have been planned, developed, tested, and refined over time with active and vigorous participation from a broad range of local, consortium, and state stakeholders and input from evaluation experts. Such systems typically include a core of statewide indices as well as local measures.

Second, evaluation is a value-added activity in successful cases, rather than an administrative burden. Collecting, analyzing, and reporting information are viewed within the context of continuous process improvement. Valid and reliable data are critical for informed decision-making and tracking the system over time.

Third, leaders at the state, consortium, and local levels support evaluation actively and visibly, holding administrators accountable for collecting and acting upon information, and rewarding local initiatives that improve evaluation practices. This includes making time and resources available for data reporting and analysis and professional development to understand and make good use of the information. Lastly, good evaluation systems have managed to make maximal use of existing data sources, use a variety of data sources and methods, and minimize the administrative and data collection burden for evaluation.

Consortia in many states continue to struggle with several evaluation issues. These include finding a balance between local autonomy in tailoring Tech Prep programs and evaluation, and state centralization to ensure conformity and accountability. Because Tech Prep developed from local initiatives, many states also struggle with common definitions, a prerequisite for comparisons of information across consortia and reporting of statewide information. A final barrier lies in the difficulty to coordinate data and information across different agencies, such as secondary, post-secondary, and work-related student outcomes.

With Perkins III, legislators have sent an unambiguous signal for accountability in state and local Tech Prep efforts, focused very clearly on student outcomes. While process measures, such as articulation agreements and numbers of advisory board members are important, they are only a means to an end and not ends in themselves. Perkins III requires clarity over exactly how Tech Prep efforts benefit students, and student outcome information must form the center of Tech Prep evaluation efforts. Reliable and valid information on a system as complex and evolving as Tech Prep is not easy to obtain. If Tech Prep is to grow, however, all stakeholders must refocus and renew their efforts to implement systematic evaluation systems that can put solid ground under the many anecdotal signs of the promises and successes of the Tech Prep education reform.

References

American Vocational Association (1992). The Carl D. Perkins Vocational and Applied Technology Education Act of 1990: The final regulations. Alexandria, VA: author.

Bragg, D. B. (1995). Working together to evaluate training. Performance and Instruction, 34(10), 26-31. Bragg, D. B. (1997). Educator, student, and employer priorities for tech prep student outcomes. (MDS-790). Berkeley: National Center for Research in Vocational Education, University of California at Berkeley.

Custer, R. L., Ruhland, S.K., Stewart, B.R. (1997). Assessing tech prep implementation. Journal of Vocational and Technical Education, 31(2), 23-35.

Office of Educational Research and Improvement. (1994). National assessment of vocational education: Final report to congress. Program Improvement: Education Reform (Volume III). Washington, DC: U.S. Department of Education.

Pucel, D.J., Brown, J. M., Kuchinke, K. P. (1996). Evaluating and improving tech prep: Development, validation, and results of the Minnesota self-assessment model. The Journal of Vocational Education Research, 21(2), 79-106.

Sommerfeld, J. (1993, March 17). Study finds little consistency in ‘tech prep’ efforts. Education Week, 12, p.12.