Assessment of Tech Prep has two important dimensions:

• Analysis of process outcomes-based on Tech Prep's status as an education reform initiative. This includes the evaluation of the elements considered key to a Tech Prep model according to Perkins III legislation as well as others identified as central to a high quality initiative.
  
  
• Analysis of student outcomes-looking at measures of the success of Tech Prep students relative to their peers or selected benchmarks. This generally includes assessment of performance and persistence in high school, transition to college, success in college, and a variety of employment indicators.

Two especially notable national evaluations have been conducted in recent years. Both address aspects of process and student outcomes. Portions of the first, a multi-dimensional four-year study by Debra Bragg, are summarized in the lead article of this newsletter. The other, entitled Focus for the Future: the Final Report of the National Tech-Prep Evaluation (1998), was conducted by Mathematica Policy Research and drew on data from 1993 through 1997 (see box on page 7). Both studies pointed out the relative ease of looking at process outcomes, and noted important successes in this realm. However, both emphasized the difficulties encountered when trying to assess the impact of this initiative on students.

The Problem with Student Outcome Assessment in Tech Prep

Tech Prep is many things to many people, with actual program services differing widely even within a given state. While all Tech Prep consortia are required to address the required elements outlined in the Perkins legislation, this can be accomplished in many ways. Some initiatives are structured as highly defined programs, while others focus on specific activities such as curriculum improvement or work-based learning.

Further, the definition of a Tech Prep student differs widely. Definitions range from "all students," to those signing up for one or more courses, to those who are following a clear education/career plan. There are practical difficulties that emerge from this range of definitions. For one thing, they are very disparate, and when data from students identified under different systems is compiled together, the resulting information is not very meaningful or useful. Second, they include students who may not have had significant program-related experiences, reducing the likelihood that significant outcomes will be found.

Another problem has to do with following a student across systems. High school and college data bases are often unable to "talk" with each other, made more difficult by privacy protection laws and the lack of standard identification systems (e.g., using social security numbers). In fact, many Tech Prep consortia in Illinois are unable to determine whether their high school participants have enrolled in postsecondary education, and most community colleges lack the capability to determine whether their students are former or current Tech Prep participants.

Finally, student outcome data are only meaningful if there is something to compare them to. This can be some form of control/comparison group, or it can be the indicator of goal attainment that a consortium has established. Knowing that 63% of your Tech Prep students completed three years of high school math doesn't mean much by itself.

Creating a System to Assess Student Outcomes

Based on a review of current practices around the country, I believe that a methodology that allows accurate assessment of student outcomes in Tech Prep requires four things:

1) Documenting student enrollment in Tech Prep on a yearly basis. Students should be expected to formally enroll in Tech Prep (by this or any other name), and know what activities they have signed up for. A number of states including Massachusetts, Texas, and New York have standardized registration procedures. Enrollment forms are used by some consortia, and are highly recommended. Enrollment forms should include parental or student (over 18) permission for sharing of student information for evaluation purposes within the consortium.

Some consortia may want to count Tech Prep students separately from Tech Prep participants. Students are those who are following the full sequence of courses and related activities in a Tech Prep career pathway, while participants receive the benefit of certain Tech Prep activities, while not necessarily committing to the full program.

2) Defining the components of your Tech Prep program. Unless you are clear on what the Tech Prep program consists of, it's hard to evaluate it. Generally a good quality program consists of: courses arranged in career pathways that span at least the 11th through 14th grades, an expectation that high level academic courses will be integrated with (or at least taken along side of) career and technical courses, opportunities for students to develop and refine career plans, access to work-based learning, and assistance in making the transition to postsecondary education.

3) Deciding on indicators of success. It is import to be clear on what you consider to be the criteria for good performance for your program. This can be done in two ways. The strongest possible evaluation design uses a control or comparison group, and compares this group's performance with the students in the Tech Prep program. However, this requires expertise and resources, and most programs prefer the alternative: to set targets against which to measure their performance. These should take into account the Perkins indicators, your state's indicators, and items that the consortium is interested in assessing.

Examples would be:

• Performance-The percent of Tech Prep students who receive C or better in their math classes will increase from 55% to 60% between 2002 and 2004.
  
• Transition-At least 25% percent of Tech Prep students who graduate from high school will enter postsecondary Tech Prep programs in the same career area in the following year.
  
• Non-traditional course taking-At least 25% of the students in each Tech Prep program will be of the non-traditional gender during the 2004- 2005 school year.

For more helpful examples, see Sheila Ruhland's manual entitled Measuring Tech Prep Excellence: A Practitioner's Guide to Evaluation (p. 29). It can be downloaded at: http://www.nccte.org/publications/projectReports.asp (Look under Research and Development Project Reports- 2001).

4) Data collecting and synthesizing mechanisms. Finally, you need a feasible way to collect data on students. In Illinois, the ISIS system does this well at the secondary level, as long as the person entering the data is clear about who should be flagged as a "Tech Prep student." Among Illinois community colleges, the A-1 system also allows this data collection-if there is a way of identifying students who were high school Tech Prep participants. However, as in many other states, it is difficult for the two systems to connect with each other, a problem that the state is working to resolve. Consortium-level solutions are also possible, especially if forms have been signed permitting information sharing.

Finally, data should be compiled at least yearly and compared with the selected indicators (or control/comparison groups). While it is important to report the results to funders and other stakeholders, the most important use of this information is for program improvement. Each consortium's data should be a topic of discussion by its governing body, and plans should be made to celebrate successes and improve areas that are lacking.

In conclusion, it is critical that Tech Prep be able to demonstrate results. Educators risk a loss of credibility if they cannot show that the program makes a difference in students' lives. This requires the ability to 1) identify students who have participated in a full Tech Prep experience, and 2) assess their performance.

Findings of Focus for the Future: the Final Report of the National Tech-Prep Evaluation (1998), conducted by Mathematica Policy Research:
Four process-related benefits were highlighted as emerging from Tech Prep initiatives around the country:

• Increased lines of communication, especially among teachers, leading to more collaboration and better articulation.
  
• Mobilization of interest and resources to improve curriculum through a focus on contextual, real world learning, and problem solving.
  
• Stimulation of greater interaction between schools and the business community.
  
• Focusing a spotlight on the need to strengthen math and science skills for vocational students.

• Encouraging more highly structured models of Tech Prep, because they show the most promise for student achievement.
  
• Strengthening preparatory services to enroll more students into Tech Prep programs, help them to stay involved, and assist with the transition to postsecondary education.
  
• Making sure that consortia are big enough to take advantage of economies of scale, and provide a range of articulation options.
  
• Reducing barriers to full utilization of articulation agreements by students.
  

Data from Illinois' Tech Prep Evaluation System (TPES) indicating trends in three statewide student outcome indicators during a three year period (OCCRL, 2002).