he relationship between assessment and accreditation has been described as a dance. But, are we looking at the fancy footwork of Fred and Ginger, or another dance entirely - The Masochism Tango? Who leads, who has to do it all backward in high heels, and how has the act been received by its audiences? The following remarks examine five propositions regarding the assessment-acccreditation relationship and raise questions about the direction in which the dance is taking us.
I. "It Takes Two to Tango": The chief driver for assessment has turned out to be accreditation.
This observation may seem so obvious as to hardly warrant mention. Since the late 1980s, the inclusion of assessment of student learning in accreditors' standards has undoubtedly inspired more assessment plans than any other single factor in American higher education. But this level of influence was not at all apparent or anticipated when the movement to assess student learning first began to gain momentum in the mid-1980s. At the time, postsecondary educators were keenly aware of calls for reform of undergraduate education from within the academy (National Institute for Education's Involvement in Learning, 1984; Association of American College's Integrity in the College Curriculum, 1985 ). They shuddered at the increasingly insistent calls for accountability coming from outside the academy (e.g. the National Governors' Association's Time for Results , 1986; state mandates). The academy responded primarily in relation to these forces, and only secondarily to accreditation.
Prodded to require assessment by the federal government in 1988, accreditation slipped in under the radar and then discovered that assessment was actually a very good fit. Many of the traditional characteristics of accreditation turned out to be characteristics of good assessment practice, as well. Accrediting agencies, particularly the regionals, are respectful of the diversity among institutions, asking institutions to demonstrate effectiveness in relation to self-defined mission and goals. They have not prescribed specific approaches to assessment, either, granting institutions the flexibility to define their own paths. The slow but steady rhythm of accreditation - typically a cycle of fifth-year reports and decennial reaffirmation - is suited to the requirements for meaningful implementation of assessment, which must be viewed as a long-term, indeed never-ending process.
When criticism of higher education grew more strident in the late 1980s, the role of accreditation in ensuring quality was largely overlooked - to the chagrin of the associations, which responded by seizing the opportunity that assessment presented. Associations revised their standards to reflect the new expectation for assessment of student learning, and their workshops on how to approach assessment and reaffirmation were among the most popular at annual American Association of Higher Education (AAHE) assessment conferences. Thus, directly or indirectly, accreditation has contributed enormously to the development of campus expertise. They also led by example, moving from an overwhelming emphasis on inputs and processes to inclusion of learning outcomes . The timing was also fortuitous. Accrediting agencies were able to join their efforts with resources provided by AAHE and the American Association of Colleges and Universities. They were able to ride the wave of new pedagogies, new attention to learning styles and non-traditional students, new definitions of scholarship - indeed, the whole paradigm shift from teaching to learning and from teacher- to student-centeredness.
The result was a happy pas de deux : accrediting agencies used assessment and the new focus on student learning to reinvent themselves and reestablish their credibility, even as assessment used accreditation to establish itself as an essential element of campus practice. Because accreditation is something few campuses can ignore, money, staff, and serious attention were invested in assessment specifically in order to prepare for upcoming accreditation deadlines. Institutions' resistance to assessment was relatively muted, compared with the suspicion and resistance that had greeted demands for assessment from national and state government. A fundamental compatibility emerged: if they synchronized their "footwork," both assessment and accreditation could serve both internally, to improve student learning, and externally, for accountability.
The trick was - and remains - to get the balance right. The partnership has not been without problems. One was the initial tendency of accreditors to ask for a plan - and for institutions to consider the job done for the next five years once they had complied. There has been a great deal of planning to plan and not nearly enough action over the past 15 years or so. Related to that delay are the 5- and 10-year cycles of assessment: an asset if there is steady progress between self-studies or reports, but a liability if it leads to only episodic attention and if campus expertise in assessment, far from increasing over time, actually decays between accreditation deadlines. Campuses tended to want a recipe that would assure a good report, instead of developing an approach to assessment that would make sense for the campus. Weak visiting teams with little knowledge of assessment and low expectations sent mixed messages about its importance. On campus, administrators could use accreditation as a lever for getting assessment started, but they frequently also overused the external threat, creating an unproductive "us-against-them" dynamic and causing the campus to lose sight of the real reason to do assessment - and ironically, the real reason accreditors want campuses to do assessment, too: because, as responsible educators, we all want to improve learning and demonstrate accountability.
The question arises: are assessment and accreditation still equal partners, or has assessment been overshadowed by accreditation? What does serve today as the chief motivation for assessment, beyond accreditation? Is assessment becoming increasingly defined by the standards of accreditors, for example, by the numerous process questions that characterize North Central's AQIP project? Or by the focus on evidence and the many legal metaphors that characterize Western Association of Schools and Colleges (WASC) literature? What are the implications of including student development along with more traditional academic learning goals, as Middle States does? Is assessment losing its intellectual center of gravity? Is Fred pushing Ginger around? Does it matter?
II. "Shall We Dance?" There is a yawning chasm between theory (accreditation standards) and practice (what happens on campus).
Many of us have talked about the difference between the "paper curriculum" (what's in the catalogue), the "taught curriculum" (what is offered), and the "learned curriculum" (what students actually take away from a course or program). There's an analogy here to accreditation and assessment.
A few years ago Cécilia López, then an associate director of the North Central Association (NCA), conducted a study of approximately 600 NCA campuses. She reviewed the self-studies and team reports submitted to the association between 1989 and 1999 and discovered that despite a consistent message from the association over those ten years, remarkably few institutions had been able to move from planning for assessment to implementing it. Fewer still had moved from implementation to actually using findings for improvement of learning.
López also identified problems that had contributed to these results. She found that simple "awareness" of assessment was not enough; faculty needed opportunities to learn about assessment and how to do it well. Related to that, there were pervasive misunderstandings - about what assessment was, why it was necessary, and how to benefit from it - that prevented implementation. As a result, she found both emotional resistance and a lack of skills or knowledge as well as a lack of rewards for individuals and programs that carried out productive assessment. Her findings are unsurprising and probably generalizable to other regions.
López did not mention something else that seems relevant here: the nature of profound, integrated learning - in assessment as in other fields - for faculty and for students and others. Deep, transformative learning is not linear but rather exponential. For a beginning learner in any field, every new word or concept is a window to a new world of knowledge, but it represents only a fragment of the full complexity of the field. As a learner becomes acquainted with more and more words, concepts, techniques, examples, proofs, exceptions and questions, there are more and more potential connections to be made, more and more situations both routine and exceptional to be dealt with. That means, to borrow from the American Council on the Teaching of Foreign Languages' conceptual framework for oral proficiency, that while progress from "novice" to "intermediate" may be swift, it takes far more work to move from "intermediate" to "advanced," and many multiples of that time and experience to approach "superior." The inverted pyramid provides an apt graphical representation of this phenomenon.
The Assessment Pyramid
The point is that most institutions and accreditors have taken a rather superficial approach to the question of what it would take for an institution to achieve implementation of assessment at a superior level, grossly underestimating the time and effort involved. A charge to a committee, a trip to a conference, a few faculty development seminars, a plan - these were supposed to be adequate. They are not. To do assessment right requires a significant amount of new knowledge, genuine engagement, a shift in habitual practices, and a sufficient investment of time and opportunity to accomplish all that. So the question is: do we have the time, energy, and humility, as institutions and as individuals, to reach some functional level of proficiency? Are we capable of persisting to that point - or are we going to get stuck? Dancing well may look effortless, but it takes endless practice.
III . "The Masochism Tango": Fear of consequences can doom good assessment; so can the absence of consequences.
I frequently argue that assessment requires a safe environment in order to function with integrity and effectiveness. In other words, it's essential to dissociate it from program review and faculty evaluation. We know that anxiety makes people risk-averse. In evaluations, there is inevitably something at stake that leads to anxiety: a career, faculty lines, new facilities, equipment, sometimes even the very existence of the program. Under those circumstances, the temptation for faculty to game the system and produce good news is virtually irresistible. But with assessment, the point is not to produce "good news." It is primarily to im prove, and only secondarily to prove. The point cannot be simply to generate reams of data and then say, "See? We're excellent. Now leave us alone. Better yet, give us more resources." The point is to say, "Here's the question we posed about learning. Here's the problem. Here's how we fixed it."
I've been making that argument for years. It makes sense to me. People have listened and said it made sense to them, too. The problem is it's not working. I am not aware of any institution, in any region, that lost accreditation over the last ten or fifteen years because of a failure to do assessment. Surely that is a risk-free environment if ever there was one! It should have led institutions to ask hard questions about learning; to devise ambitious, innovative assessments; to make useful discoveries; and arrive at creative solutions. It didn't - or if it did, those findings are a closely-held secret.
Peter Ewell argues that assessment does, in fact, need to be "consequential," and he's right. I would only add that assessment needs to be consequential in the right way. We can't simply go back to rewarding programs that prove they are "strong" and starving the "weak." What would that mean? Maybe it means that the quality of an assessment effort is judged by a different set of criteria: not just the presence of a plan or a lot of Xs in a matrix, but the specificity of its questions and the effectiveness of its responses. Maybe it means broad participation and open discussion, both within and beyond the campus, about what has been found and what has been done to achieve improvement in order to determine whether there is more general relevance to a discovery: broader applicability for a solution. Maybe it means collective introspection and meta-awareness of the sort that WASC now requires in its reflective essays, or the open dialogue between programs and reviewers that the ABET process now employs.
Consequences need to be there, too: significant rewards, but awarded because the campus can document that it has an effective assessment process and can show value added, not for maintaining the status quo , no matter how good that is. Consequences need to flow at the institutional and program levels, and on the individual level, too. Much has been made of faculty as the chief obstacle to assessment. But faculty are like everyone else. They're not masochists; they read the signals and follow the money. Why has it been so difficult to provide tangible, public rewards to faculty in tenure and promotion reviews? How far are we from that goal, and why on earth is it taking so long?
IV. "Twist and Shout": Reporting is not communicating, and none of us, accreditors or academics, have communicated often enough and substantively enough to each other, to policy makers, or the general public.
Good, meaningful assessment leads to substantive findings about student learning; we discover through assessment what students learn easily and well, what is more difficult, where the difficulties lie, and what to do about them. And yet, how many of us, at any assessment conference, have talked about the assessment process - strategies, resources, techniques, pitfalls, and the like - and how many of us have shared actual findings? Assessment gatherings over the last 10 to 15 years have offered a mind-numbing array of panels on process; actual findings or results of an intervention, in contrast, are exceedingly rare.
What sort of findings? Let me cite a few examples from my own experience. At a community college, the assessment committee looked at writing across the curriculum. We developed a template for a common assignment and a rubric for scoring the essays. Students were asked to respond to an article related to their course and then do four things: take a position in relation to the reading, organize the essay to support their position, develop their position drawing on their knowledge or experience, and demonstrate command of standard American English. We discovered that, across the board, students had particular difficulty with the third criterion: ability to develop or elaborate on their position. This was a useful finding; it provided explicit guidance to instructors in all kinds of courses about a central critical thinking and writing skill, and it gave them a focus for talking to their students about effective writing. Or there are the students who do well in math when they can plug in a formula but don't know how to approach an unstructured problem; surely this is a shared concern. Or there are the students who've studied abroad. In my language department, we routinely ask students to write a reflective essay about their experiences and what they learned. They readily describe details of everyday life; what they seem unable to do, even with much urging, is get past the purely descriptive, past the level of "Oh, wow" and "I really changed a lot" to a deeper analysis of cultural values or how and why, exactly, they have changed.
I am dead certain that none of these phenomena are unique; I believe they are widely shared traits of U.S. students more generally, and they are things it would be useful to address beyond the local campus. Why don't we? Perhaps faculty think such specifics would be of interest only to writing or math or foreign language instructors; perhaps they think they belong only at discipline-based conferences. Perhaps institutions are reluctant to reveal any problems at all. In that case, we are not modeling very effectively the courage and candor that good assessment requires.
I'm convinced that panels on problems like these would make for far more compelling assessment conferences than the current focus on how we recruited the members of our assessment committee, or why we introduced e-portfolios, or which data management program we purchased. It's a little like scientists only ever talking about why they used a spectrophotometer, but never saying what they found when they used it.
We need to share our findings, aggregate them, see how generalizable they are across institutions, states or regions, and share the solutions. Peter Ewell, in WASC's Evidence Guide , writes about how evidence of student learning should involve "multiple judgments of student performance." There's a need, he argues, for "more than one person to evaluate evidence of student learning," and "data should be submitted for broad faculty discussion and action to make recommendations that will improve student learning results." I submit that those "broad faculty judgments" should occur inter -institutionally, as well. Why not sessions on improving specific aspects of critical thinking or writing or mathematical understanding? Taking this notion a step farther, why not share these deliberations with the broader public? Why not make them partners, so that when a 15-year-old shows his parents an essay, they can say, "Yes, well, you've taken a position, but you haven't really supported it. You just keep re-asserting it in slightly different words"? Imagine discussion about what is good critical thinking or responsible use of statistics actually entering our public discourse.
V. "Dancing Cheek to Cheek": Accrediting agencies and higher education, informed by assessment and making common cause, must convey a concrete vision of educational quality and accountability to policy makers and the wider public.
Of course, if we open up the discussion, chances are we will attract some attention. Would the lay public or policy makers be interested? Possibly. Should they be? Absolutely - but it should be an informed interest. How to achieve that? There would be a translation challenge, to start with; we would have to explain our work in jargon-free terms. But it's a challenge worth taking up, because the result could be educative both for us and for those other audiences, and we need their help. The public would learn something about writing or math or cultural analysis. They would also learn something important about education more generally: that it is about learning to think and express those thoughts, not just about memorization or regurgitation or responding to multiple-choice questions on standardized tests. They would learn that context matters and that most questions don't have straightforward "right" answers. They might actually get engaged in debates about the nature of good education, or the value of critical thinking. So why don't we do this? Why don't we share more concrete information - with each other and with the general public - about what good academic performance is and how students can get there? Why don't we fling open that window? The fresh air could be bracing. We might all dance faster and better.
At an AAHE conference last year I ran into Lee Shulman. I explained that I was there to lead a panel on accountability. "Well," he said, "You tell them that the problem with policy makers is not that they're asking hard questions. The problem is that the questions they're asking aren't hard enough. " That's certainly not the way we've been inclined to approach accountability. Our approach has been to provide the requisite data - FTEs, retention and completion rates, tuition levels, rates of increase, whatever - knowing full well that these surrogate indicators do not say anything about the quality of the learning that students pay for or take away. Along with policy makers, we prefer to avoid the really hard questions.
Higher education has a responsibility to be accountable. We owe that much not only to legislatures or budget committees but also to taxpayers, students, their families, and each other. But we need to be clear about the kind of education we want to be accountable for. Higher education needs go on the offensive instead of being perpetually on the defensive - or missing in action. We need to seize the initiative and set the terms of discourse about educational quality and how we - not just as educators, but as a society - propose to achieve it. After all, students are the product of the sum of their experiences, in the classroom but also in their lives beyond the campus. We academics can influence their learning but we don't control it, and all our audiences need to understand this.
How can we get from here to there? Higher education needs to exploit the possibilities of assessment and collaborate with accreditation associations. We need to go public with shared standards for college-level performance, with the problems we have diagnosed, and the solutions we propose. We need to demonstrate the complexity of educational outcomes that matter and create an informed, supportive public audience for our work. We need to put the onus on the public to complement our work, in the home, in the community, in the media. In the process, education might actually become a cultural value and our entire educational system, pre-K through 16, would benefit.
I am appalled by the mania for standardized testing that No Child Left Behind has spawned, but there are a couple of salutary things about this legislation that we must acknowledge. One of them is the message the legislation sends that education matters . Another is that we cannot afford to write off whole segments of our school population as incapable of learning. As a nation we pay lip service to the power of education to help people get ahead and lead rewarding lives. But the cultural messages that bombard young people today do not glorify educational values. Instead, popular culture celebrates wealth, celebrity, athletic prowess, power, and ruthless survivor instincts.
I'm reminded of an article that appeared in the Hartford Courant last winter. It dealt with a delay that school districts faced in getting their test scores back from the testing company. One of the things kids have to do, for NCLB and other purposes, is write an essay. Along with the rubric, the newspaper article included an excerpt from a piece of student writing that had been scored a "4" on a scale from "1" (the low end) to "5." It was a nice piece of writing, but the rubric suggested why it was not a "5." Surely this piece of writing would make an interesting subject for dinnertime conversation.
A public conversation would provide a more supportive environment for efforts to link higher education with K-12, too. Exhortations could be replaced by substantive collaboration on ways to achieve specific, publicly endorsed outcomes that are developed in primary and secondary schools and then brought to a higher level of proficiency in postsecondary study. In other words, articulation efforts could move beyond the current focus on curriculum and tackle concrete, widely understood deficiencies in outcomes.
Accreditation associations have a long tradition of non-disclosure and have been discomfited by recent calls for more transparency regarding member institutions' status and the findings of accreditation reviews. Assessment is a younger phenomenon with more freedom to open this debate and set its terms, if we can find the courage to do so. Assessment gives us the tools for understanding and communicating more clearly about what good education is. It may be time for assessment to take the lead and open that more public conversation. The fact is that ultimately, education in this country needs not isolated improvements in this program or on that campus; to achieve real improvements in quality, education needs a more encompassing conceptual and practical infrastructure in a new environment of broad social support. That degree of reform won't be accomplished by a single entity; it will take collaboration from many directions.
Assessment and accreditation can demonstrate how it's done by taking the first steps together. How will the act be received by its audience? Whether we consider the audience to be academics, policy makers, or the wider public, the answer is: we don't know, because they're really not tuned in yet. But education, in contrast to entertainment, is an endeavor in which we - educators and the public alike - cannot afford simply to be passive consumers of the spectacle.  Dr. Barbara D. Wright is the Assessment Coordinator of Academic Affairs at Eastern Connecticut State University . Prior to that, she served as a faculty member in German at the University of Connecticut . During her tenure at UCONN she directed a FIPSE-funded project to assess a new general education curriculum, and from 1990 to 1992 she served as director of the American Association for Higher Education's Assessment Forum. From 1995 to 2001 she was a member of the New England Association of Schools and Colleges' Commission on Institutions of Higher Education, and she has participated in team visits for several regional accreditors. She has published on assessment and is frequently invited to campuses to speak and conduct workshops. She is especially interested in qualitative approaches to the assessment of general education's more challenging, seemingly ineffable goals. She can be reached at WrightB@easternct.edu.
|