Reconceptualising the evaluation of early childhood education: seeking answers to 'the Reggio Emilia question'

Emeritus Professor Peter Moss, Thomas Coram Research Unit, UCL Institute of Education

(paper from presentation to Sightliness Initiative's Skylight group, Reggio Emilia, October 2017)

I’d like to raise with you this morning a subject I’ve been thinking about for some time: evaluation and, more specifically, what I call the ‘Reggio Emilia question’. I am going to draw on a book chapter I have just finished writing with my Swedish colleague Gunilla Dahlberg, and which will appear next year in the second edition of the book ‘Reconceptualising Early Childhood Education: a Reader’ (Dahlberg and Moss, forthcoming). The chapter is, I hope, the start of a longer project on evaluation and the Reggio Emilia question. So I’d really welcome your thoughts and comments.

But before I explain what I mean by the RE question, I want to make something clear. My starting point is that evaluation is a pressing and essential issue for early childhood education (and the rest of education), but also ultimately unanswerable. Pressing because evaluation has important consequences, for good or ill; it can be a means of control and management or of empowerment and transformation. Essential because a public resource and service must be democratically accountable to citizens. And unanswerable because evaluation can never produce a definitive and stable statement about complex and evolving systems, whether they be individual schools or networks of schools; it is always work in progress.

Reggio Emilia has coined the term ’languages of evaluation’; it was the sub-title they used for the Italian version of the book ’Beyond Quality in Early Childhood Education and care’, which Gunilla and I wrote with Alan Pence (Dahlberg, Moss and Pence, 2013). This term – ’languages of evaluation’ – expresses an important understanding: that there is not just one approach to evaluation, but many from which to choose. Furthermore, the language chosen needs to be suited to the subject of evaluation: it needs to be commensurate.

Which brings me to the Reggio Emilia question. This can be put as follows. What language of evaluation is commensurate with the epistemological, philosophical, political and ethical positions adopted by Reggio Emilia in its early childhood education? Or put another way, what language can do justice to the particular pedagogical identity of the early childhood education and municipal schools that have evolved over so many years?

These positions and that identity have been extensively explored and discussed elsewhere, so I will only offer a few pointers here. Reggio Emilia has:

• A social constructionist epistemology, in which knowledge is constructed in relationship with others, and is necessarily perspectival, partial and provisional.
• A philosophical position which, as we wrote in Beyond Quality, “in many respects seems to us postmodern” (ibid., p.122), a view confirmed by Carlina Rinaldi, albeit with an important qualification: “although Reggio may be postmodern in its perspectives, we are not for postmodernism, because ‘isms’ are risky…they simplify and lock you in” (Rinaldi, 2006, p.182).
• Strong values including subjectivity, uncertainty, the unexpected (wonder, surprise), inter-connectedness, research, experimentation and democracy.

• An understanding that education is intrinsically political, meaning choices must be made between conflicting alternatives. In the words of Loris Malaguzzi, “[pedagogy is] always a political discourse whether we know it or not. It is about working with cultural choices, but it clearly also means working with political choices” (Cagliari et al., 2016, p.267). Reggio Emilia has made particular and explicit political choices expressed, for example, in the image of the ‘rich’ child, in democracy as a fundamental value and in a pedagogy of relationships and listening.
• A relational ethics that requires deep respect for otherness, dialogue and listening.

Any evaluation that is to do justice to Reggio Emilia must, in my view, acknowledge, understand and work with these positions that have been taken by Reggio Emilia. That must be the case, too, with the evaluation of any other culturally-specific educational project. For the question posed here – how to evaluate in ways that are commensurate with the identity of an educational project – is not unique to Reggio Emilia. It applies far more widely, to any project of early childhood education that has a distinct, context-specific identity.

As time has passed, the Reggio Emilia question has become increasingly pressing, because of the growing risk of harm to Reggio Emilia (and other experiences of early childhood education with distinct identities), a risk posed by being evaluated in totally incommensurate ways, by the application of inappropriate languages.

In particular, I am thinking of a language of evaluation that seeks to measure a unique project against a standardised set of norms and criteria that claim universal applicability. This language of evaluation is a language of quantification, objectivity and certainty, and is expressed through rating scales, check lists, inspection procedures or other similar standardised measures. This is an evaluation born out of the new public management movement, which applies methods common in private business to public services, and in the process treats them as just another business. This movement has sought to manage more effectively through goals, targets and performance measures, all predetermined and quantifiable. This language of evaluation, what we might term the language of managerial accounting, has benefited from a great deal of investment of brainpower, time and money.

It poses great risks for projects like Reggio Emilia. What are the risks? To use an incommensurate language of evaluation like managerial accounting involves measuring predetermined standards and outcomes, decided on by those who are unaware of or uninterested in the singular identity of the pedagogical work in Reggio Emilia (or elsewhere). The speakers of this language of evaluation will apply their own epistemological, philosophical, political and ethical choices to an educational project that has made its own, very different choices.

In this process, much of what is so singular and important about that local project will be totally missed; evaluators only seeing what they want to see and what they can see from their perspectival position. The end product is what Loris Malaguzzi described, when referring to what he termed ‘Anglo-Saxon testology’, as “nothing but a ridiculous simplification of knowledge, and a robbing of meaning from individual histories” (Cagliari et al., 2016, p.378).

Developing this theme of simplification and meaning-robbing, and drawing on the incisive critique of Bill Readings, we can say that languages of evaluation based on standardized, externally-determined measures “presume that evaluations can be directly deduced from descriptive statements, [which is to confuse] statements of fact with judgements of value” (Readings, 1996, p.131). Evaluation understood as a statement of fact can reduce a project to a number; evaluation understood as a judgement of value requires a far more complex, conditional and provisional conclusion. For once you let values into the reckoning any hope of a universal and objective language of evaluation goes out the window as “there exists no homogeneous standard of value that might unite all poles of the pedagogical scene so as to produce a single scale of evaluation”.

Technical languages of evaluation, like managerial accounting, with their desire for quantification and their belief in objectivity and certainty, can be harmful in other ways. For applying a particular system of externally and predefined categories and classifications to projects with distinct identities, such as Reggio Emilia, does not respect the otherness of the subject of evaluation. Instead it ‘grasps’ the Other, negating its distinct identity and pressing it to conform, to become the Same. For as Gunilla Dahlberg has written,

"putting everything one encounters into pre-made categories implies we make the Other into the Same, as everything which does not fit into these categories, which is unfamiliar and not taken-for-granted has to be overcome ...To think another, Gunilla continues, whom I cannot grasp is an important shift and it challenges the whole scene of pedagogy. It poses other questions to us pedagogues. Questions such as how the encounter with Otherness, with difference, can take place as responsibly as possible." (Dahlberg, 2003, p. 270).

A current study illustrates such ‘grasping’ behaviour in evaluation, with its effect of making the Other into the Same. I refer to the International Early Learning and Child Welfare Study (IELS) initiated by the Organisation for Economic Cooperation and Development (OECD), a comparative evaluation of ‘learning outcomes’ among young children (five-year-olds) in participating countries. The IELS follows in the wake of PISA (the Programme for International Student Assessment), the well-established comparative assessment of 15-year-olds undertaken by OECD since 2000, and which at the last round of evaluation in 2015 included over half a million students in 72 countries. The first round of IELS is due to begin in 2018, and will involve the testing of samples of children in participating countries, using standardised measures of a range of cognitive, social and emotional skills.

The IELS will, in effect, be an evaluation project, evaluating not only children but early childhood education systems in participating countries.

In principle, therefore, it raises similar concerns to those raised by the Reggio Emilia question: that much of importance in a particular country’s early childhood education will be missed; and that harm may be occasioned by applying an incommensurate language of evaluation that seeks to ‘grasp’ a singular pedagogical identity and make the Other into the Same. These concerns have been expressed from New Zealand, where many years work has produced a distinctive early childhood pedagogy. In a recent article, Mathias Urban and I wrote that

"[New Zealanders] fear the IELS will lead to 'teaching to the OECD measures' and a consequent 'pedagogy of compliance', as governments are tempted 'to call on the apparent precision of numbers to prescribe and measure context-free and curriculum-free internationally developed and validated outcomes over time'. This would be to the detriment of 'the Aotearoa New Zealand early childhood sociocultural and bicultural curriculum…[which] has established a set of priorities for teaching and learning that are different from most of the other OECD countries." (Carr et al., 2016: 451, 453) (Moss and Urban, 2017, p. 254).

Only three two countries have signed up to participate in the first round of the IELS (England, Estonia and the USA); many more, including New Zealand, have declined to take part. Confined to two Anglophone countries, the IELS is unlikely to have significant effects. But other countries might still join the first round at this late stage, and more may feel impelled to participate in a future second round. Then the concerns will mount.

A second example of the risks of applying an incommensurate language of evaluation is closer to home. It is an ‘Evaluation of the Reggio Approach to Early Education’, undertaken by a team of economists led by James Heckman from the University of Chicago (Biroli et al., 2017). I encourage you to read the recently published report of this evaluation, and decide for yourselves the language used and its commensurability to Reggio Emilia’s local educational project.

Such examples make the need for languages of evaluation that are commensurate with the positions adopted by Reggio Emilia (and many other projects as well) all the more pressing and essential. Languages that can provide not only more understanding, but that are ethically responsible in their relationship to what is evaluated.

But what might those languages of evaluation be? In the chapter that Gunilla and I have written we explore one possible language that we believe would be suited to evaluating the early childhood education in Reggio Emilia and its municipal schools. This might be termed the language of democratic accountability, and one way in which this language can be expressed is through pedagogical documentation

As you know, pedagogical documentation is a process that first makes visible the subject of interest, for example children’s learning processes and strategies or democratic practices, doing so through varied types of documenting (e.g. hand-written notes of what is said and done, audio recordings and video camera recordings, still photographs, computer graphics, children’s work itself etc., etc.); then subjects this documenting to processes of interpretation or meaning making through listening, dialogue, reflection and negotiation, an essential element being that this meaning making is undertaken in democratic relationship with others. Those others may be children, teachers, other school workers, parents or other citizens. The combination of making the subject of interest visible through materiality and then working to make meaning of that subject, leads first to constructing and deepening understandings; but then, if wanted, can move on to making judgements of value – or evaluations.

Pedagogical documentation is very different to the language of evaluation represented by, for example, OECD’s International Early Learning Study or the economists’ evaluation of Reggio Emilia. They differ greatly in their paradigmatic position, their suppositions and their ways of working, and in particular, in their understanding of what evaluation is. As I have said, the widespread language of evaluation epitomised by OECD or the economists wants to make statements of fact, expressed in numbers. It treats evaluation as an essentially technical practice, about measuring performance against externally derived standards; it expects to achieve a definitive representation of what is evaluated and to make a categorical statement of fact.

By contrast, the language of democratic accountability, working with pedagogical documentation, treats evaluation as a judgement of value and an essentially political and ethical practice. This involves: understandings constructed and judgements reached in relation to political questions and choices; and relationships of respect for otherness expressed through listening and dialogue. It views evaluation as inherently provisional (there is always more to be understood), partial (there are always other perspectives, other ways of looking and understanding), and participatory (a wide range of people should take part).

So far, pedagogical documentation has mainly been used at the level of the group or classroom, for example to research children’s learning processes in project work. One of the big challenges, as I see it, is whether the principles of pedagogical documentation can be applied at and to a larger scale, for example to the whole school or to the system of services in a particular area - in Reggio Emilia’s case its network of 47 municipal schools. Can it, for example, respond to the questions often asked by visitors to Reggio Emilia: what difference does the network of municipal schools make to the city and its citizens? Does the early childhood education leave traces on children as they move into and through compulsory schooling? A commensurate language of evaluation needs to be able to accommodate and work at all of these levels.

Similarly, a commensurate language of evaluation needs to be wide-ranging. Reggio Emilia’s municipal schools clearly have education and learning as major purposes, so a language of evaluation must be able to offer insights into these purposes. But the schools have other purposes: for example, being places for democratic practice, for creating culture, and for building solidarities within the community. A language of evaluation must be able to encompass these and other purposes.

Gunilla and I think that pedagogical documentation has the potential to work at different scales, but much more work needs to be done on this aspect. How exactly might you evaluate, using pedagogical documentation, at the school or system levels? What new sources of documentation might be needed? And what conditions would be necessary to ensure such evaluation involved widespread participation and achieved truly democratic accountability?
.
These are just some of the many questions that Gunilla and I hope to work on further. But we don’t have a monopoly of wisdom when it comes to finding answers to the Reggio Emilia question. All those who value and respect Reggio Emilia’s early childhood education, and who believe that evaluation is pressing, essential and unanswerable can contribute to this important task of reconceptualising evaluation. So, I look forward to hearing about your discussions and receiving your comments.

References

Biroli, P., Del Boca, D., Heckman, J.J., Heckman, L.P., Koh, Y.K., Kuperman, S., Moktan, S., Pronzato, C.D. and Ziff, A. (2017) Evaluation of the Reggio Approach to Early Education (IZA DP No. 10742). Available at: http://ftp.iza.org/dp10742.pdf.

Cagliari, P., Castegnetti, M., Giudici, C., Rinaldi, C., Vecchi, V. and Moss, P. (2016) Loris Malaguzzi and the Schools of Reggio Emilia: A selection of his writings and speeches 1945-1993, London: Routledge.

Dahlberg, G. (2003) ‘Pedagogy as a loci of an ethics of an encounter’, in M. Bloch, K. Holmlund, I. Moqvist and T. Popkewitz (eds) Governing Children, Families and Education: Restructuring the Welfare State. New York, NY: Palgrave

Dahlberg, G. and Moss, P. (forthcoming) ‘Reconceptualising evaluation in early childhood education’, in M.Bloch, G. Cannella and B. Swadener (eds) Reconceptualizing Early Childhood Care and Education: Critical Questions, New Imaginaries and Social Activism: A Reader (2nd edn). New York, NY: Peter Lang.
Dahlberg, G., Moss, P. asnd pence, A. (2013, 3rd edn.) Beyond Quality in Early Childhood education and Care: Languages of Evaluation. London: Routledge.

Moss, P. and Urban, M. (2017) ‘The Organisation for Economic Co-operation and Development’s International Early Learning Study: What happened next?’, Contemporary Issues in Early Childhood, 18 (2), 250-258.

Readings, B. (1996) The University in Ruins. Cambridge, MA: Harvard University press

Rinaldi, C. (2006) In Dialogue with Reggio Emilia: Listening, researching and learning. London: Routledge