When you talk about "the use of indicators within development models of ICT integration in education – to study the progressive phases through which teachers and students adopt and use ICT" it sounds to me like you're talking about classic Change Management – the systematic process of creating organizational change.

In that context, I think there should be great agreement. We can assess the amount of chnage (ICT adoption, usage) that happens in an educational system, and draw parallels with other change processes that its often much harder, and more ambiguous to measure the impact of that change.

Many thanks for these reflections on how to develop the matrix for greater analytical flexibility – and I like the analogy of moving from grey murkiness into the light of full colour – as institutions and schools move towards fuller autonomy and understanding of using ICT for knowledge deepening and creation.

Developing the descriptive model with case studies from partner countries to provide practical ideas for planning, monitoring and evaluation approaches using the matrix is our next stage of the toolkit development.

Yes keep on dreaming and rambling – this is exactly what is needed as we approach these challenges!

Best wishes,

Mary

You raise a pertinent question regarding the usefulness of evaluations if the devices are already different in capabilities and price by the time results are disseminated etc.

Perhaps the focus should be not the devices then. ICT does not exist in isolation but within a larger socio-cultural context of the school and education system. In relation to Mark’s question above on school assessment for e-readiness, we see that the use of ICT devices is interconnected with so many other elements in schooling systems – policy, curriculum & assessment, infrastructure, pedagogy, organization & management and professional learning.

Papert (1993) observed that as ICT enters the socio-cultural setting of the school it ‘can weave itself into learning in many more ways than the original promoters could possible have anticipated’ (p53).

So to provide policy makers with useful information we essentially require a more holistic model for studying and understanding how ICT can weave its way into the learning process.

In April of this year we in GeSCI held a North South Partnership Research workshop in Dublin gathering more than 50 participants from different education and research networks to brainstorm exactly the question you present Rob – to ask what kind of models and experiences are there in ICT in Education research and evaluation that can provide our partners with useful information for policy and practice? (see workshop details at: http://www.gesci.org/partnerships.html ).

One approach that was presented in the workshop was an Activity System model derived from Activity Theory (Engstorm 2001) currently being piloted in an Expansionist School Programme in the Southern African Development Community (SADC) Region – in a joint collaboration between the Universities of Botswana and Helsinki (Engstrom 2009).

It is a model that focuses evaluation and research on the activity system (tools, subject, rules, community, labour division, object, outcome) in which the technology is located. It is an approach which seeks to understand the tensions and contradictions that emerge as technology is introduced into school activities – for example as teachers experiment with technology (tools) in their practice to integrate ICT across the curriculum (object), how are classroom activities supported by the broader socio-cultural context of the education system – the curriculum, teaching loads, timetables (rules)? – what needs to be negotiated/ renegotiated in terms of the roles of teacher and learner (division of labour)? – what problems are encountered as you noted in your first posting Rob between the different understandings of technology use of teachers, learners, managers and parents (community)? .

This is a model that has been used successfully used for assessing ICT integration in Singaporean schools through successive Masterplans for ICT in Education – to analyse ‘successes’, ‘failures’ and ‘contradictions’ in their totality at different system levels – and to develop pedagogical models and approaches of ICT integration in schools based on emerging understanding (Lim and Hang 2003).

So whether it’s OLPC or a hole in the wall deployments – if studies focus on the totality of the ‘successful’ and ‘unsuccessful’ elements of integration of the device – then when the study is over and the device is obsolete – policy makers, administrators and teachers are still left with usable knowledge on opportunities and limitations of technology take up and how technology (whatever device it may be) can best be integrated within the broader socio-cultural context of their systems.

Best wishes,

Mary

Engestrom, Y. 2001. Expansive Learning at Work: toward and activity theoretical reconceptualization. Journal of Education and Work [Online].14 (1), pp 133-156. Available from: Academic Search Premier
http://www.library.dcu.ie/Eresources/databases-az... [Accessed 01 April 2008]

Engstrom, R. 2009. Expansive School Transformation in the SADC Region IN Workshop on North/South Research Partnerships for ICT – Education. 21 April 2009, Dublin, Ireland [Online]. Available from: http://www.gesci.org/old/files/docman/Presentatio... [Accessed 24 November 2009]

Lim, C.P. and Hang, D. 2003. An activity theory approach to research of ICT integration in Singapore schools. Computers and Education [Online]. 41, pp49-63. Available from: Academic Search Premier http://www.library.dcu.ie/Eresources/databases-az... [Accessed 15 December 2008]

Papert, S. 2003. The Children’s Machine: Rethinking Schoool in the Age of the Computer. New York: Basic Books

You raise a pertinent question regarding the usefulness of evaluations if the devices are already different in capabilities and price by the time results are disseminated etc.

Perhaps the focus should be not the devices then. ICT does not exist in isolation but within a larger socio-cultural context of the school and education system. In relation to Mark’s question above on school assessment for e-readiness, we see that the use of ICT devices is interconnected with so many other elements in schooling systems – policy, curriculum & assessment, infrastructure, pedagogy, organization & management and professional learning.

Papert (1993) observed that as ICT enters the socio-cultural setting of the school it ‘can weave itself into learning in many more ways than the original promoters could possible have anticipated’ (p53).

So to provide policy makers with useful information we essentially require a more holistic model for studying and understanding how ICT can weave its way into the learning process.

In April of this year we in GeSCI held a North South Partnership Research workshop in Dublin gathering more than 50 participants from different education and research networks to brainstorm exactly the question you present Rob – to ask what kind of models and experiences are there in ICT in Education research and evaluation that can provide our partners with useful information for policy and practice? (see workshop details at: http://www.gesci.org/partnerships.html ).

One approach that was presented in the workshop was an Activity System model derived from Activity Theory (Engstorm 2001) currently being piloted in an Expansionist School Programme in the Southern African Development Community (SADC) Region – in a joint collaboration between the Universities of Botswana and Helsinki (Engstrom 2009).

It is a model that focuses evaluation and research on the activity system (tools, subject, rules, community, labour division, object, outcome) in which the technology is located. It is an approach which seeks to understand the tensions and contradictions that emerge as technology is introduced into school activities – for example as teachers experiment with technology (tools) in their practice to integrate ICT across the curriculum (object), how are classroom activities supported by the broader socio-cultural context of the education system – the curriculum, teaching loads, timetables (rules)? – what needs to be negotiated/ renegotiated in terms of the roles of teacher and learner (division of labour)? – what problems are encountered as you noted in your first posting Rob between the different understandings of technology use of teachers, learners, managers and parents (community)? .

This is a model that has been used successfully used for assessing ICT integration in Singaporean schools through successive Masterplans for ICT in Education – to analyse ‘successes’, ‘failures’ and ‘contradictions’ in their totality at different system levels – and to develop pedagogical models and approaches of ICT integration in schools based on emerging understanding (Lim and Hang 2003).

So whether it’s OLPC or a hole in the wall deployments – if studies focus on the totality of the ‘successful’ and ‘unsuccessful’ elements of integration of the device – then when the study is over and the device is obsolete – policy makers, administrators and teachers are still left with usable knowledge on opportunities and limitations of technology take up and how technology (whatever device it may be) can best be integrated within the broader socio-cultural context of their systems.

Best wishes,

Mary

Engestrom, Y. 2001. Expansive Learning at Work: toward and activity theoretical reconceptualization. Journal of Education and Work [Online].14 (1), pp 133-156. Available from: Academic Search Premier
http://www.library.dcu.ie/Eresources/databases-az... [Accessed 01 April 2008]

Engstrom, R. 2009. Expansive School Transformation in the SADC Region IN Workshop on North/South Research Partnerships for ICT – Education. 21 April 2009, Dublin, Ireland [Online]. Available from: http://www.gesci.org/old/files/docman/Presentatio... [Accessed 24 November 2009]

Lim, C.P. and Hang, D. 2003. An activity theory approach to research of ICT integration in Singapore schools. Computers and Education [Online]. 41, pp49-63. Available from: Academic Search Premier http://www.library.dcu.ie/Eresources/databases-az... [Accessed 15 December 2008]

Papert, S. 2003. The Children’s Machine: Rethinking Schoool in the Age of the Computer. New York: Basic Books

You raise a pertinent question regarding the usefulness of evaluations if the devices are already different in capabilities and price by the time results are disseminated etc.

Perhaps the focus should be not the devices then. ICT does not exist in isolation but within a larger socio-cultural context of the school and education system. In relation to Mark’s question above on school assessment for e-readiness, we see that the use of ICT devices is interconnected with so many other elements in schooling systems – policy, curriculum & assessment, infrastructure, pedagogy, organization & management and professional learning.

Papert (1993) observed that as ICT enters the socio-cultural setting of the school it ‘can weave itself into learning in many more ways than the original promoters could possible have anticipated’ (p53).

So to provide policy makers with useful information we essentially require a more holistic model for studying and understanding how ICT can weave its way into the learning process.

In April of this year we in GeSCI held a North South Partnership Research workshop in Dublin gathering more than 50 participants from different education and research networks to brainstorm exactly the question you present Rob – to ask what kind of models and experiences are there in ICT in Education research and evaluation that can provide our partners with useful information for policy and practice? (see workshop details at: http://www.gesci.org/partnerships.html ).

One approach that was presented in the workshop was an Activity System model derived from Activity Theory (Engstorm 2001) currently being piloted in an Expansionist School Programme in the Southern African Development Community (SADC) Region – in a joint collaboration between the Universities of Botswana and Helsinki (Engstrom 2009).

It is a model that focuses evaluation and research on the activity system (tools, subject, rules, community, labour division, object, outcome) in which the technology is located. It is an approach which seeks to understand the tensions and contradictions that emerge as technology is introduced into school activities – for example as teachers experiment with technology (tools) in their practice to integrate ICT across the curriculum (object), how are classroom activities supported by the broader socio-cultural context of the education system – the curriculum, teaching loads, timetables (rules)? – what needs to be negotiated/ renegotiated in terms of the roles of teacher and learner (division of labour)? – what problems are encountered as you noted in your first posting Rob between the different understandings of technology use of teachers, learners, managers and parents (community)? .

This is a model that has been used successfully used for assessing ICT integration in Singaporean schools through successive Masterplans for ICT in Education – to analyse ‘successes’, ‘failures’ and ‘contradictions’ in their totality at different system levels – and to develop pedagogical models and approaches of ICT integration in schools based on emerging understanding (Lim and Hang 2003).

So whether it’s OLPC or a hole in the wall deployments – if studies focus on the totality of the ‘successful’ and ‘unsuccessful’ elements of integration of the device – then when the study is over and the device is obsolete – policy makers, administrators and teachers are still left with usable knowledge on opportunities and limitations of technology take up and how technology (whatever device it may be) can best be integrated within the broader socio-cultural context of their systems.

Best wishes,

Mary

Engestrom, Y. 2001. Expansive Learning at Work: toward and activity theoretical reconceptualization. Journal of Education and Work [Online].14 (1), pp 133-156. Available from: Academic Search Premier
http://www.library.dcu.ie/Eresources/databases-az... [Accessed 01 April 2008]

Engstrom, R. 2009. Expansive School Transformation in the SADC Region IN Workshop on North/South Research Partnerships for ICT – Education. 21 April 2009, Dublin, Ireland [Online]. Available from: http://www.gesci.org/old/files/docman/Presentatio... [Accessed 24 November 2009]

Lim, C.P. and Hang, D. 2003. An activity theory approach to research of ICT integration in Singapore schools. Computers and Education [Online]. 41, pp49-63. Available from: Academic Search Premier http://www.library.dcu.ie/Eresources/databases-az... [Accessed 15 December 2008]

Papert, S. 2003. The Children’s Machine: Rethinking Schoool in the Age of the Computer. New York: Basic Books

Mary, I agree with Rob that your quadrant helps illustrate the assessment challenge. It reminds me a little of Bloom's Taxonomy, but the matrix allows for greater analytical flexibility, elevating shades of gray to a full color palette. Perhaps assessment should be keyed to a descriptive model describing ideal learning environments and through structured observation, conversation and product-analysis, individual cases could be analyzed against a "best practice" standard.

By the same token for learners, goal-based learning attributes could be described against which learners' achievement might be assessed. Some of this assessment can indeed be quantitative, even test-centered, but not all of it. The point is that success would be measured against things deemed worthy of achieving, and narrative valued as highly as numbers, if not more so.

But I dream — and ramble…. Back to the real world where I'll pick up my laundry, pondering the degree to which my shirts match my ideal model of cleanliness.

I will kick off the cross commenting with a problem I see in the ICT4E evaluations.

Any halfway thorough evaluation of a new technology in education will take at least three years. In education, the time horizon is in decades. So in general, this time is not a problem.

However, if that technology is a specific ICT technology, none of the devices tested will be available anymore at the end of the study. An example would be a 1:1 laptop initiative, eg, the OLPC. At the end of the test period, new devices with new capabilities and lower prices will be available, making the old evaluation worthless for policy decisions.

Unless, of course, the evaluation is much more general with results that can be transfered to other, newer, devices and technologies.

Your matrix promotes such a more general view.

But can you explain more about how a specific program can be rationally and empirically evaluated for policy purposes without becoming obsolete by the time of publication?

For illustrative purposes, we could take an evaluation of deploying OLPC laptops in a third world country, like Nepal, as an example. But any other example would be fine, eg, the hole in the wall initiative in, say, India.

What studies could be done to decide whether this deployment would be good for education with results valid even when after the study is over, the devices will be different in capabilities and price?

Rob van Son