Current federal legislation (NCLB)
decrees that by 2013 educators will obtain “technology literacy” (TL) for all
students by the time these students complete the 8th grade. The U.S.
Department of Education has mandated that states must report annually,
beginning in 2008, on the percentage of students who have achieved TL by the
end of the 8th grade[1]
— leaving interpretation and implementation of this requirement to the
discretion of each state. To comply with the federal mandate, the Colorado
Department of Education has ordered school districts to report by fall 2008 the
percentage of their students who have achieved 8th grade TL, declaring
a common definition and standards for TL but ceding to districts the authority
(and responsibility) for determining the proficiency levels that indicate
attainment of TL and for devising the mechanism for determining attainment of
said literacy. (See Appendix C — CDE Statement of Technology Literacy Reporting
Requirement.)
In response to these federal and
state expectations,
Now, Colorado Power Results grant
funds create the opportunity for the state’s educators to shift from muddled
NCLB compliance to transformative support for 21st century learning. This
proposal presents a multi-year project designed to achieve that transformation.
The project outlined in this
proposal, collaboratively conducted by a consortium of school districts and
BOCES[4],
would accomplish the following:
·
Determine a common
definition, standards, and proficiency criteria for TL;
·
Develop or adapt an
assessment instrument that addresses the determined standards and
proficiency criteria;
·
Develop common
procedures for delivery of the assessment that will apply to all districts
that use the assessment instrument (“participating districts”);
·
Establish assessment
results reporting mechanisms for all participating districts;
·
Facilitate development
of assessment analysis practices and application of assessment analyses
to improved curricula and instructional design;
·
Work to incorporate
assessment and acquisition of TL into
The project builds on the experience of consortial partners and other districts in devising and implementing a limited-response assessment that addresses three of the six CDE-endorsed TL standards (Research & Information Fluency, Digital Citizenship, and Technology Operations & Concepts). Investing at least $120,000 in staff time, assessment design consultants[5], technology, and other resources, these districts collaborated from September 2007 through May 2008 to create an assessment that would be easy to administer and would provide reliable results to inform instructional planning as well as fulfill NCLB and state reporting requirements. (See Appendix D — Existing Consortial Technology Literacy Assessment Project.) While that work provides a solid foundation for the proposed project, much more remains to be accomplished in creating and using a comprehensive TL assessment. Additional work includes:
· Benchmarking the CDE-endorsed TL standards to grade levels;
· Developing an assessment for the more complex and sophisticated elements of TL (Creativity & Innovation; Communication & Collaboration; Critical Thinking, Problem Solving, & Decision-Making)
· Developing an assessment that emphasizes performance-based demonstrations of student proficiency;
· Making the assessment instrument available in formats that maximize usability for all interested school districts, with a preference for online delivery;
· Establishing a data reporting system and procedures;
· Designing and delivering professional learning related to implementation of a TL assessment and the use of assessment data to improve instruction;
· Incorporating TL assessment into instructional design and practice.
The additional work outlined above comprises the core of this proposal. Further explanation and elaboration of the intended project outcomes are provided in the paragraphs that follow.
Common definition, standards, and proficiency criteria
— In order to assess (and report) TL,
The 2007 ISTE-NETS standards, which CDE modified and endorsed in August 2007, provide valuable common ground for linking technology and information literacies[6], re-orienting the conceptualization of these literacies as instrumental rather than terminal.[7] Significantly, these standards reflect the growing national agreement that technology is a means rather than an end, and, correlatively, that TL should be understood as a way to support learning and working rather than an isolated accumulation of skills and knowledge.[8] The six CDE-endorsed and ISTE-NETS categories (Creativity & Innovation; Communication & Collaboration; Research & Information Fluency; Critical Thinking, Problem Solving, & Decision-Making; Digital Citizenship; and Technology Operations & Concepts) reflect this growing agreement, and portend a shift toward Information Age learning.
Accordingly, guiding principles for a common understanding of TL include:
· Emphasis on information and communication skills;
· Emphasis on application of tools to learning and productivity outcomes;
· Connection of TL to student achievement in content areas;
· Emphasis on broad tool-use and problem-solving strategies rather than narrow mastery of particular technologies;
· Focus on enduring skills, understandings, and dispositions that transcend short- and medium-term technology developments.
Significantly, the CDE-endorsed modification of the ISTE-NETS standards adds elements of technology literacy not addressed by the national standards — that of “design-build” technologies and processes (e.g., the creation of solutions, process, design, and products).[9] This addition adds an important element to TL that presents interesting performance-based assessment design challenges.
An assessment instrument — The project will develop
or adapt a TL assessment that addresses all six state-endorsed TL standards and
follows the guiding principles described above. The developmental process will
include review and, potentially, adaptation of assessment instruments already
developed (or in the process of development) by other organizations with proven
assessment expertise. To the extent practicable, the assessment instrument will
be performance-based. The assessment will be available free to all
Experience gained through the consortial work already conducted member districts of this project, in creating the existing 3-standard TL assessment, lays the groundwork for the extensive validity and reliability studies that must be conducted to create an instrument that produces usable results. Assessment staff within the districts guided such studies on the existing assessment; that work included analysis of questions for bias and age-appropriate language, placement and order of questions, misleading answers, non-indicative answers, and test-delivery of the assessment to representative populations (considering not only demographic, language, and ability diversity but also anticipated technology literacies of the populations to be assessed). Through this process the districts’ assessment staff acquired deeper understanding of the particular challenges presented by TL assessments, and their increasing expertise will guide the proposed project. In addition, grant funds make it possible to consult with other assessment design experts who have particular knowledge in this area. (See Appendix D — Existing Consortial Technology Literacy Assessment Project.)
The project will also explore the
value and feasibility of assessing students in the 4th and 8th
grades — the earlier assessment serving a formative function that helps
districts design and adapt curricular and instructional plans.
Common procedures for delivery — Although district flexibility in timing and
manner of implementation will be a priority, core implementation procedures
will be standardized in order to obtain reliable, usable results. Development
of these procedures will seek to minimize the burden of implementation on
The project-produced assessment will be available for online delivery (using a single platform, preferably one, like Moodle[10], that is free or inexpensive), in order to make reporting of results more efficient and uniform. Variations in the mode of delivery will accommodate districts' divergent technology capacities. Alternatives to online delivery (e.g., making some or all parts of the assessment available in non-digital environments, or facilitating administration by individual districts through their LANs, with uploaded reports following implementation) will be explored to ensure maximum feasible accommodation.
Reporting mechanisms — A primary value in conducting a common assessment is that it facilitates extensive analysis of the results — including, for example, demographic and item analysis of assessment results. Online delivery of the assessment will enable automated and uniform reporting of results, which is one reason online delivery will be the primary method of implementation. Whatever the assessment delivery modality, the project will develop common reporting procedures, tools, and platforms to ensure that all results reported from districts that use the project-developed assessment are consistent, and can be collectively and meaningfully analyzed. The project consortium will coordinate with CDE to establish common proficiency ratings, reporting procedures, and a state database that makes assessment results easily and promptly available to all education stakeholders. Individual student identities will be protected in the same manner as they are in CSAP administration.
Assessment analysis practices — The primary application of TL assessments should be to enable Colorado educators to improve learning and curriculum design — not just with respect to students’ acquisition of TL but in content-area learning as well. Professional learning related to the assessments will emphasize analysis of assessment results (e.g., use of data-driven dialogue) that guides teaching and learning practices.
Work to incorporate assessment and acquisition of TL into
All products developed through this project will be
available free to all school districts in
The member districts of the project
consortium have more than 150 high need schools, based on the CDE
criteria. Full-scale use of the assessment, of course, will apply to all of
these schools. In addition, pilot assessment efforts will focus on high need
schools in order to ensure that the assessment serves not only their capacities
in delivering the assessment but their needs in using the assessment to improve
student learning.
The Technology Literacy consortium has conducted its own research, in the context of its existing work creating a TL assessment; and project work will apply these findings, while continuing to pursue additional research. In addition, research nationwide on TL and other related assessments, ranging across age cohorts and subject areas, have demonstrated (1) the viability and value of performance-based assessments for measuring the skills and knowledge described by the CDE-endorsed TL standards, and (2) the feasibility to deliver such assessments online (and obtain reports of the assessment results). Significant examples of these efforts include:
· Student Tool for Technology Literacy — Florida Department of Education — assesses the six ISTE/NETS TL standards[11];
· iSkills — Educational Testing Service — assess the Information and Communication Technology (ICT) literacy of students seeking admission to four-year colleges[12];
· College Learning Assessment — Council for Aid to Education — assesses information, communication, and problem-solving skills of four-year college students[13]
· TechYES — Generation Yes — assesses the six ISTE/NETS TL standards through student projects[14];
· Learning.com — may assess the six ISTE/NETS TL standards (though it appears to emphasize technical operations), primarily through a limited response test but partly through simulations[15]
All of the assessments listed here have substantial research
support, undertaken by the organizations creating the assessments. This
research informs the proposed TL assessment project, especially since the
organizations involved, along with others, will be sought as partners in the
project. The project’s assessment development plan makes examination of other instruments, especially
In addition, the project is informed by the research already conducted by the work of the consortial-member districts that have created the existing 3-standard TL assessment. That research included extensive validity and reliability studies, conducted under the guidance of those districts’ assessment directors. (See Appendix D — Existing Consortial Technology Literacy Assessment Project.)
[1] For the sake of brevity and simplicity, further references
to the expectation that students will attain technology literacy by the end of
the 8th grade will be stated as “8th grade TL”. In fact,
the imagined literacy could be attained by a student (and assessed) at any time
prior to the completion of the 8th grade.
[2] A sample of district assessment strategies is shown in
Table 1 at the end of this document.
[3] www.iste.org/Content/NavigationMenu/NETS/ForStudents/2007Standards/NETS_for_Students_2007.htm
(accessed 5-12-08).
[4] Consortium membership includes Centennial BOCES,
[5] The consortium’s primary consultant was Bernajean Porter,
a nationally recognized expert in technology literacy and assessment design.
See www.bjpconsulting.com/index.html
[6] The integration of technology and information literacies
was initiated by CDE’s Educational Technology and Information Literacy (ET-IL)
initiative (begun in 2002). Integration is also reflected in the more common
national (and international) label of “ICT” (Information and Communications
Technology) literacy. (See the ICT Digital Literacy webpage, http://www.ictliteracy.info/,
and the Educational Testing Service’s iSkills test webpage,
www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=1ebb0e3c27a85110VgnVCM10000022f95190RCRD&vgnextchannel=34b4a79898a85110VgnVCM10000022f95190RCRD.
[7] http://www.cde.state.co.us/edtech/StandardsForStudents.htm
[8] See “Tests of Tech
Literacy Still Not Widespread Despite NCLB Goals,” in Education Week, 1-30-08, pp. 1, 12; www.edweek.org/ew/articles/2008/01/30/21techtests.h27.html
(accessed 6-8-08).
[9] See
[10] The Moodle platform is used here as an example only, not a
recommendation.
[11] See www.flinnovates.org/sttl/default.htm
(accessed 6-8-08) for a description of the
[12] See
www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=159f0e3c27a85110VgnVCM10000022f95190RCRD&vgnextchannel=e5b2a79898a85110VgnVCM10000022f95190RCRD
[13] According the Generation Yes promotional material,
“TechYES encourages all students to complete technology projects that are
creative and personally involving. The projects can also meet requirements for
core curriculum classes or community service. These projects are the basis for
the TechYES evaluation and certification.” See
www.cae.org/content/pro_collegiate.htm.
[15] Publicity materials from Learning.com
claim, “When taking the online test, students interact with assessment content
in ways that allow them to demonstrate their proficiencies. Often, they must
perform actions via simulations, rather than pick answers from among multiple
choices. Thus, students must be able to format a paragraph, apply a spreadsheet
formula, or conduct a database search. And they must demonstrate durable skills
via generic menus and commands, not through brand-specific memorized
shortcuts.” See Arizona Pioneers Statewide Measurement of Students’ Technology
Literacy Skills, from Learning.Com, http://www.learning.com/casestudies/arizona.htm
(accessed 6-7-08).