Understanding the mathematical formula to calculate lift and thrust is still a long way from designing a 747 airplane, and the U.S. Department of Education is trying to get students to cross that bridge with the development of a new way to gauge how well they both understand and apply technology and engineering principles.
The National Center for Education Statistics is nearing completion of a 15,000-student pilot test鈥攖he largest in the history of the National Assessment of Educational Progress鈥攖o craft a new technology- and engineering-literacy test, or the TEL.
鈥淲hat we鈥檙e talking about here is trying to put the 鈥楾-E in STEM,鈥� said NCES Commissioner Sean P. 鈥淛ack鈥� Buckley, referring to the common term for science, technology, engineering, and mathematics. 鈥淲e鈥檝e been assessing the [science and math] for some time, but it鈥檚 been much harder to figure out the framework for an actual, practical, functional field assessment for technology and engineering components.鈥�
The current pilot, on track to be finished by the end of this month, targets 8th graders. In 2014, a final version of the test is slated to be administered to a nationally representative sample of 20,000 such students, with results expected in 2015. Eventually, the TEL will cover the 4th, 8th, and 12th grades.
鈥淭his is really important, and I鈥檓 glad to see it,鈥� said Adam Gamoran, a member of the National Board of Education Sciences, the Education Department鈥檚 research advisory group, and the director of the Wisconsin Center for Education Research at the University of Wisconsin-Madison.
While a few curricula, such as the International Baccalaureate program, include engineering and technology courses, Mr. Gamoran noted there is little research on how well even today鈥檚 鈥渄igital native鈥� generation understands technology and engineering.
鈥淔rom my vantage point as a sociology researcher, I suspect there remains a substantial digital divide鈥攖hat children from different backgrounds will have vastly different experiences with these questions about technology,鈥� he said. 鈥淭his will provide evidence of something we have many suspicions about but virtually no evidence.鈥�
New Direction
The TEL represents a significant shift for the battery of tests commonly dubbed 鈥渢he nation鈥檚 report card.鈥� It will be NAEP鈥檚 first entirely computer-based test and the first to use a majority of interactive scenario-based questions.
International assessments, in particular the Program for International Student Assessment, already gauge proficiency in more comprehensive and applied-science questions, which is in part why experts say American students鈥� performance tends to lag behind that of students in other countries on PISA.
More than 2,000 engineering and technology professionals from around the United States contributed to the development of the test鈥檚 framework, which covers three interconnected areas: the design process and principles of dealing with technology in daily life; information and communication-systems technology, such as computer networks and mobile devices; and the social and ethical implications of technology鈥檚 effects in the natural world.
鈥淲e鈥檙e pretty good at assessing students in science, but how do we assess the difference between a scientific solution鈥攕ome sort of global, perfect universal solution鈥攁nd engineering, which is a lot more about trade-offs and constraints in a given situation to get a solution that works?鈥� Mr. Buckley said.
Testing 鈥楽cenarios鈥�
The solution, he said, is to include 鈥渕uch more complex and higher-order-thinking items鈥� than have previously been used in NAEP.
Roughly 20 percent of the test鈥檚 questions will cover concrete facts and information. The rest will use a new kind of question, which requires students to interact in engineering or technology 鈥渟cenarios,鈥� to apply ways of critical thinking and problem-solving that are associated with engineering.
Each scenario is 10, 20, or 30 minutes long and gauges a student鈥檚 mastery of engineering practices, such as systematically using technology, tools, and skills to solve a problem or achieve a specific goal, or using technology to communicate and collaborate with a team and consult experts.
For example, a student may be asked to collaborate with a simulated 鈥渂oss鈥� via videoconference to improve the consumer 鈥渓ife cycle鈥� of a toaster.
鈥淣AEP is not alone in the world of large-scale standardized assessment in trying to come up with ways to better assess how people work collaboratively,鈥� Mr. Buckley said.
Moreover, the test will also begin to use student-activity data to report and evaluate how the student solves each problem. For example, NAEP鈥檚 writing test collects information about how students used the in-test word-processing software to check spelling and edit sentences, but does not use that information to evaluate students鈥� performance. In this assessment, a student might get more points for answering a problem efficiently and making the best use of the tools available.
鈥淭he intent is to be much more authentic and closer to a real project,鈥� Mr. Buckley said.
During the test鈥檚 administration, the NCES will also collect data on students鈥� access to technology at home and teachers鈥� use of technology in the classroom.
