NAPLAN Online 2014 Development Studies (2014)

Australian Curriculum, Assessment and Reporting Authority (ACARA)

Tom Lowrie

ACARA has recently developed a new tailored (multi-stage) test design of the National Assessment Plan for Literacy and Numeracy (NAPLAN) that will be administered digitally. This new test is tailored to a student’s ability level according to his/her responses, with five possible pathways that students could be taken through. In addition, given the online test format, ACARA has developed technically enhanced items—a design that is possible only through a digital mode.

ACARA has funded our research team to conduct two studies:

  1. Cognitive interviews research activity 2a: Perceived difficulty of challenging numeracy items: To examine one branch of the multi-stage test pathway (Testlet F, challenging items); and
  2. Cognitive interviews research activity 3: Technically enhanced items (Numeracy): To examine the technically enhanced items for Numeracy.

Through in-depth cognitive interviews with primary and secondary students, our research will provide insight into the cognitive and behavioural engagement of students with the new items, and students’ knowledge, thinking skills and strategy use. Specifically, we will identify how students interact with the new design, how they comprehend and solve the new items, and what strategies they employ.

These findings are intended to help ACARA further refine and improve its curriculum and assessment program in a digital mode.

Study 1: Perceived Difficulty of Challenging Numeracy Items

This study examines one branch of ACARA’s new multi-stage test design for NAPLAN. Specifically, our team will look at the most challenging branch of items, Testlet F. This investigation attempts to:

  1. Establish the extent to which the proposed challenging items in Testlet F provide adequate testing context for highly capable students;
  2. Examine the performance of such students on Testlet F of the tailored test; and
  3. Monitor the students’ knowledge, thinking skills and strategy use, and how these relate to the intended assessment outcomes envisaged by the test developers

We use in-depth cognitive interviews with students from Grades 3, 5, 7 and 9 to document how students access and engage with challenging mathematics items, what design elements most impact student access and performance, what performance characteristics are common among these students, and what strategies they use to solve the challenging items.

Study 2: Technically Enhanced Items for Numeracy

This study tests the effectiveness of technically enhanced items for numeracy within the new NAPLAN design. The two objectives of the project are to:

  1. Investigate the cognitive and behavioural engagement as students interact with technically enhanced numeracy items proposed for NAPLAN online; and
  2. Monitor and assess the knowledge, thinking skills and strategies students possess and utiltise when solving these technically enhanced items.

With in-depth cognitive interviews of students from Grades 3, 5, 7 and 9, we will identify how and what design elements most impact student access and performance, as well as what problem-solving processes are most effective for solving these new items. The study also attempts to clarify which items more adequately assess numeracy knowledge and skills not easily assessed by traditional test items.