disformat . Offline sync

Ensuring fair, equitable assessment

The Programme for International Student Assessment - PISA - is a globally recognized benchmark that evaluates 15 year old students worldwide in mathematics, reading, and science.

Conducted across countries with diverse technological infrastructures, it faces a critical challenge: Ensuring a standardized and equitable assessment experience for all students, regardless of local resource limitations.

The Sync Tool was developed to deliver reliable assessment in environments with limited or no internet connectivity. It enables the download of the assessment content, including test packages, rostering data, and associated metadata, to a local device prior to test administration.

Once assessments are completed, the tool manages the upload and synchronization of student results, response data, and test session logs back to the central server when connectivity is restored.

Challenge - Business requirements

To eliminate disparities caused by technological and infrastructural differences across regions worldwide, PISA needed a solution that would:

Ensure offline assessment: Students in areas with limited or no internet connectivity should experience the same testing conditions as those in highly connected environments.

Provide reliable data synchronization: Enable seamless and efficient transfer of student data between offline and online systems.

Standardize diverse local infrastructure: Operate effectively across different schools and test centers, with diverse hardware and network capabilities.

The goal was to enable offline test delivery and automated synchronization, allowing test center managers to conduct PISA assessments effectively without technological disparities that could potentially impact student performance.

Interviews - Qualitative research

I collaborated closely with customer-side project managers and my UX peers to better understand the needs of real users. Together, we coordinated and conducted 3 interviews with actual Test center managers from 3 different countries - Vietnam, Moldova and Turkey - that actively participated in PISA 2022.

These sessions provided valuable qualitative insights into their responsibilities, technical challenges, and workflow while conducting PISA assessments. The resulting user persona is based in their experiences and provided basis for key UX decisions throughout the design process.

Test center manager - User persona

Could be a school director, teacher, or IT staff member and is responsible for managing and executing the PISA assessment protocol locally.

Generally have basic to moderate tech proficiency, comfortably using standard digital tools like email, file systems, and basic software installations. However, they may lack experience with complex systems, advanced settings or troubleshooting, and would require intuitive design to perform more technical tasks.

Primary Responsibilities
- Prepares the test environment and ensures equipment is ready, setting up devices, software, and physical space for assessments.
- Manages the student roster and test sessions, ensuring that the right students are assigned to the correct test sessions.
- Oversees the assessment process from start to finish, ensuring all assessment data is successfully transferred to the national or central PISA system.
- Verifies if student responses and logs are uploaded accurately after testing.
  "...used to double and sometimes triple-check everything."
Main Goals
- Deliver assessments accurately and reliably, and wants it to run without technical issues or interruptions.
- Operate the system with minimal need for support or troubleshooting.
  "...something that just works, without needing IT help..."
- Have peace of mind that student data is safe. Needs confirmation and assurance that no data has been lost or corrupted, before and after the assessment.
Pain Points
- Anxiety around losing student results during sync.
- Fear that technical issues could result in irreversible data loss.
- Lack of clear feedback when something goes wrong. Uncertainty about what to do when errors appear or process stalls, lead to frustration or premature interruption.
  "Refreshing - the screen - feels like defusing a bomb."

Technical solution - in a nutshell

A software package will be provided, as a Virtual Machine (from now on, VM, specifically Oracle VirtualBox, a free, open-source solution) together with installation instructions and hardware requirements to the Test centers.

A VM can transform any standard computer into a local server, once installed, and other computers can then connect to the local server, allowing students to take the test, via a local network.

Before and after the assessment, the local server application will need internet access to download all necessary assessment data and content, then upload collected responses to the regular online instance.

These steps will happen through the Sync Tool and the setup ensures that all students can benefit from the same experience and functionalities as the online assessment version, even in fully offline environments.

Sync Tool - Before the assessment

Test center manager need to visit a location with internet access to initiate a one-time synchronization. Using the built-in Sync Tool, they authenticate with their credentials and are able to access their specific assessment data, to download everything needed for the test: download student accounts, organization structure, test sessions, and relevant test content.

Once synced, the VM can operate entirely offline, delivering the full test experience while capturing responses, reports, timestamps, interaction events, and diagnostic data.

Sync Tool - After the assessment

Test center manager need to visit a location with internet access and the Sync Tool is used to upload all collected data back to the central server.

From there, the data is accessed just like it would from an online session, ready for scoring, analysis, and review.

The offline and online data flows are indistinguishable in quality or format, ensuring a seamless integration into PISA assessment infrastructure.