Data Critique

Overview

The dataset analyzed in this project is the SEDA Administrative District Annual Subgroup Test Score Dataset (seda_admindist_annualsub_cs_2024.2), produced by the Stanford Education Data Archive (SEDA). This dataset contains standardized test score data for United States public and charter school students in grades 3-8 from 2008 to 2024. Rather than detailing individual student results, these scores are reported for each administrative district and categorized annually by mathematics and reading language arts (RLA). Moreover, these scores are averaged across grades and disaggregated by all students, race, gender, and economic status (economically disadvantaged [ECD] and non-ECD) allowing us to compare scores across time and demographics on a national scale.

Timeframe and Focus

By focusing on data from 2019 through 2024, this project uses 2019 as a pre-pandemic baseline, 2022 as the first reliable post-pandemic testing year, and 2023-2024 as indicators of early and later recovery in student learning. Comparing outcomes over these years reveals relationships among learning loss, pandemic school disruptions, and differences in recovery rates across the aforementioned demographics. This disaggregation highlights broader educational gaps, exhibiting which populations experience the largest declines in test performance and how these setbacks persist in later years. Overall, seda_admindist_annualsub_cs_2024.2 provides insight into educational patterns associated with the COVID-19 pandemic and whether the pandemic exacerbated pre-existing racial and socioeconomic inequalities.

Dataset Limitations

Although comprehensive, seda_admindist_annualsub_cs_2024.2 has significant limitations. Firstly, it also does not include data on individual students, so we cannot track the same students over time or measure how COVID-19 affects them personally. The dataset also lacks contextual information that further explains distinct test score changes, such as remote days, attendance absences, technological disadvantages, teacher shortages, funding, tutoring programs, family income shocks, or mental health. While the dataset can show correlations between time and scores, it cannot determine which specific COVID-related conditions cause changes in test results. 

Methodology

SEDA 2024 is a standardized set of estimates of academic achievement across U.S. school districts. To generate these measures, the SEDA team takes standardized test proficiency counts from state-reported data and applies statistical models to estimate average test score levels for each district, subject, and grade. The results are then linked to the National Assessment of Educational Progress (NAEP) to place all district estimates on a nationally comparable scale. 

Original Sources of Data

The original sources for the COVID-19 dataset come primarily from U.S. state education agencies that report standardized test results at the district level. For pre-pandemic and early pandemic years, these data are drawn largely from the U.S. Department of Education’s EDFacts database, which aggregates state-reported assessment outcomes. For more recent years, when EDFacts coverage is limited, the dataset incorporates test data from states’ public releases compiled through the Zelma data system. To ensure comparability across states and time, these district-level results are linked to the National Assessment of Educational Progress (NAEP), which provides a common national scale. Additional contextual information in related analyses is drawn from sources such as the Common Core of Data and the American Community Survey.

Funding and Contributors

The construction of the SEDA dataset was supported by a grant from the Gates Foundation. The source data used to construct the estimates come from Zelma, the National Center for Education Statistics (NCES), and the National Assessment Governing Board. State assessment data files are provided by Emily Oster, Clare Halloran, and the Zelma team. Stanford Education Data Archive data are produced by Sadie Richardson, Sofia Wilson, Julia Paris, Ishita Panda, Amelia Bloom, Nahian Haque, and Jackson Kinsella. Research partnerships with Tom Kane, Dan Dewey, Doug Staiger, and Harvard University’s Center for Education Policy Research.

Data Gaps

The data set excludes data from the years 2020 and 2021, as reporting during that period lacked sufficient detail. This is largely due to COVID-19 related test cancellations and disruptions from remote learning. Additionally, not all states have reported data for all demographic subgroups; certain subgroups at certain schools are missing reported scores in one or both subjects. This is due to states’ varying suppression rules and reporting thresholds; some states drop certain cells because of small sample sizes or concerns about insufficient data quality. Finally, the dataset does not include individual student-level data. All of the reported information consists of aggregated counts at the district, state, or subgroup level rather than individual student scores.

Demographic and Performance Variables (Ontology of the Dataset)

The dataset organizes information around the ontology of racial and economic background, using variables such as state, year, race, and multiple measures of economic status, including household income, parental education, parental unemployment, single motherhood, and receipt of federal benefits like SNAP or free lunch. Race categories are mostly comprehensive, though mixed-race households are aggregated into a single category, and some economic measures apply to the household as a whole while others apply to individual parents. These variables are examined alongside student performance on standardized math and reading tests, allowing analysis of how racial and economic factors correlate with educational outcomes. However, if this dataset were the only source, significant information would be missing: it contains no data on single-father households, does not disaggregate mixed-race students, excludes non-primary test scores, and only reports math and reading outcomes. As a result, the dataset presents a constrained view of households and student achievement shaped by its choices in categorization and measurement.