Projects five HE sub-tables from the fplida spine, matching the real PLIDA Higher Education data structure. One row per student-year (enrol), per course (course), per unit-of-study (load), per completion (completions), and per student-year HELP debt (help).
Usage
generate_he(
spine = NULL,
seed = 42L,
years = 2005L:2021L,
output_dir = NULL,
format = c("parquet", "csv"),
return_data = FALSE,
chunk_size = 250000L
)Arguments
- spine
Data.frame (from
generate_spine()) or NULL. If NULL, the most recent spine is loaded from the run directory (only the columns needed for HE generation are read, saving memory at large N).- seed
Integer. Random seed for HE-specific generation.
- years
Integer vector. Reporting years to generate (default 2005:2021).
- output_dir
Character or NULL. Base output directory. If NULL, uses
get_data_path().- format
Character. Output format: "parquet" (default) or "csv".
- return_data
Logical. If TRUE, return data.frames in memory; if FALSE (default), write to disk and return metadata only. Tests should pass TRUE.
- chunk_size
Integer. Maximum number of HE spells to process per Rust call for the load table. Default 250000. At 25M people (~6M HE spells), this yields ~25 chunks, each producing ~5-6M load rows (~1.5GB in R). Reduce if memory-constrained.
Value
If return_data = TRUE, a named list with five data.frames.
If return_data = FALSE, a metadata summary.
Details
For large N (>chunk_size participants), the load and help tables are generated in spell-chunks to keep peak memory bounded. Each chunk of spells is processed separately and written to disk before the next chunk, allowing 25M+ person runs on 32GB machines.
Dataset and variable information
The Department of Education student data
website gives information about this dataset. Use dataset_info("HE") for
dataset information. Use variable_info("HE") for variables, sources,
value support, and topic tags.
Examples
if (FALSE) { # \dontrun{
spine <- generate_spine(n = 1000L, seed = 1L)
he <- generate_he(spine = spine, seed = 1L, return_data = TRUE)
str(he$enrol)
} # }