Tabulated Data🔗

Tabulated data contain participant-level summaries for the majority of HBCD behavioral/phenotypical instruments (as well as tabulated pipeline derivatives). Tables follow the BIDS organizational structure so data from different sources can be linked by participant ID and visit number. Files are stored in rawdata/phenotype/:

hbcd/
└── rawdata/ 
    └── phenotype/ 
        ├── sed_basic_demographics.*        # Basic Demographics
        ├── par_visit_data.*                # Visit Level Data
        ├── bio_biosample_{nails|urine}.*   # Toxicology
        └── [instrument_name].*             # Instrument Data

File Formats🔗

Each table is available as:

  • TSV/CSV: plain text files for easy inspection and broad compatibility
  • Parquet: compressed files optimized for efficient analysis in Python and R (see details)
  • Shadow matrix: a companion file that records why individual values are missing

TSV/CSV vs. Parquet🔗

 WARNING: Incorrect Data Types Inferred for CSV/TSV

CSV and TSV files do not contain an embedded data schema. Because column metadata are provided separately, import tools in Python and R may infer some column types incorrectly. For example:

  • Categorical codes stored as strings, such as "0" and "1" for “No” and “Yes,” may be imported as numbers.
  • Numeric columns may be imported as text when missing values are represented as n/a, as required for TSV files by the BIDS specification.

It is therefore critical that you specify column types during import, particularly data type (type_data), using the accompanying metadata. See NBDCtools for available functions to automate this process (e.g. read_dsv_formatted() for R users).

One of the key difference between these file types is that TSV/CSV file types store metadata in accompanying .json files, whereas Parquet stores metadata directly in the file, reducing import errors and improving performance for large datasets. Review the table below to choose the optimal format for your needs:

Format Best for Advantages Limitations
TSV/CSV Quick inspection and spreadsheets
  • Easy to open
  • Widely compatible
  • Slower for large files
  • Metadata stored separately
  • No selective column loading
Parquet Analysis in Python or R
  • Fast and compact
  • Embedded metadata
  • Preserves data types
  • Supports selective column loading
  • Not easily viewed in Excel
  • Not currently supported by BIDS
 

Loading parquet files in Python (polars or pandas module):
  # Using `polars` module [RECOMMENDED]:
  import polars as pl
  parquet_df = pl.read_parquet("path/to/file.parquet")

  # Using `pandas` module:
  import pandas as pd
  parquet_df = pd.read_parquet("path/to/file.parquet")

Loading Parquet file in R (arrow package):
  # Using `arrow` package:
  library(arrow)
  parquet_df <- read_parquet("path/to/file.parquet")

Shadow Matrices for Missing Data🔗

Every TSV or Parquet file in rawdata/phenotype/ has a corresponding shadow matrix in the same format. The shadow matrix has the same structure and column names as its data file but records why values are missing. For example, non-response codes such as 999 (“Don't Know”) and 777 (“Decline to Answer”) are converted to blank cells in the main data file. Their meaning is preserved in the corresponding shadow matrix. For each cell:

  • Data value present → shadow matrix cell is blank.
  • Data value missing → shadow matrix cell contains the reason for missingness.

Common shadow matrix values include:

  • Decline to Answer- participant declined to answer a question
  • Don't Know- participant did not know the answer
  • Missed Visit- participant did not attend a visit
  • Missed Instrument- participant did not complete assessment
  • Logic Skipped- question skipped due to branching logic
  • Unknown Missing- reason for missing value unknown and/or instrument was not administered (check against the Administration field included for instruments)

The following domains/instruments have additional unique shadow matrix values used where applicable:

Table(s) Unique Shadow Matrix Values [+Variable Name If Specific]
BioSpecimens (All)
  • "Please refer to corresponding categorical field for more details"
Basic Demographics
  • "Child's DOB not reported or available for participant" [{gestational|mother}_age_delivery]
  • "Missing Information From Ripple" [ACS-derived fields]
Visit Level Data
  • "Data not available for participants at this timepoint"
  • "No candidate age for V01" [candidate_age]
  • "Gestational Age at Administration is only at V01 and not calculated for V02 onwards" [gestational_age]

Why Use Shadow Matrices?🔗

Separating missingness reasons from the primary data:

  • Prevents placeholder codes such as 777 or 999 from being interpreted as valid numeric values
  • Keeps column data types consistent
  • Preserves information about non-response without cluttering the main dataset

In some analyses, the reason a value is missing may itself be meaningful. For example, researchers may want to examine how often participants report that they do not understand a question. In these cases, missingness information can be joined back to the primary data via the methods below.

# Example 1: Load CSV/TSV and corresponding shadow matrix and add '_missing_reason' columns for missing values.
import pandas as pd
import os

def load_data_with_shadow(data_path, shadow_path):  
    # Detect delimiter from file extension and load data
    def get_delimiter(path):
        ext = os.path.splitext(path)[1].lower()
        return "\t" if ext == ".tsv" else ","
    data = pd.read_csv(data_path, delimiter=get_delimiter(data_path))  
    shadow = pd.read_csv(shadow_path, delimiter=get_delimiter(shadow_path))

    # Annotate data with non-empty missingness reason columns (excluding participant_id, session_id) in shadow matrix 
    for col in data.columns[2:]:  
        if col in shadow.columns:
            if not shadow[col].isna().all() and not (shadow[col] == '').all():
                data[f"{col}_missing_reason"] = shadow[col]
    return data

# Example usage:
df = load_data_with_shadow("data.tsv", "shadow_matrix.tsv")
# Example: View reasons for missing data for a given column/variable in the data file 
df[df["<COLUMN NAME>"].isna()][["<COLUMN NAME>_missing_reason"]]


# Example 2: Using NBDCtools Python package
# install R backend with `NBDCtools` is required to run this code
from NBDCtools import create_dataset
create_dataset(
    dir_data="path/to/data",
    study="hbcd",
    vars=["var1", "var2", "var3"],
    tables=["table1", "table2"],
    bind_shadow=True
)

library(NBDCtools)
create_dataset(
  dir_data = "path/to/data",
  study = "hbcd",
  vars = c("var1", "var2", "var3"),
  tables = c("table1", "table2"),
  bind_shadow = TRUE
)