Particle input and conversion

PyHermes separates catalogue I/O from field projection. Every built-in particle reader returns the same small contract: pos with shape (N, 3), size, and any requested one-dimensional particle fields. Once a catalogue has this form, the same arrays can be passed to SFCProjection regardless of the original file format.

The executable Particle I/O notebook downloads the original single-file Quijote group_tab catalogue, reads it directly, and converts all 406,728 haloes to equivalent NPZ and BIN files. The complete notebook remains suitable for a local first run.

Read a group_tab file directly

The public FoF file reports Nfiles = 1 in its header, so it contains the complete catalogue. It can therefore be downloaded, cached, and read without constructing a directory tree or unpacking an archive:

from pyhermes.io import read_particle_data

particles = read_particle_data(
    "https://pyhermes.astroslacker.com/downloads/group_tab_004.0",
    data_format="fof",
    download={
        "cache_path": "./data/quijote_halos/8000/groups_004/group_tab_004.0",
        "sha256": "4a1c6ca4f6747a70e9e552685226ecf5d678c6c97551e2caa7cc3883502eac85",
    },
    redshift=0.0,
    fields={
        "vx": "vel_x",
        "vy": "vel_y",
        "vz": "vel_z",
        "mass": "mass",
        "npart": "npart",
    },
)

For a split catalogue, pass its local .0 file and keep every numerically suffixed sibling beside it. The reader obtains Nfiles from the header and loads the remaining pieces. Catalogue-root input with snapnum remains available for existing workflows.

Reuse the Quick Start input

read_particle_data accepts local paths and HTTP(S) URLs. The Quick Start YAML records the same FoF URL, exact cache destination, and optional SHA256 digest used above:

from pyhermes.io import read_particle_data
from pyhermes.param.parambase import read_param

config = read_param("./configs/param_sfc_projection.yaml")
fin = config["SFCProjection"]["fin"]

particles = read_particle_data(
    fin["path"],
    data_format=fin["format"],
    download=fin["download"],
)

The first call downloads and verifies the group_tab file. A valid file at download.cache_path is reused on later calls, so the example remains offline-friendly after its first run. sha256 is recommended for a public scientific dataset but is not required for a user’s own local file.

Make a portable NPZ catalogue

NPZ is a convenient interchange format for a compact user catalogue. It is constructed locally here to demonstrate the NPZ reader; PyHermes does not distribute a second copy of the example catalogue in this format. Keep positions in one (N, 3) array and give each reusable scalar or vector component a descriptive key:

import numpy as np

np.savez_compressed(
    "./data/my_halo_catalogue.npz",
    pos=pos.astype("float32"),
    vel_x=vel_x.astype("float32"),
    vel_y=vel_y.astype("float32"),
    vel_z=vel_z.astype("float32"),
    mass=mass.astype("float32"),
)

particles = read_particle_data(
    "./data/my_halo_catalogue.npz",
    data_format="npz",
    fields={"mass": "mass", "vz": "vel_z"},
)

When fields is omitted, all arrays other than the position key are exposed. The conversion cells in particle_io.ipynb make the local format contract explicit: array names, dtypes, shapes, and physical units are shown beside the code that creates them.

Describe a raw BIN table where it is used

A headerless binary table does not describe its own column layout. Keep that mapping in the reader call or the task YAML rather than hiding it in a catalogue-specific helper file:

particles = read_particle_data(
    "./data/catalogue.bin",
    data_format="bin",
    dtype="float32",
    ncols=7,
    pos_cols=(0, 1, 2),
    fields={
        "vel_x": 3,
        "vel_y": 4,
        "vel_z": 5,
        "mass": 6,
    },
)

This makes the otherwise implicit binary contract visible next to the code that depends on it.

Native simulation readers

For production data, PyHermes also reads legacy Gadget snapshots, Gadget HDF5 snapshots, Gadget FoF catalogues, and Quijote/Pylians group_tab files or directories. These readers preserve format-specific unit controls while returning the same shared particle dictionary. See Input and output for their complete parameters.

Source catalogue versus projection companion

The source catalogue may contain velocities, mass, particle counts, and other scientific columns, whether it is FoF, NPZ, BIN, or another supported format. By contrast, SFCProjection.save_particle_data writes a compact companion containing only pos, catalog_weight, and field_value. The companion records exactly what entered one projection and supports particle-centred estimators; it is not intended to replace the richer source catalogue.

With the reader boundary understood, continue to Build an SFCField to build number, weighted, and physical SFCField objects.