Quick Start#
This notebook follows the same two YAML files as the documentation. Starting from an empty data/ directory, PyHermes downloads and caches the original single-file Quijote FoF group_tab catalogue, projects it into a reusable SFCField, measures an isotropic 2PCF, reloads the saved result, and plots it.
The first run downloads about 33 MB. Later runs reuse ./data/quijote_halos/8000/groups_004/group_tab_004.0.
[1]:
import os
from pathlib import Path
import matplotlib.pyplot as plt
from pyhermes.base.sfc_projection import SFCProjection
from pyhermes.io import Corr2PCFData
from pyhermes.param.parambase import read_param
from pyhermes.theory.corr2pcf import Corr_2PCF
cwd = Path.cwd().resolve()
if cwd.name == "notebooks" and cwd.parent.name == "examples":
examples_dir = cwd.parent
elif cwd.name == "examples":
examples_dir = cwd
elif (cwd / "examples").is_dir():
examples_dir = cwd / "examples"
else:
raise RuntimeError("Run this notebook from the project root, examples/, or examples/notebooks/.")
os.chdir(examples_dir)
Path("./output").mkdir(exist_ok=True)
Path("./figs").mkdir(exist_ok=True)
print(f"Working directory: {Path.cwd()}")
Working directory: /Users/xutengpeng/xutp/PycharmProjects/PyHermes/examples
1. Project the catalogue#
param_sfc_projection.yaml is the single source of truth for the URL, cache path, checksum, box, MRA basis, and output names. Only MPI rank 0 reads or downloads the catalogue.
[2]:
projection_params = read_param("./configs/param_sfc_projection.yaml")
field = SFCProjection(projection_params).run(overwrite=True)
print(field.epsilon.shape, field.weight_normalization, field.field_integral)
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Reading configure file: './configs/param_sfc_projection.yaml'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Input parameter file format is YAML
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Set default parameters of module <base> ...
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.fin.path' from 'empty' to 'https://pyhermes.astroslacker.com/downloads/group_tab_004.0'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.fin.format' from 'bin' to 'fof'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.fin.download.cache_path' from 'empty' to './data/quijote_halos/8000/groups_004/group_tab_004.0'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.fin.download.sha256' from 'empty' to '4a1c6ca4f6747a70e9e552685226ecf5d678c6c97551e2caa7cc3883502eac85'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.threads' from '1' to '2'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.save_particle_data' from 'False' to 'True'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.particle_data_path' from 'empty' to './output/quijote8000_snap004_particles.npz'
10:31:23 - INFO - pyhermes.param.parambase:ParamBase - Default 'SFCProjection.fout_path' from 'empty' to './output/quijote8000_snap004_sfc.pkl'
10:31:23 - INFO - pyhermes.pipeline.pipeline:SFCProjection - SFCProjection runtime configuration: running on 1 MPI ranks with 2 threads per rank
10:31:23 - INFO - pyhermes.pipeline.pipeline:SFCProjection - Preparing SFCProjection input fields ...
10:31:23 - INFO - pyhermes.pipeline.pipeline:SFCProjection - J=8, L=256, box_size=1000.0, phi_resolution=1024, wavelet_mode=db2, wavelet_level=10
10:31:23 - INFO - pyhermes.io.resources:resolve_data_path - Downloading particle data from https://pyhermes.astroslacker.com/downloads/group_tab_004.0
10:31:29 - INFO - pyhermes.io.resources:resolve_data_path - Cached particle data at data/quijote_halos/8000/groups_004/group_tab_004.0
10:31:29 - INFO - pyhermes.io.readers:read_particle_data - Selected input particle format: fof
10:31:29 - INFO - pyhermes.io.readers:read_fof - Reading Quijote FoF halo data from ---> data/quijote_halos/8000/groups_004/group_tab_004.0 <---
10:31:29 - INFO - pyhermes.pipeline.pipeline:SFCProjection - Input particles ready | source=file=path=https://pyhermes.astroslacker.com/downloads/group_tab_004.0 format=fof | particle_count=406728 | catalog_weight_key=None | field_value_key=None | weight_normalization=catalog | catalog_weight_sum=4.067280e+05 | field_integral=1.000000e+00
10:31:29 - INFO - pyhermes.pipeline.pipeline:SFCProjection - Saved particle positions, catalogue weights, and field values to ./output/quijote8000_snap004_particles.npz
10:31:29 - INFO - pyhermes.pipeline.pipeline:SFCProjection - Single process mode
10:31:30 - INFO - pyhermes.pipeline.pipeline:SFCProjection - The time for scaling function: 1.1423 sec
10:31:30 - WARNING - pyhermes.io.funcs:check_fout - Output file './output/quijote8000_snap004_sfc.pkl' already exists and will be overwritten (overwrite=True).
10:31:30 - INFO - pyhermes.io.base:SFCField - Writing SFCField data to ---> ./output/quijote8000_snap004_sfc.pkl <---
10:31:30 - INFO - pyhermes.pipeline.pipeline:SFCProjection - The time for task: 6.8105 sec
(256, 256, 256) catalog 1.0
2. Measure the isotropic 2PCF#
The 2PCF task reads that saved field, uses the analytic uniform reference density, and samples the thin-shell binning window at the separations defined in param_2pcf.yaml. No additional vertex smoothing is applied in this first example.
[3]:
corr_params = read_param("./configs/param_2pcf.yaml")
Corr_2PCF(corr_params).run(overwrite=True)
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Reading configure file: './configs/param_2pcf.yaml'
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Input parameter file format is YAML
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Set default parameters of module <theory> ...
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Default 'Corr_2PCF.sfc_field' from 'empty' to './output/quijote8000_snap004_sfc.pkl'
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Default 'Corr_2PCF.random' from 'None' to 'uniform'
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Using user-provided replacement value for 'Corr_2PCF.binning_window'.
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Using user-provided replacement value for 'Corr_2PCF.sampling'.
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Default 'Corr_2PCF.products' from 'xi' to '['dd', 'dr', 'rd', 'xi']'
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Default 'Corr_2PCF.threads' from '1' to '2'
10:31:30 - INFO - pyhermes.param.parambase:ParamBase - Default 'Corr_2PCF.fout_path' from 'empty' to './output/quijote8000_snap004_2pcf.pkl'
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Corr_2PCF runtime configuration: running on 1 MPI ranks with 2 threads per rank
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Field weight normalization rule | task weight_normalization=catalog | catalog fields are converted to the task rule; derived fields are used as-is except for task='unit'.
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Preparing Corr_2PCF input fields ...
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Sampling: s: n=31, min=0.0, max=150.0, threads=2
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Products: requested=[dd, dr, rd, xi] | computed=[dd, dr, rd, delta_dd, rr] | derived=[xi]
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Binning window: type=shell | mapping=s_to_R | kernel_mode=octant
10:31:30 - INFO - pyhermes.io.base:SFCField - Reading SFCField data from ---> ./output/quijote8000_snap004_sfc.pkl <---
10:31:30 - INFO - pyhermes.io.base:SFCField - epsilon: Shape(256, 256, 256), Min = -3.215e-06, Max = 1.343e-05, Mean = 5.96e-08, Sum = 1
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Corr_2PCF input compatibility check passed.
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Shared required parameters | J=8, box_size=1000.0, phi_resolution=1024, wavelet_mode=db2, wavelet_level=10
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Field weight normalization | field_kind=catalog_field | input=catalog | task=catalog | effective=catalog
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Field vertex 1 ready | source=path=./output/quijote8000_snap004_sfc.pkl | window=no additional window convolution
10:31:30 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Field weight normalization | field_kind=catalog_field | input=catalog | task=catalog | effective=catalog
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Field vertex 2 ready | source=path=./output/quijote8000_snap004_sfc.pkl | window=no additional window convolution
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Random vertex 1 ready | source=uniform random density | window=uniform shortcut | rho=5.96046e-08
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Random vertex 2 ready | source=uniform random density | window=uniform shortcut | rho=5.96046e-08
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Start to calculate 2PCF ...
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Pre-2PCF setup time: 0.3900 sec
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Sampling tasks: total=31, ranks=1, per_rank=31-31
10:31:31 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 0.00%
10:31:33 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 9.68%
10:31:35 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 19.35%
10:31:37 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 29.03%
10:31:39 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 38.71%
10:31:40 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 48.39%
10:31:41 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 58.06%
10:31:43 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 67.74%
10:31:44 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 77.42%
10:31:46 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 87.10%
10:31:47 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 96.77%
10:31:48 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 100.00%
10:31:48 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - 2PCF main loop time: 17.0508 sec
10:31:48 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Main 2PCF loop finished, gathering results on rank 0 ...
10:31:48 - WARNING - pyhermes.io.funcs:check_fout - Output file './output/quijote8000_snap004_2pcf.pkl' already exists and will be overwritten (overwrite=True).
10:31:48 - INFO - pyhermes.io.base:Corr2PCFData - Writing 2PCF data to ---> ./output/quijote8000_snap004_2pcf.pkl <---
10:31:48 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - The time for task: 17.4480 sec
[3]:
<pyhermes.io.corr2pcf.Corr2PCFData at 0x13058c9e0>
3. Load and plot the result#
Task products are read through their public data classes. Here we plot \(s^2\xi(s)\) to make the broad correlation structure easier to see.
[4]:
corr = Corr2PCFData(data_path="./output/quijote8000_snap004_2pcf.pkl")
fig, ax = plt.subplots(figsize=(7.0, 4.6))
ax.plot(corr.s, corr.s**2 * corr.xi, color="#1f77b4", lw=2.0)
ax.axhline(0.0, color="0.35", lw=0.8)
ax.set(xlabel=r"$s\ [h^{-1}\mathrm{Mpc}]$", ylabel=r"$s^2\xi(s)$")
ax.grid(alpha=0.25)
fig.tight_layout()
fig.savefig("./figs/quick_start_2pcf.png", dpi=180, bbox_inches="tight")
plt.show()
10:31:48 - INFO - pyhermes.io.base:Corr2PCFData - Reading 2PCF data from ---> ./output/quijote8000_snap004_2pcf.pkl <---
10:31:48 - INFO - pyhermes.io.base:Corr2PCFData - s: Num = 31, Min = 0, Max = 150
10:31:48 - INFO - pyhermes.io.base:Corr2PCFData - Products loaded: ['dd', 'dr', 'rd', 'delta_dd', 'rr', 'xi']
