Run all in one
Of course users can run all code in just one python script.
For example to calculate 3PCF, we create parameter file param_multi.json, which contains all the parameters to calculate:
{
"Convols": {
"J": 9,
"fin": {
"path": "https://pyhermes.astroslacker.com/_downloads/906e0695649e3634a5fe8081b9ab2086/quijote10000.bin",
"format": "generic",
},
"fout_path": "./multi_test.npy",
"SampRate": 1024,
"SimBoxL": 1000,
"window": {
"type": "shell",
"R": 5,
},
"wavelet_mode": "db2",
"wavelet_level": 10,
"bandwidth": 1,
"threads": 2
},
"Corr_2PCF": {
"deltac_in_path": "", //not necessary, cause will load from argv
"fout_path": "./multi_test_2pcf.txt",
"threads": 20,
"R1": 1.0,
"R2": 150.0,
"xi_num": 200
},
"Corr_3PCF": {
"fin": {
"path": "", //not necessary, cause will load from argv
"format": "" //not necessary, cause will load from argv
},
"deltac_in_path": "", //not necessary, cause will load from argv
"corr2pcf_in_path" : "", //not necessary, cause will load from argv
"fout_path": "./multi_test_3pcf.txt",
"NStheta": 20,
"R1": 20.0,
"R2": 40.0,
"rot_num": 3
}
}
Not a fan of JSON? No worries! PyHermes happily supports both JSON and YAML for your parameter files. Pick your favorite and get started :)
For instance, param_multi.yaml:
Convols:
J: 9
fin:
path: "https://pyhermes.astroslacker.com/_downloads/906e0695649e3634a5fe8081b9ab2086/quijote10000.bin"
format: "generic"
fout_path: "./multi_test.npy"
SampRate: 1024
SimBoxL: 1000
window:
type: "shell"
R: 5
wavelet_mode: "db2"
wavelet_level: 10
bandwidth: 1
threads: 2
Corr_2PCF:
deltac_in_path: "" # not necessary, cause will load from argv
fout_path: "./multi_test_2pcf.txt"
threads: 20
R1: 1.0
R2: 150.0
xi_num: 200
Corr_3PCF:
fin:
path: "" # not necessary, cause will load from argv
format: "" # not necessary, cause will load from argv
deltac_in_path: "" # not necessary, cause will load from argv
corr2pcf_in_path: "" # not necessary, cause will load from argv
fout_path: "./multi_test_3pcf.txt"
NStheta: 20
R1: 20.0
R2: 40.0
rot_num: 3
this is completely equivalent to the JSON format file param_multi.json above!
Now lets create the python script, run_multi.py:
from pyhermes.base.convols import Convols
from pyhermes.theory.corr2pcf import Corr_2PCF
from pyhermes.theory.corr3pcf import Corr_3PCF
from pyhermes.param.parambase import read_param
# Read all parameters
# Both JSON (.json) and YAML (.yaml) formats are supported!
param_input = read_param(config_path='./param_multi.json')
# Calculate deltac, then return deltac and particle data p_dm
deltac, p_dm = Convols(param_task=param_input).run(return_pData=True)
# Calculate 2pcf, using the deltac from the result of above step
corr2pcf = Corr_2PCF(param_task=param_input).run(deltac=deltac)
# Calculate 3pcf, using the deltac from the results of above steps
Corr_3PCF(param_task=param_input).run(deltac=deltac, corr2pcf=corr2pcf, p_dm=p_dm)
just run it in your terminal, like
mpirun -n 8 python run_multi.py
15:00:14 - INFO - pyhermes.param.parambase:JsonBase - Reading configure file: './param_multi.json'
15:00:14 - INFO - pyhermes.param.parambase:JsonBase - Set default parameters of module <base> ...
15:00:14 - INFO - pyhermes.param.parambase:JsonBase - Default 'Convols.fin.path' from './data.bin' to 'https://pyhermes.astroslacker.com/_downloads/906e0695649e3634a5fe8081b9ab2086/quijote10000.bin'
15:00:14 - INFO - pyhermes.param.parambase:JsonBase - Default 'Convols.fout_path' from 'empty' to './multi_test.npy'
15:00:14 - INFO - pyhermes.param.parambase:JsonBase - Adding customizable window arg: 'Convols.window.R' as '5'
15:00:14 - INFO - pyhermes.param.parambase:JsonBase - Default 'Convols.threads' from '1' to '2'
15:00:14 - INFO - pyhermes.pipeline.pipeline:Convols - The task will run on 8 MPI ranks
15:00:14 - INFO - pyhermes.io.funcs:read_particle_data - Selected input particle format: generic
15:00:15 - INFO - pyhermes.io.funcs:dl_rich_pbar - File 'quijote10000.bin' already exists. Skipping download.
15:00:15 - INFO - pyhermes.io.funcs:read_generic - Reading paricle data from ---> quijote10000.bin <---
15:00:15 - INFO - pyhermes.pipeline.pipeline:Convols - Start partition ...
15:00:15 - INFO - pyhermes.pipeline.pipeline:Convols - The time for partition data: 0.0083 sec
15:00:15 - INFO - pyhermes.pipeline.pipeline:Convols - Start to calculate scaling coefficient...
15:00:18 - INFO - pyhermes.pipeline.pipeline:Convols - The time for scaling function: 3.2623 sec
15:00:18 - INFO - pyhermes.utils.math_util:set_window_function - Using window function: shell
15:00:21 - INFO - pyhermes.pipeline.pipeline:Convols - Start to calculte FFT
15:00:24 - INFO - pyhermes.pipeline.pipeline:Convols - The time for FFT: 2.8320 sec
15:00:24 - INFO - pyhermes.io.base:ConvolsData - Writing data to ---> ./multi_test.npy <---
15:00:26 - INFO - pyhermes.pipeline.pipeline:Convols - The time for task: 11.2947 sec
15:00:26 - INFO - pyhermes.pipeline.pipeline:Convols - Bye.
15:00:26 - INFO - pyhermes.param.parambase:JsonBase - Set default parameters of module <theory> ...
15:00:26 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_2PCF.fout_path' from 'empty' to './multi_test_2pcf.txt'
15:00:26 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_2PCF.threads' from '1' to '20'
15:00:26 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_2PCF.xi_num' from '150' to '200'
15:00:26 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - The task will run on 8 MPI ranks
15:00:26 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Loading DeltaC from argument 'deltac'
15:00:27 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Start to calculate 2PCF ...
15:00:29 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 0.00%
15:00:38 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 14.00%
15:00:42 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 22.00%
15:00:46 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 30.00%
15:00:53 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 40.00%
15:00:59 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 53.00%
15:01:03 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 61.00%
15:01:10 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 72.00%
15:01:14 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 80.00%
15:01:21 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 93.00%
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Progress: 100.00%
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - The time for 2PCF: 56.0502 sec
15:01:23 - INFO - pyhermes.io.base:Corr2PCFData - Writing data to ---> ./multi_test_2pcf.txt <---
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - The time for task: 56.7735 sec
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_2PCF - Bye.
15:01:23 - INFO - pyhermes.param.parambase:JsonBase - Set default parameters of module <theory> ...
15:01:23 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_3PCF.fin.path' from './data.bin' to ''
15:01:23 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_3PCF.fin.format' from 'generic' to ''
15:01:23 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_3PCF.fout_path' from 'empty' to './multi_test_3pcf.txt'
15:01:23 - INFO - pyhermes.param.parambase:JsonBase - Default 'Corr_3PCF.rot_num' from '10000' to '3'
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - The task will run on 8 MPI ranks
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Loading Corr2pcf from argument 'corr2pcf'
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Hi tristan, now you already have the corr2pcf info!
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Loading DeltaC from argument 'deltac'
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Loading Particle data from argument 'p_dm'
15:01:23 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Start to calculate 3PCF ...
15:01:25 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 0.00%
15:01:25 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 16.67%
15:01:25 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 19.05%
15:01:26 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 28.57%
15:01:26 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 38.10%
15:01:26 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 47.62%
15:01:27 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 57.14%
15:01:27 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 66.67%
15:01:27 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 76.19%
15:01:28 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 85.71%
15:01:28 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Progress: 100.00%
15:01:28 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Finished in 4.4 sec
15:01:28 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - The time for 3PCF: 4.4160 sec
15:01:28 - INFO - pyhermes.io.base:Corr3PCFData - Writing data to ---> ./multi_test_3pcf.txt <---
15:01:28 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - The time for task: 4.9321 sec
15:01:28 - INFO - pyhermes.pipeline.pipeline:Corr_3PCF - Bye.