vcstools scripts
The following scripts are provided as part of the vcstools package:
RVM_fit.py
A script that performs a least squares fit to the Rotating Vector Model
usage: RVM_fit.py [-h] --archive ARCHIVE [--outfile OUTFILE]
[--phase_ranges PHASE_RANGES [PHASE_RANGES ...]] [-L LOGLVL]
optional arguments:
-h, --help show this help message and exit
--archive ARCHIVE The pathname of the archive (.ar) file to attempt to
fit (default: None)
--outfile OUTFILE The pathname of the file to write the ouput to. If
None, will not write. Note: file outputs are in
radians (default: None)
--phase_ranges PHASE_RANGES [PHASE_RANGES ...]
The phase range(s) to fit the RM to. If unsupplied,
will find the on-pulse and fit that range. Supports
multiple ranges. eg. 0.1 0.15 0.55 0.62 will fit
values from 0.1 to 0.15 and from 0.55 to 0.62.
(default: None)
-L LOGLVL, --loglvl LOGLVL
Logger verbostity level (default: INFO)
calibrate_vcs.py
Script for creating RTS configuration files and submitting relevant jobs in the VCS pulsar pipeline
usage: calibrate_vcs.py [-h] -o OBSID -O CAL_OBSID -m METAFITS -s SRCLIST
[--gpubox_dir GPUBOX_DIR]
[--rts_output_dir RTS_OUTPUT_DIR]
[--beam_model {ANALYTIC,FEE2016}]
[--n_vis_grp N_VIS_GRP] [--offline]
[--vcstools_version VCSTOOLS_VERSION] [--nosubmit]
[-L {DEBUG,INFO,WARNING}] [-V]
optional arguments:
-h, --help show this help message and exit
-o OBSID, --obsid OBSID
Observation ID of target (default: None)
-O CAL_OBSID, --cal_obsid CAL_OBSID
Calibrator observation ID for the target observation.
Can be the same as --obs if using in-beam visibilities
(default: None)
-m METAFITS, --metafits METAFITS
Path to the relevant calibration observation metafits
file (default: None)
-s SRCLIST, --srclist SRCLIST
Path to the desired source list (default: None)
--gpubox_dir GPUBOX_DIR
Where the *_gpubox*.fits files are located (ONLY USE
IF YOU WANT THE NON-STANDARD LOCATIONS.) (default:
None)
--rts_output_dir RTS_OUTPUT_DIR
Parent directory where you want the /rts directory and
/batch directory to be created (ONLY USE IF YOU WANT
THE NON-STANDARD LOCATIONS.) (default: None)
--beam_model {ANALYTIC,FEE2016}
Which beam model to use to create the calibration
solution (default: FEE2016)
--n_vis_grp N_VIS_GRP
The number of visbility groups for the RTS to
construct. This changes with the MWA configuration
(i.e. long baselines require more visibility groups to
combat decorrelation) (default: 6)
--offline Tell the RTS to read calibrator data in the offline
correlated data format (default: False)
--vcstools_version VCSTOOLS_VERSION
The version of vcstools to use. Only relevant for
offline correlation (default: master)
--nosubmit Write jobs scripts but DO NOT submit to the queue
(default: True)
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: INFO (default: INFO)
-V, --version Print version and quit (default: False)
checks.py
scripts to check sanity of downloads and recombine.
usage: checks.py [-h] [-m {download,recombine}]
[-d {11,15,16,raw,ics,tar_ics}] [-o OBS ID] [-b start]
[-e stop] [-a] [-i time increment] [-s SIZE] [-S SIZE_ICS]
[-w WORK_DIR] [-V] [-L {DEBUG,INFO,WARNING}]
optional arguments:
-h, --help show this help message and exit
-m {download,recombine}, --mode {download,recombine}
Mode you want to run: download, recombine
-d {11,15,16,raw,ics,tar_ics}, --data_type {11,15,16,raw,ics,tar_ics}
Only necessary when checking downloads. Types refer to
those as definded in voltdownload.py: 11 = Raw, 15 =
ICS only, 16 = ICS and tarballs of recombined data.
-o OBS ID, --obs OBS ID
Observation ID you want to process [no default]
-b start, --begin start
gps time of first file to check on [default=None]
-e stop, --end stop gps time of last file to check on [default=None]
-a, --all Perform on entire observation span. Use instead of -b
& -e. [default=False]
-i time increment, --increment time increment
Effectively the number of seconds to ckeck for
starting at start time [default=None]
-s SIZE, --size SIZE The files size in bytes that you expect all files to
have. Per default will figure this out from files on
he archiveWe expect 253440000 (download raw),
327680000 (recombined, not ics), 7865368576 (tarballs)
-S SIZE_ICS, --size_ics SIZE_ICS
Size in bytes thatyou expect the ics files to have.
Default = 30720000
-w WORK_DIR, --work_dir WORK_DIR
Directory to check the files in. Default is
/astro/mwavcs/vcs/[obsID]/[raw,combined]
-V, --version Print version and quit
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: INFO
cleanup.py
Tool for cleaning up after processing MWA Pulsar Data on Galaxy
usage: cleanup.py [-h] [-o OBS] [--raw] [--beamformed] [-V]
[-L {DEBUG,INFO,WARNING}]
optional arguments:
-h, --help show this help message and exit
-o OBS, --obs OBS Observation ID you want to process [no default]
--raw Add this option if you want to remove the raw and/or
recombined files for an observation
--beamformed Add this option if you want to remove the recombined
files for an observation
-V, --version Print version and quit
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: INFO
combine_calibrations.py
If there is no calibration observation with the correct channels, use this script to combine the result of two calibration solutions to make a new calibration solution with all the required frequency channels.
usage: combine_calibrations.py [-h] [-o OBSID] [-c1 CAL1] [-c2 CAL2]
[--out_dir OUT_DIR] [-V]
[-L {DEBUG,INFO,WARNING}]
optional arguments:
-h, --help show this help message and exit
-o OBSID, --obsid OBSID
Observation ID you want to process
-c1 CAL1, --cal1 CAL1
The first calibration observation ID you want to
combine
-c2 CAL2, --cal2 CAL2
The second calibration observation ID you want to
combine
--out_dir OUT_DIR The output directory of the combined solution. Default
is /astro/mwavcs/vcs/[obsid]/cal/[cal1]_[cal2]/rts
-V, --version Print version and quit
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: INFO
create_ics_psrfits.py
positional arguments: obsID Observation ID for which to create the incoherent sum PSRFITS
usage: create_ics_psrfits.py [-h] [-b BASE_DIR] [-l {DEBUG,INFO,WARNING}] [-V]
obsID
optional arguments:
-h, --help show this help message and exit
-b BASE_DIR, --base_dir BASE_DIR
Base directory to look and then create the PSRFITS. In
the directory provided, there MUST exist a 'combined'
subfolder. An additional subfolder named 'ics' will be
created as a result of the incoherent sum creation.
Default is /group/mwavcs/vcs/[obsID]
-l {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Desired logging verbosity level
-V, --version Print version and quit
find_pulsar_in_obs.py
This code is used to list the sources within the beam of observations IDs or using –obs_for_source list all the observations for each source. The sources can be input serval ways: using a list of pulsar names (–pulsar), using a complete catalogue file of pulsars (–dl_PSRCAT) or RRATs (–RRAT and –dl_RRAT), using a compatable catalogue (–in_cat with the help of –names and –coordstype) or using a RA and DEC coordinate (–coords). The observation IDs can be input (–obsid) or gathered from a directory (–FITS_dir). The default is to search all observation IDs from http://mwa- metadata01.pawsey.org.au/metadata/ that have voltages and list every known pulsar from PSRCAT in each observation ID.
usage: find_pulsar_in_obs.py [-h] [--obs_for_source] [-b BEAM] [-m MIN_POWER]
[--sn_est] [--flux_est] [--plot_est]
[--output OUTPUT] [-L {DEBUG,INFO,WARNING}] [-V]
[-p [PULSAR [PULSAR ...]]] [--max_dm MAX_DM]
[--source_type SOURCE_TYPE] [--in_cat IN_CAT]
[-c [COORDS [COORDS ...]]] [--FITS_dir FITS_DIR]
[-o [OBSID [OBSID ...]]] [--min_time MIN_TIME]
[--freq_chan FREQ_CHAN] [--contig] [--all_volt]
[--cal_check]
optional arguments:
-h, --help show this help message and exit
--obs_for_source Instead of listing all the sources in each observation
it will list all of the observations for each source.
For increased efficiency it will only search OBSIDs
within the primary beam. (default: False)
-b BEAM, --beam BEAM Decides the beam approximation that will be used.
Options: "analytic" the analytic beam model (2012
model, fast and reasonably accurate), "advanced" the
advanced beam model (2014 model, fast and slighty more
accurate) or "full_EE" the full EE model (2016 model,
slow but accurate). " Default: "analytic" (default:
analytic)
-m MIN_POWER, --min_power MIN_POWER
The minimum fraction of the zenith normalised power
that a source needs to have to be recorded. Default
0.3 (default: 0.3)
--sn_est Make a expected signal to noise calculation using the
flux densities from the ANTF pulsar catalogue and
include them in the output file. Default: False.
(default: False)
--flux_est Make a expected flux density calculation using the
flux densities from the ANTF pulsar catalogue and
include them in the output file. Default: False.
(default: False)
--plot_est If used, will output flux estimation plots while
sn_est arg is true. Default: False. (default: False)
--output OUTPUT Chooses a file for all the text files to be output to.
The default is your current directory (default: ./)
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: INFO (default: INFO)
-V, --version Print version and quit (default: False)
--min_time MIN_TIME The minimum observation duration to include in output
files. Default 0 (default: 0)
--freq_chan FREQ_CHAN
Only use observations that include this frequency
channel. (default: None)
--contig Only use observations that have contiguous frequency
channels. (default: False)
mwa_metadb_utils.py
Returns information on an input Obs ID
usage: mwa_metadb_utils.py [-h] [-c] obsid
positional arguments:
obsid Input Observation ID
pabeam.py
Script to calculate the array factor required to model the tied-array beam for the MWA. This is an MPI-based simulation code and will use all available processes when run (i.e. there is no user choice in how many to use)
usage: pabeam.py [-h] [-o OBSID] [--metafits METAFITS] [-f FREQ [FREQ ...]]
[--flagged_tiles FLAGGED_TILES [FLAGGED_TILES ...]]
[--delays DELAYS [DELAYS ...]] [-e EFFICIENCY] [--coplanar]
[-p TARGET] [-b BEGIN] [-d DURATION] [-s STEP]
[--grid_res GRID_RES GRID_RES] [--az AZ AZ] [--za ZA ZA]
[--ra_dec_projection] [--ra_dec_res RA_DEC_RES RA_DEC_RES]
[--ra_radius RA_RADIUS] [--dec_radius DEC_RADIUS]
[--out_dir OUT_DIR] [--out_name OUT_NAME] [--write]
[--beam_model BEAM_MODEL] [--ord_version] [--no_delays]
[-L {DEBUG,INFO,WARNING,ERROR}]
optional arguments:
-h, --help show this help message and exit
process_vcs.py
process_vcs.py is a script for processing the MWA VCS data on Galaxy in steps. It can download data from the archive, call on recombine to form course channels, run the offline correlator, make tile re-ordered and bit promoted PFB files or for a coherent beam for a given pointing.
usage: process_vcs.py [-h] [-d PARALLEL_DL] [-j UNTAR_JOBS] [-k]
[--ft_res FREQ RES,TIME RES FREQ RES,TIME RES]
[-p [POINTINGS [POINTINGS ...]]]
[--pointing_file POINTING_FILE] [--DI_dir DI_DIR]
[--bf_out_format {psrfits,vdif,both}] [--incoh] [--sum]
[--flagged_tiles FLAGGED_TILES] [--cal_type CAL_TYPE]
[-E EXECPATH] [--ipfb_filter {LSQ12,MIRROR}]
[--beam_version {ANALYTIC,FEE2016}]
[-m {download,download_ics,download_cal,recombine,correlate,beamform}]
[-o OBS ID] [--cal_obs CALIBRATOR OBS ID] [-b BEGIN]
[-e END] [-a] [--all_avail] [-i INCREMENT] [-s] [-w DIR]
[-c NCOARSE_CHAN] [-n NFINE_CHAN] [--mail] [-L LOGLVL]
[-V] [--vcstools_version VCSTOOLS_VERSION] [--nice NICE]
optional arguments:
-h, --help show this help message and exit
-m {download,download_ics,download_cal,recombine,correlate,beamform}, --mode {download,download_ics,download_cal,recombine,correlate,beamform}
Mode you want to run. ['download', 'download_ics',
'download_cal', 'recombine', 'correlate', 'beamform']
(default: None)
-o OBS ID, --obs OBS ID
Observation ID you want to process [no default]
(default: None)
--cal_obs CALIBRATOR OBS ID, -O CALIBRATOR OBS ID
Only required in 'download_cal' mode.Observation ID of
calibrator you want to process. In case of in-beam
calibration should be the same as input to -o (obsID).
[no default] (default: None)
-b BEGIN, --begin BEGIN
First GPS time to process [no default] (default: None)
-e END, --end END Last GPS time to process [no default] (default: None)
-a, --all Perform on entire observation span. Use instead of -b
& -e. (default: False)
--all_avail Uses all of the available combined files available
(does not check for gaps). [default=False] (default:
False)
-i INCREMENT, --increment INCREMENT
Increment in seconds (how much we process at once)
(default: 64)
-s Single step (only process one increment and this is it
(False == do them all) (default: False)
-w DIR, --work_dir DIR
Base directory you want run things in. USE WITH
CAUTION! Per default raw data will will be downloaded
into /astro/mwavcs/vcs/[obsID] and data products will
be in /group/mwavcs/vcs/[obsID]. If set, this will
create a folder for the Obs. ID if it doesn't exist
(default: None)
-c NCOARSE_CHAN, --ncoarse_chan NCOARSE_CHAN
Coarse channel count (how many to process) (default:
24)
-n NFINE_CHAN, --nfine_chan NFINE_CHAN
Number of fine channels per coarse channel (default:
128)
--mail Enables e-mail notification about start, end, and fail
of jobs. Currently only implemented for beamformer
mode. (default: False)
-L LOGLVL, --loglvl LOGLVL
Logger verbosity level. Default: INFO (default: INFO)
-V, --version Print version and quit (default: False)
--vcstools_version VCSTOOLS_VERSION
VCSTools version to load in jobs (i.e. on the queues)
(default: master)
--nice NICE Reduces your priority of Slurm Jobs. (default: 0)
prof_estimate.py
A utility file for calculating a variety of pulse profile properties
usage: prof_estimate.py [-h] [--bestprof BESTPROF] [--ascii ASCII]
[--archive ARCHIVE] [--max_N MAX_N]
[--min_comp_len MIN_COMP_LEN]
[--clip_type {regular,noisy,verbose}]
[--plot_name PLOT_NAME] [-L LOGLVL]
optional arguments:
-h, --help show this help message and exit
recombine.py
Combines raw VCS data
usage: recombine.py [-h] [-o OBSID] [-s START] [-w DATA_DIR] [-e RECOMBINE]
[-L {DEBUG,INFO,WARNING,ERROR}] [-V]
optional arguments:
-h, --help show this help message and exit
-o OBSID, --obsid OBSID
Input Observation ID
-s START, --start START
GPS time to test
-w DATA_DIR, --data_dir DATA_DIR
Directory containing the raw data
-e RECOMBINE, --recombine RECOMBINE
filename of the recombine function
-L {DEBUG,INFO,WARNING,ERROR}, --loglvl {DEBUG,INFO,WARNING,ERROR}
Logger verbosity level. Default: INFO
-V, --version Print version and quit
rename_corr_output.py
Script to rename the files that are produced by the offline correlator to the convention assumed by Andr Offringa’s tools.
usage: rename_corr_output.py [-h] -o OBS ID -b start [-e stop] [-n n_secs]
[-w WORKDIR] [-V]
optional arguments:
-h, --help show this help message and exit
-o OBS ID, --obs OBS ID
Observation ID you want to process [no default]
-b start, --begin start
Time string of first file that needs renaming.
Format=YYYYMMDDHHMMSS. [default=None]
-e stop, --end stop Time string of last file that needs renaming.
Format=YYYYMMDDHHMMSS. [default=None]
-n n_secs, --nsec n_secs
Number of seconds that need renaming. Maximal
100.[default=100]
-w WORKDIR, --workdir WORKDIR
Directory that contains the output files of the
offline Correlator. Default is
/astro/mwavcs/vcs/<obsID/vis.
-V, --version Print version and quit
reorder_chans.py
Print the coarse channel sequence as in gpubox/subband order. Make sure you have *_metafits_ppds.fits
usage: reorder_chans.py [-h] -o OBSID [-m MFITS] [-V]
optional arguments:
-h, --help show this help message and exit
-o OBSID, --obsid OBSID
Observation ID of target
-m MFITS, --mfits MFITS
Please provide the path of the *_metafits_ppds.fits if
it is not
/astro/mwavcs/vcs/[OBID]/*_metafits_ppds.fits
-V, --version Print version and quit
rm_synthesis.py
A script that performs the RM synthesis technique
usage: rm_synthesis.py [-h] -a ARCHIVE
[--phase_ranges PHASE_RANGES [PHASE_RANGES ...]]
[--phi_steps PHI_STEPS]
[--phi_range PHI_RANGE [PHI_RANGE ...]]
[--force_single] [--label LABEL] [--write] [--plot_rm]
[--plot_gfit] [--keep_QUV] [--cliptype CLIPTYPE]
[-d WORK_DIR] [-L {DEBUG,INFO,WARNING,ERROR}]
optional arguments:
-h, --help show this help message and exit
rts2ao.py
rts2ao.py is a tool for converting an RTS solution to an Offringa-style solution.
usage: rts2ao.py [-h] [-d DIR] [-m METAFITS] [-r RTS_GLOB] [-o OUTFILE]
[-L {DEBUG,INFO,WARNING}] [-V]
optional arguments:
-h, --help show this help message and exit
-d DIR, --dir DIR Directory containing RTS solution files (DI_Jones...)
(default: ./)
-m METAFITS, --metafits METAFITS
Metafits file from which to determine antenna and
channel order (default: None)
-r RTS_GLOB, --rts_glob RTS_GLOB
Filename globbing pattern for the RTS direction
independent solution files (default:
DI_JonesMatrices_node[0-9]*.dat)
-o OUTFILE, --outfile OUTFILE
Name of the file to write (default:
calibration_solution.bin)
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: INFO (default: INFO)
-V, --version Print version and quit (default: False)
sn_flux_est.py
.. _splice-wrapper.py-label:
splice_wrapper.py
Wraps the splice_psrfits.sh script to automate it. Should be run from the foulder containing the files.
usage: splice_wrapper.py [-h] [-o OBSERVATION] [-i] [-d] [-w WORK_DIR]
[-c CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS]
optional arguments:
-h, --help show this help message and exit
-o OBSERVATION, --observation OBSERVATION
The observation ID to be used.
-i, --incoh Use this check if there are and incoh files from the
beamformer.
-d, --delete This will cause the script to remove the unspliced
files if splice_psrfits.sh succeeds (error code of 0).
-w WORK_DIR, --work_dir WORK_DIR
Working directory of the vcs files.
-c CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS, --channels CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS CHANNELS
A list of the observations channel IDs for example "-c
109 110 111 112 113 114 115 116 117 118 119 120 121
122 123 124 125 126 127 128 129 130 131 132". If this
option is not used a metadata call will be used to
find the channel IDs.
submit_to_database.py
This code is used to submit pulsar detections and observation calibrations to the MWA Pulsar Database. It can calculate all need values (flux density, width, scattering) for each detection and submit them to the pulsar database without having to manually input them. It can also submit diagnostic files such as pulse profiles and calibrations. This code assumes that the observation is coherent and correlated using the RTS so please use –incoh and –andre if your observation is incoherent or correlated using Andre’s Tools respectively.
usage: submit_to_database.py [-h] [-o OBSID] [-O CALID] [-p PULSAR] [--incoh]
[--andre] [--sefd_file SEFD_FILE] [--simple_sefd]
[--vcstools_version VCSTOOLS_VERSION]
[-L {DEBUG,INFO,WARNING}] [-V] [-b BESTPROF]
[--ascii ASCII] [--dont_upload]
[--pointing POINTING] [--start START]
[--stop STOP] [--trcvr TRCVR]
[--cal_dir_to_tar CAL_DIR_TO_TAR]
[--srclist SRCLIST] [-c CALIBRATION] [-a ARCHIVE]
[--single_pulse_series SINGLE_PULSE_SERIES]
[--ppps PPPS] [-i IPPD] [-w WATERFALL]
A common use case is to simply upload a calibration file (this can be done before a detection is uploaded)
> submit_to_database.py --cal_dir_to_tar <calibration directory> --srclist <srclist> -o <obs ID> -O <calibrator obs ID>
Another is to record a pulsar detection and its paramters, such as flux density and pulse width, to the database. To do this the bestprof file is needed
> submit_to_database.py -b <bestprof file location>
Diagnostic files (see Upload Options) such as PRESTO plots can be uploaded using:
> submit_to_database.py -o <obs ID> -O <calibrator obs ID> -p <pulsar> --ppps <PRESTO prepfold output post script file location>
The diagnostic files can be created using the Dspsr Calculation Options (not robustly tested).
zapchan.py
Output the correctly formated MWA band edge channels and user-defined channels to remove for aliasing/RFI excision. If no options are given, returns an empty string.
usage: zapchan.py [-h] [-N Nchan] [-C Nchan Nchan] [-Z] [-n Nedge] [-m]
[-z chan [chan ...]] [-r] [-p filename]
[-L {DEBUG,INFO,WARNING}] [-V]
optional arguments:
-h, --help show this help message and exit
-N Nchan, --nchans Nchan
TOTAL number of frequency channels in data set.
Assumes 128 fine channels per coarse channel.
(default: None)
-C Nchan Nchan, --n_coarse_fine Nchan Nchan
Number of coarse channels followed by the number of
fine channels per coarse channel. This options
overrides --nchans. (default: None)
-Z, --zapedges Zap the fine channel edges of the coarse channels.
(default: False)
-n Nedge, --nzap Nedge
Number of edge channels to remove from each side of a
coarse channel. If given, but --zapedges is not then
this argument is ignored. (default: 20)
-m, --middle Flag the center fine channels for each coarse channel?
(default: False)
-z chan [chan ...] Individual channels to zap (default: None)
-r, --rfifind Output to screen the edge channels in a format
readable by the PRESTO rfifind routine (default:
False)
-p filename, --paz filename
Output channels for PSRCHIVE paz routine into the
given file, using paz's '-k filename' option (default:
None)
-L {DEBUG,INFO,WARNING}, --loglvl {DEBUG,INFO,WARNING}
Logger verbosity level. Default: WARNING (default:
WARNING)
-V, --version Print version and quit (default: False)