= Name =
sxprocess - Miscellaneous Commands: Carry out various SPARX commands on image series, and generate data and initialize database for demo script.

= Usage =


''usage in command line''

sxprocess.py  inputfile  outputfile  micrograph_prefix  --order  --order_lookup  --order_metropolis  --order_pca  --initial=INITIAL  --circular  --radius=RADIUS  --changesize  --ratio=RATIO  --pw  --wn=WN  --phase_flip  --makedb=param1=value1:param2=value2  --generate_projections=param1=value1:param2=value2  --isacgroup=ISACGROUP  --isacselect  --params=PARAMS  --adjpw  --rotpw=ROTPW  --transformparams=TRANSFORMPARAMS  --importctf=IMPORTCTF  --input=INPUT  --defocuserror=DEFOCUSERROR  --astigmatismerror=ASTIGMATISMERROR  --scale=SCALE  --adaptive_mask  --nsigma=NSIGMA  --threshold=THRESHOLD  --ndilation=NDILATION  --kernel_size=KERNEL_SIZE  --gauss_standard_dev=GAUSS_STANDARD_DEV  --binary_mask  --bin_threshold=BIN_THRESHOLD  --ne=NE  --nd=ND  --postprocess  --mtf=MTF_FILE_NAME  --fsc_adj  --B_enhance=B_ENHANCE  --fl=lpf_cutoff_freq  --aa=lpf_falloff  --mask=MASK  --do_adaptive_mask  --mask_threshold=MASK_THRESHOLD  --consine_edge=CONSINE_EDGE  --dilation=DILATION  --output=OUTPUT  --pixel_size=PIXEL_SIZE  --B_start=B_START  --B_stop=B_STOP  --window_stack  --box=BOX  --angular_distribution  --round_digit=ROUND_DIGIT  --box_size=BOX_SIZE  --particle_radius=PARTICLE_RADIUS  --cylinder_width=CYLINDER_WIDTH  --cylinder_length=CYLINDER_LENGTH


=== Typical usage ===

sxprocess does not support MPI.

 1. Phase flip a stack of images and write output to new file:

''' sxprocess.py  input_stack.hdf  output_stack.hdf  --phase_flip '''

 2. Resample (decimate or interpolate up) images (2D or 3D) in a stack to change the pixel size. The window size will change accordingly.

''' sxprocess  input.hdf  output.hdf  --changesize  --ratio=0.5 '''

 3. Compute average power spectrum of a stack of 2D images with optional padding (option wn) with zeroes.

''' sxprocess.py  input_stack.hdf  powerspectrum.hdf  --pw  [--wn=1024] '''

 4. Generate a stack of projections bdb:data and micrographs with prefix mic (i.e., mic0.hdf, mic1.hdf etc) from structure input_structure.hdf, with CTF applied to both projections and micrographs.

''' sxprocess.py  input_structure.hdf  data mic  --generate_projections  format="bdb":apix=5.2:CTF=True:boxsize=64 '''

 5. Retrieve original image numbers in the selected ISAC group (here group 12 from generation 3).

''' sxprocess.py  bdb:test3  class_averages_generation_3.hdf  list3_12.txt  --isacgroup=12  --params=originalid '''

 6.  Retrieve original image numbers of images listed in ISAC output stack of averages/

''' sxprocess.py  select1.hdf  ohk.txt '''

 7. Adjust rotationally averaged power spectrum of an image to that of a reference image or a reference 1D power spectrum stored in an ASCII file. Optionally use a tangent low-pass filter.  Also works for a stack of images, in which case the output is also a stack.

''' sxprocess.py  vol.hdf ref.hdf  avol.hdf  < 0.25 0.2>  --adjpw '''
''' sxprocess.py  vol.hdf pw.txt   avol.hdf  < 0.25 0.2>  --adjpw '''

 8. Generate a 1D rotationally averaged power spectrum of an image.

''' sxprocess.py  vol.hdf  --rotwp=rotpw.txt '''

# Output will contain three columns:
(1) rotationally averaged power spectrum
(2) logarithm of the rotationally averaged power spectrum
(3) integer line number (from zero to approximately to half the image size)

 9. Apply 3D transformation (rotation and/or shift) to a set of orientation parameters associated with projection data.

''' sxprocess.py  --transfromparams=phi,theta,psi,tx,ty,tz  input.txt  output.txt '''

The output file is then imported and 3D transformed volume computed.

''' sxheader.py  bdb:p  --params=xform.projection  --import=output.txt '''
''' mpirun  -np  2  sxrecons3d_n.py  bdb:p tvol.hdf  --MPI '''

The reconstructed volume is in the position of the volume computed using the input.txt parameters and then transformed with rot_shift3D(vol, phi,theta,psi,tx,ty,tz).

10. Import ctf parameters from the output of sxcter into windowed particle headers. There are three possible input files formats:  (1) all particles are in one stack, (2 aor 3) particles are in stacks, each stack corresponds to a single micrograph. In each case the particles should contain a name of the micrograph of origin stores using attribute name 'ptcl_source_image'. Normally this is done by e2boxer.py during windowing. Particles whose defocus or astigmatism error exceed set thresholds will be skipped, otherwise, virtual stacks with the original way preceded by G will be created.

''' sxprocess.py  --input=bdb:data  --importctf=outdir/partres  --defocuserror=10.0  --astigmatismerror=5.0 '''

# Output will be a vritual stack bdb:Gdata.

''' sxprocess.py  --input="bdb:directory/stacks*"  --importctf=outdir/partres  --defocuserror=10.0  --astigmatismerror=5.0 '''

To concatenate output files,

''' cd directory '''
''' e2bdb.py  .  --makevstack=bdb:allparticles  --filt=G '''

IMPORTANT: Please do not move (or remove!) any input/intermediate EMAN2DB files as the information is linked between them.

11. Scale 3D shifts. The shifts in the input five columns text file with 3D orientation parameters will be DIVIDED by the scale factor.

''' sxprocess.py  orientationparams.txt  scaledparams.txt  scale=0.5 '''

12. Generate soft-edged 3D mask from input 3D volume automatically or using the user-provided threshold.

# Automatically compute the threshold to intially obtain the largest density cluster.

''' sxprocess.py  vol3d.hdf  mask3d.hdf  --adaptive_mask  --nsigma=3.0  --ndilation=1  --kernel_size=9  --gauss_standard_dev=5 '''

# Use the user-provided threshold to intially obtain the largest density cluster.

''' sxprocess.py  vol3d.hdf  mask3d.hdf  --adaptive_mask --threshold=0.05  -ndilation=0  --kernel_size=9  --gauss_standard_dev=5 '''

13. Generate binary 3D mask from input 3D volume using the user-provided threshold.

''' sxprocess.py  vol3d.hdf  mask3d.hdf  --binary_mask  --threshold=0.05  --ne=3  --nd==3 '''

14. Postprocess two unfiltered 3D volumes produced by meridien: This command executes the following processes. (a) Calculate FSC with provided mask and adjust the FSC by random phases FSC, (b) Sum two volume, (c) Apply mask, (d) Apply MTF correction (optional), (e) Adjust power spectrum by 2*FSC/(1+FSC) (optional), (f) Estimate B-factor from 10 Angstrom (default) to the resolution (optional), (g) Apply negative B-factor to enhance the volume (optional), (h) Apply low_pass filter to the volume (optional). Options are independent of each others.
--fl               : =0.0, low_pass filter to resolution; =-1.0, no low_pass filter; =5.8 low_pass filter to 5.8 Angstrom; =0.2 low_pass filter to 0.2.
--B_enhance        : =-1, B-factor is not applied; =0, program estimates B-factor from options. B_start (usually set as 10 Angstrom) to the resolution determined by FSC 0.143; =128.0, program uses the given value 128.0 to enhance map.
--mtf              : =mtf.txt, for those high resolution maps, mtf correction would significantly enhance structural features.
--fsc_adj          : fsc adjustment of power spectrum is inclined to increase the slope of power spectrum of the summed volume.
--do_adaptive_mask : =True, the program adaptively creates 3d mask file using summed volumes. This takes a couple of minutes. For map with dimension of 384*384*384, it takes 6 minutes.
--output           : output volume 

''' sxprocess.py  --postprocess  vol_0_unfil.hdf  vol_1_unfil.hdf  --mask=mask15.hdf  --pixel_size=1.12  --fl=-1  --mtf=mtf.txt  --fsc_adj  --output=vol_post.hdf '''

''' sxprocess.py  --postprocess  vol_0_unfil.hdf  vol_1_unfil.hdf  --mask=mask15.hdf  --pixel_size=1.12  --fl=4.7  --mtf=mtf.txt  --fsc_adj '''

''' sxprocess.py  --postprocess  vol_0_unfil.hdf  vol_1_unfil.hdf  --do_adaptive_mask  --pixel_size=1.12  --mtf=mtf.txt  --fsc_adj '''

15. Window stack file -reduce size of images without changing the pixel size.

16. Create angular distribution .build file
'''sxprocess.py --angular_distribution  inputfile=example/path/params.txt --pixel_size=1.0  --round_digit=5  --box_size=500  --particle_radius=175  --cylinder_width=1  --cylinder_length=10000'''

== Input ==
    inputfile:: input file: required for some options. (default none)

    changesize:: Resample 2D or 3D images: Resample (decimate or interpolate up) images (2D or 3D) in a stack to change the pixel size. (default False)
    ratio:: Ratio of new to old image size: if < 1, the pixel size will increase and image size decrease. if > 1, the other way round. (default 1.0)

    pw:: Compute average power spectrum of a 2D image stack: with optional padding (option wn) with zeroes. (default False)
    wn:: Size of window to use: it should be larger/equal than particle box size, default padding to max(nx,ny). (default -1)

    phase_flip:: Phase flip the input stack: (default False)

    generate_projections:: Generate projections and simulated micrographs from 3D volume: three arguments are required. name of input structure from which to generate projections, desired name of output projection stack, and desired prefix for micrographs (e.g. if prefix is 'mic', then micrographs mic0.hdf, mic1.hdf etc will be generated). optional arguments specifying format, apix, box size and whether to add CTF effects can be entered as follows after --generate_projections: format='bdb':apix=5.2:CTF=True:boxsize=100, or format='hdf', etc., where format is bdb or hdf, apix (pixel size) is a float, CTF is True or False, and boxsize denotes the dimension of the box (assumed to be a square). if an optional parameter is not specified, it will default as follows: format='bdb', apix=2.5, CTF=False, boxsize=64. (default none)

    isacgroup:: Retrieve original image numbers in selected ISAC group: see ISAC documentation for details. (default -1)

    isacselect:: Create ISAC particle ID list: Retrieve original image numbers listed in ISAC output average stack. see ISAC documentation for details. (default False)

    params:: name of parameter in image file header: which one depends on specific option. (default none)

    adjpw:: Adjust rotationally averaged power spectrum of an image: (default False)

    rotpw:: Compute rotationally averaged power spectrum of the input image: store to output text file name specified by this option. (default none)

    importctf:: Import sxcter CTF parameters to stack file: specify file name containing CTF parameters produced by sxcter with this option. (default none)
    input:: input particle image stack file: to which CTF parameters will be imported. (default none)
    defocuserror:: defocus error threshold: exclude micrographs whose relative defocus error as estimated by sxcter is larger than defocuserror percent. the error is computed as (std dev defocus)/defocus*100%. (default 1000000.0)
    astigmatismerror:: astigmatism error threshold: set to zero astigmatism for micrographs whose astigmatism angular error as estimated by sxcter is larger than astigmatismerror degrees. (default 360.0)

    scale:: Divide shifts in input 3D orientation parameters text file by this scale factor: (default -1.0)

    adaptive_mask:: Create soft-edged 3D mask from reference volume: (default False)
    nsigma:: Density standard deviations threshold: Defines the threshold used to find the main volume within the data. All voxels with density <= mean + nsigma standard deviations will be included into the main volume. This option will not be used if the option threshold is larger than -9999.0. (default 1.0)
    threshold:: Binarization threshold: Below this value the data is assumed to not belong to the main volume. With the value lower than the default, the option will be ignored and the mask will be set according to nsigma. (default -9999.0)
    ndilation:: Mask extension cycles: The initial mask will be extended this number of cycles. To keep the size of the main volume, set this to kernel_size/2 (default 3)
    kernel_size:: Gaussian kernel size: Size of the gaussian kernel used to smooth the binary mask. (default 11)
    gauss_standard_dev:: Kernel standard deviation : Standard deviation used in the construction of the gaussian smoothing of the mask. (default 9)

    binary_mask:: Create binary 3D mask from reference volume: (default False)
    bin_threshold:: Binarization threshold: Below this value the data is assumed to not belong to the main volume. (default 0.0)
    ne:: Erosion cycles: After initial binarization the volume is eroded to remove fragmented pieces of the volume. (default 0)
    nd:: Dilation cycles: After erosing the binary volume is dilated back to smooth the surface and match the original size. (default 0)

    postprocess:: Apply B-factor to sharpen the power spectrum: B-factor is estimated from two unfiltered 3D volumes produced by meridien. (default False)
    mtf:: MTF file: File contains the MTF (modulation transfer function) of the detector used. (default none)
    fsc_adj:: Apply FSC-based low-pass filter: Applies an FSC-based low-pass filter to the merged volume before the B-factor estimation. (default False)
    B_enhance:: B-factor enhancement: -1.0: B-factor is not applied; 0.0: program estimates B-factor from options. B_start (usually 10 Angstrom) to the resolution determined by FSC143; 128.0: program use the given value 128.0 [A^2] to enhance map. (default 0.0)
    fl:: Low-pass filter frequency: 0.0: low-pass filter to resolution; A value > 0.5: low-pass filter to the value in Angstrom; A value > 0.0 and < 0.5: low-pass filter to the value in absolute frequency; -1.0: no low-pass filter. (default 0.0)
    aa:: Low-pass filter fall-off: Low-pass filter fall-off. (default 0.1)
    mask:: User-provided mask: Path to user-provided mask. (default none)
    do_adaptive_mask:: Apply adaptive mask: Program creates mask adaptively with given density threshold. (default False)
    mask_threshold:: Adaptive mask threshold: Density threshold for creating adaptive surface mask. (default 0.02)
    consine_edge:: Cosine edge width [Pixels]: Width of cosine transition area for soft-edge masking. (default 6.0)
    dilation:: surface dilation size [Pixels]: Size of surface dilation or erosion. (default 3.0) 
    output:: Output file: Output file name. (default vol_postrefine_masked.hdf)
    pixel_size:: Pixel size [A]: Pixel size of input data. (default 0.0)
    B_start:: B-factor lower limit [A]: Lower limit for B-factor estimation. (default 10.0)
    B_stop:: B-factor higher limit [A]: Higher limit for B-factor estimation. (default  0.0)

    window_stack:: Window stack images using a smaller window size: (default False)
    box:: new window size: (default 0)

    angular_distribution:: Create angular distribution file: The file will contain a 3D representation of the given angular distribution, which can be viewed with UCFS Chimera (default False)
    pixel_size:: Pixel size [A]: Pixel size in A. (default 1.0)
    round_digit:: Number precision: Decimal numbers will be rounded to this number of decimal points. (default 5)
    box_size:: Box size [Pixels]: Box size in pixel used for calculating the center of the particle. (default 500)
    particle_radius:: Particle radius [Pixels]: Used for the representation in Chimera. Defines where the cylinders representing the histogram must start. (default 175)
    cylinder_width:: Cylinder width: Used for the representation in Chimera. This will define the width of the cylinders representing the histogram.(default 1)
    cylinder_length:: Cylinder length: Used for the representation in Chimera. This will define the relative size of the cylinders representing the histogram. (default 10000)

    * The remaining parameters are optional.
    order:: Order (unimplemented): two arguments are required, (1) name of input stack and (2) desired name of output stack. The output stack is the input stack sorted by similarity in terms of cross-correlation coefficient. (default False)
    order_lookup:: order_lookup (unimplemented): test/debug. (default False)
    order_metropolis:: order_metropolis (unimplemented): test/debug. (default False)
    order_pca:: order_pca (unimplemented): test/debug. (default False)
    initial:: initial (unimplemented): specifies which image will be used as an initial seed to form the chain. By default, use the first image (default 0)
    circular:: circular (unimplemented): select circular ordering (first image has to be similar to the last) (default False)
    radius:: radius (unimplemented): radius of a circular mask for similarity based ordering (default False)

    makedb:: Generate a database file containing a set of parameters: one argument is required, name of key with which the database will be created. Fill in database with parameters specified as --makedb param1=value1:param2=value2 (e.g. 'gauss_width'=1.0:'pixel_input'=5.2:'pixel_output'=5.2:'thr_low'=1.0). (default none)

    transformparams:: Transform 3D projection orientation parameters: using six 3D parameters (phi,theta,psi,sx,sy,sz). input is --transformparams=45.,66.,12.,-2,3,-5.5 desired six transformation of the reconstructed structure. output is file with modified orientation parameters. (default none)



== Output ==
    outputfile::  output file: required for some options. (default none)
    micrograph_prefix:: prefix for output micrographs: required for some options. (default none)


== Development Notes ==
- 2017/04/13 Toshio Moriya
Before remove randomphasesafter options
sxprocess.py  inputfile  outputfile  micrograph_prefix  --order  --order_lookup  --order_metropolis  --order_pca  --initial=INITIAL  --circular  --radius=RADIUS  --changesize  --ratio=RATIO  --pw  --wn=WN  --phase_flip  --makedb=param1=value1:param2=value2  --generate_projections=param1=value1:param2=value2  --isacgroup=ISACGROUP  --isacselect  --params=PARAMS  --adjpw  --rotpw=ROTPW  --transformparams=TRANSFORMPARAMS  --importctf=IMPORTCTF  --input=INPUT  --defocuserror=DEFOCUSERROR  --astigmatismerror=ASTIGMATISMERROR  --scale=SCALE  --adaptive_mask  --nsigma=NSIGMA  --threshold=THRESHOLD  --ndilation=NDILATION  --kernel_size=KERNEL_SIZE  --gauss_standard_dev=GAUSS_STANDARD_DEV  --binary_mask  --bin_threshold=BIN_THRESHOLD  --ne=NE  --nd=ND  --postprocess  --mtf=MTF_FILE_NAME  --fsc_adj  --B_enhance=B_ENHANCE  --randomphasesafter=FSC  --fl  --aa=AA  --mask=MASK  --do_adaptive_mask  --mask_threshold=MASK_THRESHOLD  --consine_edge=CONSINE_EDGE  --dilation=DILATION  --output=OUTPUT  --pixel_size=PIXEL_SIZE  --B_start=B_START  --B_stop=B_STOP  --window_stack  --box=BOX  --angular_distribution  --round_digit=ROUND_DIGIT  --box_size=BOX_SIZE  --particle_radius=PARTICLE_RADIUS  --cylinder_width=CYLINDER_WIDTH  --cylinder_length=CYLINDER_LENGTH
randomphasesafter:: Randomise phases after this FSC: Set Fourier pixels random phases after this FSC value. (default 0.8)


= Author / Maintainer =
Jia Fang

= Keywords =
 category 1:: UTILITIES
 category 1:: APPLICATIONS

= Files =
sxprocess.py

= Maturity =
 beta:: works for author, often works for others.
