This is gnuastro.info, produced by makeinfo version 7.2 from
gnuastro.texi.

This book documents version 0.24 of the GNU Astronomy Utilities
(Gnuastro).  Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.

   Copyright © 2015-2025 Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this
     document under the terms of the GNU Free Documentation License,
     Version 1.3 or any later version published by the Free Software
     Foundation; with no Invariant Sections, no Front-Cover Texts, and
     no Back-Cover Texts.  A copy of the license is included in the
     section entitled "GNU Free Documentation License".
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro).       GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.

* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.

* bug-gnuastro: (gnuastro)Report a bug. How to report bugs

* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.

* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro's library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.

* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.

* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.

* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.

* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.

* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.

* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.

* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.

* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.

* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.

* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.

* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.

* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.

* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.

* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.

* astscript: (gnuastro)Installed scripts. Gnuastro's installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY


File: gnuastro.info,  Node: Zero point tutorial with reference catalog,  Prev: Zero point tutorial with reference image,  Up: Zero point of an image

2.7.2 Zero point tutorial with reference catalog
------------------------------------------------

In *note Zero point tutorial with reference image::, we explained how to
use the ‘astscript-zeropoint’ for estimating the zero point of one image
based on a reference image.  Sometimes there is not a reference image
and we need to use a reference catalog.  Fortunately,
‘astscript-zeropoint’ can also use the catalog instead of the image to
find the zero point.

   To show this, let's download a catalog of SDSS in the area that
overlaps with the cropped J-PLUS image (used in the previous section).
For more on Gnuastro's Query program, please see *note Query::.  The
columns of ID, RA, Dec and magnitude in the SDSS _r_ filter are called
by their name in the SDSS catalog.

     $ astquery vizier \
                --dataset=sdss12 \
                --overlapwith=jplus-crop.fits \
                --column=objID,RA_ICRS,DE_ICRS,rmag \
                --output=sdss-catalog.fits

   To visualize the position of the SDSS objects over the J-PLUS image,
let's use ‘astscript-ds9-region’ (for more details please see *note SAO
DS9 region files from table::) with the command below (it will
automatically open DS9 and load the regions it created):

     $ astscript-ds9-region sdss-catalog.fits \
                            --column=RA_ICRS,DE_ICRS \
                            --color=red --width=3 --output=sdss.reg \
                            --command="ds9 jplus-nc.fits[INPUT-NO-SKY] \
                                           -scale zscale"

   Now, we are ready to estimate the zero point of the J-PLUS image
based on the SDSS catalog.  To download the input image and understand
how to use the ‘astscript-zeropoint’, please see *note Zero point
tutorial with reference image::.

   Many of the options (like the aperture size) and magnitude range are
the same so we will not discuss them further.  You will notice that the
only substantive difference of the command below with the last command
in the previous section is that we are using ‘--refcat’ instead of
‘--refimgs’.  There are also some cosmetic differences for example a new
output name, not using ‘--refimgszp’ since it is only necessary for
images) and the ‘--*column’ options which are used to identify the names
of the necessary columns of the input catalog:

     $ astscript-zeropoint jplus-nc.fits --hdu=INPUT-NO-SKY \
                           --refcat=sdss-catalog.fits \
                           --refcatmag=rmag \
                           --refcatra=RA_ICRS \
                           --refcatdec=DE_ICRS \
                           --output=jplus-zeropoint-cat.fits \
                           --magnituderange=16.4,18.5 \
                           --aperarcsec=2,2.5,3,3.5,4 \
                           --keepzpap

Let's inspect the output with the command below.

     $ asttable jplus-zeropoint-cat.fits -Y
     2.000          26.337         0.034
     2.500          26.386         0.036
     3.000          26.417         0.041
     3.500          26.439         0.043
     4.000          26.455         0.050

   As you see, the values and standard deviations are very similar to
the results we got previously in *note Zero point tutorial with
reference image::.  The Standard deviations are generally a little
higher here because we didn't do the photometry ourselves, but they are
statistically similar.

   Before we finish, let's open the two outputs (from a reference image
and reference catalog) with the command below.  To confirm how they
compare, we are showing the result for ‘APER-3’ extension in both
(following the TOPCAT plotting recipe in *note Zero point tutorial with
reference image::).

     $ astscript-fits-view jplus-zeropoint.fits jplus-zeropoint-cat.fits \
                           -hAPER-3


File: gnuastro.info,  Node: Pointing pattern design,  Next: Moire pattern in coadding and its correction,  Prev: Zero point of an image,  Up: Tutorials

2.8 Pointing pattern design
===========================

A dataset that is ready for scientific analysis is usually composed of
many separate exposures and how they are taken is usually known as
"observing strategy".  This tutorial describes Gnuastro's tools to
simplify the process of deciding the pointing pattern of your observing
strategy.

   A "pointing" is the location on the sky that each exposure is aimed
at.  Each exposure's pointing is usually moved (on the sky) compared to
the previous exposure.  This is done for reasons like improving
calibration, increasing resolution, expending the area of the
observation and etc.  Therefore, deciding a suitable pointing pattern is
one of the most important steps when planning your observation strategy.

   There are commonly two types of pointings: "dither" and "offset".
These are sometimes used interchangeably with "pointing" (especially
when the final coadd is roughly the same area as the field of view.
Alternatively, "dither" and "offset" are used to distinguish pointings
with large or small (on the scale of the field of view) movement
compared to a previous one.  When a pointing has a large distance to the
previous pointing, it is known as an "offset", while pointings with a
small displacement are known as a "dither".  This distinction originates
from the mechanics and optics of most modern telescopes: the overhead
(for example the need to re-focus the camera) to make small movements is
usually less than large movements.

   In this tutorial, let's simulate a hypothetical pointing pattern
using Gnuastro's ‘astscript-pointing-simulate’ installed script (see
*note Pointing pattern simulation::).  Since we will be testing very
different displacements between pointings, we'll ignore the difference
between offset and dither here, and only use the term pointing.

   Let's assume you want to observe M94
(https://en.wikipedia.org/wiki/Messier_94) in the H-alpha and rSDSS
filters (to study the extended star formation in the outer rings of this
beautiful galaxy!).  Including the outer parts of the rings, the galaxy
is half a degree in diameter!  This is very large, and you want to
design a pointing pattern that will allow you to cover as much area,
while not loosing your ability to calibrate properly.

*Do not start with this tutorial:* If you are new to Gnuastro and have
not already completed *note General program usage tutorial::, we
recommend going through that tutorial before starting this one.  Basic
features like access to this book on the command-line, the configuration
files of Gnuastro's programs, benefiting from the modular nature of the
programs, viewing multi-extension FITS files, and many others are
discussed in more detail there.

* Menu:

* Preparing input and generating exposure map::  Download and image and build exposure map.
* Area of non-blank pixels on sky::  Account for the curved area on the sky.
* Script with pointing simulation steps so far::  Summary of steps for easy testing.
* Larger steps sizes for better calibration::  The initial small dither is not enough.
* Pointings that account for sky curvature::  Sky curvature will cause problems!
* Accounting for non-exposed pixels::  Parts of the detector do not get exposed to light.


File: gnuastro.info,  Node: Preparing input and generating exposure map,  Next: Area of non-blank pixels on sky,  Prev: Pointing pattern design,  Up: Pointing pattern design

2.8.1 Preparing input and generating exposure map
-------------------------------------------------

As mentioned in *note Pointing pattern design::, the assumed goal here
is to plan an observations strategy for M94.  Let's assume that after
some searching, you decide to write a proposal for the JAST80 telescope
(https://oaj.cefca.es/telescopes/jast80) at the Observatorio Astrofísico
de Javalambre (https://oaj.cefca.es), OAJ(1), in Teruel (Spain).  The
field of view of this telescope's camera is almost 1.4 degrees wide,
nicely fitting M94!  It also has these two filters that you need(2).

   Before we start, as described in *note Pointing pattern simulation::,
it is just important to remember that the ideal pointing pattern depends
primarily on your scientific objective, as well as the limitations of
the instrument you are observing with.  Therefore, there is no single
pointing pattern for all purposes.  However, the tools, methods,
criteria or logic to check if your pointing pattern satisfies your
scientific requirement are similar.  Therefore, you can use the same
methods, tools or logic here to simulate or verify that your pointing
pattern will produce the products you expect after the observation.

   To start simulating a pointing pattern for a certain telescope, you
just need a single-exposure image of that telescope with WCS
information.  In other words, after astrometry, but before warping into
any other pixel grid (to combine into a deeper coadd).  The image will
give us the default number of the camera's pixels, its pixel scale
(width of pixel in arcseconds) and the camera distortion.  These are
reference parameters that are independent of the position of the image
on the sky.

   Because the actual position of the reference image is irrelevant,
let's assume that in a previous project, presumably on NGC 4395
(https://en.wikipedia.org/wiki/NGC_4395), you already had the download
command of the following single exposure image.  With the last command,
please take a look at this image before continuing and explore it.

     $ mkdir pointing-tutorial
     $ cd pointing-tutorial
     $ mkdir input
     $ siapurl=https://archive.cefca.es/catalogues/vo/siap
     $ wget $siapurl/jplus-dr3/reduced/get_fits?id=1050345 \
            -O input/jplus-1050345.fits.fz

     $ astscript-fits-view input/jplus-1050345.fits.fz

*This is the first time I am using an instrument:* In case you haven't
already used images from your desired instrument (to use as reference),
you can find such images from their public archives; or contacting them.
A single exposure images is rarely of any scientific value
(post-processing and coadding is necessary to make high-level and
science-ready products).  Therefore, they become publicly available very
soon after the observation date; furthermore, calibration images are
usually public immediately.

   As you see from the image above, the T80Cam images are large (9216 by
9232 pixels).  Therefore, to speed up the pointing testing, let's
down-sample the image by a factor of 10.  This step is optional and you
can safely use the full resolution, which will give you a more precise
coadd.  But it will be much slower (maybe good after you have an almost
final solution on the down-sampled image).  We will call the output
‘ref.fits’ (since it is the "reference" for our test).  We are putting
these two "input" files (to the script) in a dedicated directory to keep
the running directory clean (and be able to easily delete temporary/test
files for a fresh start with a '‘rm *.fits’').

     $ astwarp input/jplus-1050345.fits.fz --scale=1/10 -oinput/ref.fits

   For a first trial, let's create a cross-shaped pointing pattern with
5 points around M94, which is centered at its center on the RA and Dec
of 192.721250, 41.120556.  We'll center one exposure on the center of
the galaxy, and include 4 more exposures that are each 1 arc-minute away
along the RA and Dec axes.  To simplify the actual command later(3),
let's also include the column names in ‘pointing.txt’ through two lines
of metadata.  Also note that the ‘pointing.txt’ file can be made in any
manner you like, for example, by writing the coordinates manually on
your favorite text editor, or through another programming language or
logic, or etc.  Here, we are using AWK because it is sufficiently
powerful for this job, and it is a very small program that is available
on any Unix-based operating system (allowing you to easily run your
programs on any computer).

     $ step_arcmin=1
     $ center_ra=192.721250
     $ center_dec=41.120556

     $ echo "# Column 1: RA  [deg, f64] Right Ascension"  > pointing.txt
     $ echo "# Column 2: Dec [deg, f64] Declination"     >> pointing.txt

     $ echo $center_ra $center_dec \
            | awk '{s='$step_arcmin'/60; fmt="%-10.6f %-10.6f\n"; \
                    printf fmt, $1,   $2; \
                    printf fmt, $1+s, $2; \
                    printf fmt, $1,   $2+s; \
                    printf fmt, $1-s, $2; \
                    printf fmt, $1,   $2-s}' \
            >> pointing.txt

   With the commands below, let's have a look at the produced file,
first as plain-text, then with TOPCAT (which needs conversion to FITS).
After TOPCAT is opened, in the "Graphics" menu, select "Plane plot" to
see the five points in a flat RA, Dec plot.

     $ cat pointing.txt
     # Column 1: RA  [deg, f64] Right Ascension
     # Column 2: Dec [deg, f64] Declination
     192.721250 41.120556
     192.737917 41.120556
     192.721250 41.137223
     192.704583 41.120556
     192.721250 41.103889

     $ asttable pointing.txt -opointing.fits
     $ astscript-fits-view pointing.fits
     $ rm pointing.fits

   We are now ready to generate the exposure map of the pointing pattern
above using the reference image that we downloaded before.  Let's put
the center of our final coadd to be on the center of the galaxy, and
we'll assume the coadd has a size of 2 degrees.  With the second
command, you can see the exposure map of the final coadd.  Recall that
in this image, each pixel shows the number of input images that went
into it.

     $ astscript-pointing-simulate pointing.txt --output=coadd.fits \
                --img=input/ref.fits --center=$center_ra,$center_dec \
                --width=2

     $ astscript-fits-view coadd.fits

   You will see that except for a thin boundary, we have a depth of 5
exposures over the area of the single exposure.  Let's see what the
width of the deepest part of the image is.  First, we'll use Arithmetic
to set all pixels that contain less than 5 exposures (the outer pixels)
to NaN (Not a Number).  In the same Arithmetic command, let's trim all
the blank rows and columns, so the output only contains the pixels that
are exposed 5 times.  With the next command, let's view the deep region
and with the last command below, let's use the ‘--skycoverage’ option of
the Fits program to see the coverage of deep part on the sky.

     $ deep_thresh=5
     $ astarithmetic coadd.fits set-s s s $deep_thresh lt nan where trim \
                     --output=deep.fits

     $ astscript-fits-view deep.fits

     $ astfits deep.fits --skycoverage
     Input file: deep.fits (hdu: 1)

     Sky coverage by center and (full) width:
       Center: 192.72125      41.120556
       Width:  1.880835157    1.392461166

     Sky coverage by range along dimensions:
       RA       191.7808324    193.6616676
       DEC      40.42058203    41.81304319

   As we see, in declination, the width of this deep field is about 1.4
degrees.  Recall that RA is only defined on the equator and actual
coverage in RA depends on the declination due to the spherical nature of
the sky.  This area therefore nicely covers the expected outer parts of
M94.  On first thought, it may seem that we are now finished, but that
is not the case unfortunately!

   There is a problem: with a step size of 1 arc-minute, the brighter
central parts of this large galaxy will always be on very similar
pixels; making it hard to calibrate those pixels properly.  If you are
interested in the low surface brightness parts of this galaxy, it is
even worse: the outer parts of the galaxy will always cover similar
parts of the detector in all the exposures; and they cover a large area
on your image.  To be able to accurately calibrate the image (in
particular to estimate the flat field pattern and subtract the sky), you
do not want this to happen!  You want each exposure to cover very
different sources of astrophysical signal, so you can accurately
calibrate the artifacts created by the instrument or environment (for
example flat field) or of natural causes (for example the Sky).

   For an example of how these calibration issues can ruin low surface
brightness science, please see the image of M94 in the Legacy Survey
interactive viewer (https://www.legacysurvey.org/viewer).  After it is
loaded, at the bottom-left corner of the window, write "M94" in the box
of "Jump to object" and press ENTER. At first, M94 looks good with a
black background, but as you increase the "Brightness" (by scrolling it
to the right and seeing what is under the originally black pixels), you
will see the calibration artifacts clearly.

   ---------- Footnotes ----------

   (1) For full disclosure, Gnuastro is being developed at CEFCA (Centro
de Estudios de Física del Cosmos de Aragón); which also hosts OAJ.

   (2) For the full list of available filters, see the T80Cam
description (https://oaj.cefca.es/telescopes/t80cam).

   (3) Instead of this, later, when you called
‘astscript-pointing-simulate’, you could pass the ‘--racol=1’ and
‘--deccol=2’ options.  But having metadata is always preferred (will
avoid many bugs/frustrations in the long-run!).


File: gnuastro.info,  Node: Area of non-blank pixels on sky,  Next: Script with pointing simulation steps so far,  Prev: Preparing input and generating exposure map,  Up: Pointing pattern design

2.8.2 Area of non-blank pixels on sky
-------------------------------------

In *note Preparing input and generating exposure map:: we generated a
pointing pattern with very small steps, showing how this can cause
calibration problems.  Later (in *note Larger steps sizes for better
calibration::) using larger steps is discussed.  In this section, let's
see how we can get an accurate measure of the area that is covered in a
certain depth.

   A first thought would be to simply multiply the widths along RA and
Dec reported before: $1.8808\times1.3924=2.6189$ degrees squared.  But
there are several problems with this:
   • It ignores the fact that RA only has units of degrees on the
     equator: at different declinations, differences in RA should be
     converted to degrees.  This is discussed further in this tutorial:
     *note Pointings that account for sky curvature::.
   • It doesn't take into account the thin rows/columns of blank pixels
     (NaN) that are on the four edges of the ‘deep.fits’ image.
   • The differing area of the pixels on the spherical sky in relation
     to those blank values can result in wrong estimations of the area.

   Let's get a very accurate estimation of the area that will not be
affected by the issues above.  With the first command below, we'll use
the ‘--pixelareaonwcs’ option of the Fits program that will return the
area of each pixel (in pixel units of degrees squared).  After running
the second command, please have a look at the produced image.

     $ astfits deep.fits --pixelareaonwcs --output=deep-pix-area.fits

     $ astfits deep.fits --pixelscale
     Basic info. for --pixelscale (remove extra info with '--quiet' or '-q')
       Input: deep.fits (hdu 1) has 2 dimensions.
       Pixel scale in each FITS dimension:
         1: 0.00154403 (deg/pixel) = 5.5585 (arcsec/pixel)
         2: 0.00154403 (deg/pixel) = 5.5585 (arcsec/pixel)
       Pixel area:
         2.38402e-06 (deg^2) = 30.8969 (arcsec^2)

     $ astscript-fits-view deep-pix-area.fits

   You see a donut-like shape in DS9.  Move your mouse over the central
(white) region of the region and look at the values.  You will see that
the pixel area (in degrees squared) is exactly the same as we saw in the
output of ‘--pixelscale’.  As you move your mouse away to other colors,
you will notice that the area covered by each pixel (its value in this
image) deceases very slightly (in the 5th decimal!).  This is the effect
of the Gnomonic projection
(https://en.wikipedia.org/wiki/Gnomonic_projection); summarized as ‘TAN’
(for "tangential") in the FITS WCS standard, the most commonly used in
optical astronomical surveys and the default in this script.

   Having ‘deep-pix-area.fits’, we can now use Arithmetic to set the
areas of all the pixels that were NaN in ‘deep.fits’ and sum all the
values to get an accurate estimate of the area we get from this pointing
pattern:

     $ astarithmetic deep-pix-area.fits deep.fits isblank nan where -g1 \
                     sumvalue --quiet
     1.93836806631634e+00

   Therefore, the actual area that is covered is less than the simple
multiplication above.  At these declinations, the dominant cause of this
difference is the first point above (that RA needs correction), this
will be discussed in more detail later in this tutorial (see *note
Pointings that account for sky curvature::).  Generally, using this
method to measure the area of your non-NAN pixels in an image is very
easy and robust (automatically takes into account the curvature,
coordinate system, projection and blank pixels of the image).


File: gnuastro.info,  Node: Script with pointing simulation steps so far,  Next: Larger steps sizes for better calibration,  Prev: Area of non-blank pixels on sky,  Up: Pointing pattern design

2.8.3 Script with pointing simulation steps so far
--------------------------------------------------

In *note Preparing input and generating exposure map:: and *note Area of
non-blank pixels on sky::, the basic steps to simulate a pointing
pattern's exposure map and measure the final output area on the sky
where described in detail.  From this point on in the tutorial, we will
be experimenting with the shell variables that were set above, but the
actual commands will not be changed regularly.  If a change is necessary
in a command, it is clearly mentioned in the text.

   Therefore, it is better to write the steps above (after downloading
the reference image) as a script.  In this way, you can simply change
those variables and see the final result fast by running your script.
For more on writing scripts, see as described in *note Writing scripts
to automate the steps::.

   Here is a summary of some points to remember when transferring the
code in the sections before into a script:

   • Where the commands are edited/changed, please also update them in
     your script.
   • Keep all the variables at the top, even if they are used later.
     This allows to easily view or changed them without digging into the
     script.
   • You do not need to include visual check commands like the
     ‘astscript-fits-view’ or ‘cat’ commands above.  Those can be run
     interactively after your script is finished; recall that a script
     is for batch (non-interactive) processing.
   • Put all your intermediate products inside a "build" directory.

   Here is the script that summarizes the steps in *note Preparing input
and generating exposure map:: (after download) and *note Area of
non-blank pixels on sky:::

#!/bin/bash
#
# Copyright (C) 2024-2025 Mohammad Akhlaghi <mohammad@akhlaghi.org>
#
# Copying and distribution of this file, with or without modification,
# are permitted in any medium under the GNU GPL v3+, without royalty
# provided the copyright notice and this notice are preserved.  This
# file is offered as-is, without any warranty.

# Parameters of the script
deep_thresh=5
step_arcmin=1
center_ra=192.721250
center_dec=41.120556

# Input and build directories (can be anywhere in your file system)
indir=input
bdir=build

# Abort the script in case of an error.
set -e

# Make the build directory if it doesn't already exist.
if ! [ -d $bdir ]; then mkdir $bdir; fi

# Build the 5-pointing pointing pattern (with the step size above).
pointingcat=$bdir/pointing.txt
echo "# Column 1: RA  [deg, f64] Right Ascension"  > $pointingcat
echo "# Column 2: Dec [deg, f64] Declination"     >> $pointingcat
echo $center_ra $center_dec \
            | awk '{s='$step_arcmin'/60; fmt="%-10.6f %-10.6f\n"; \
                    printf fmt, $1,   $2; \
                    printf fmt, $1+s, $2; \
                    printf fmt, $1,   $2+s; \
                    printf fmt, $1-s, $2; \
                    printf fmt, $1,   $2-s}' \
            >> $pointingcat

# Simulate the pointing pattern.
coadd=$bdir/coadd.fits
astscript-pointing-simulate $pointingcat --output=$coadd \
           --img=input/ref.fits --center=$center_ra,$center_dec \
           --width=2

# Trim the regions shallower than the threshold.
deep=$bdir/deep.fits
astarithmetic $coadd set-s s s $deep_thresh lt nan where trim \
               --output=$deep

# Calculate the area of each pixel on the curved celestial sphere:
pixarea=$bdir/deep-pix-area.fits
astfits $deep --pixelareaonwcs --output=$pixarea

# Report the final area (the empty 'echo's are for visual help in outputs)
echo; echo
echo "Area with step of $step_arcmin arcminutes, at $deep_thresh depth:"
astarithmetic $pixarea $deep isblank nan where -g1 \
              sumvalue --quiet

   For a description of how to make it executable and how to run it, see
*note Writing scripts to automate the steps::.  Note that as you start
adding your own text to the script, be sure to add your name (and year
that you modified) in the copyright notice at the start of the script
(this is very important!).


File: gnuastro.info,  Node: Larger steps sizes for better calibration,  Next: Pointings that account for sky curvature,  Prev: Script with pointing simulation steps so far,  Up: Pointing pattern design

2.8.4 Larger steps sizes for better calibration
-----------------------------------------------

In *note Preparing input and generating exposure map:: we saw that a
small pointing pattern is not good for the reduction of data from a
large object like M94!  M94 is about half a degree in diameter; so let's
set ‘step_arcmin=15’.  This is one quarter of a degree and will put the
center of the four exposures on the four corners of the M94's main ring.
Furthermore, *note Script with pointing simulation steps so far::, the
steps were summarized into a script to allow easy changing of variables
without manually re-entering the individual/separate commands.

   After you change ‘step_arcmin=15’ and re-run the script, you will get
a total area (from counting of per-pixel areas) of approximately 0.96
degrees squared.  This is just roughly half the previous area and will
barely fit M94!  To understand the cause, let's have a look at the full
coadd (not just the deepest area):

     $ astscript-fits-view build/coadd.fits

   Compared to the first run (with ‘step_arcmin=1’), we clearly see how
there are indeed fewer pixels that get photons in all 5 exposures.  As
the area of the deepest part has decreased, the areas with fewer
exposures have also grown.  Let's define our deep region to be the
pixels with 3 or more exposures.  Please set ‘deep_thresh=3’ in the
script and re-run it.  You will see that the "deep" area is now almost
2.02 degrees squared!  This is (slightly) larger than the first run
(with ‘step_arcmin=1’)!

   The difference between 3 exposures and 5 exposures seems a lot at
first.  But let's calculate how much it actually affects the achieved
signal-to-noise ratio and the surface brightness limit.  The surface
brightness limit (or upper-limit surface brightness) are both calculated
by applying the definition of magnitude to the standard deviation of the
background.  So we should first calculate how much this difference in
depth affects the sky standard deviation.  For a complete discussion on
the definition of the surface brightness limit, see *note
Metameasurements on full input::.

   Deep images will usually be dominated by *note Photon counting
noise:: (or Poisson noise).  Therefore, if a single exposure image has a
sky standard deviation of $\sigma_s$, and we combine $N$ such exposures
by taking their mean, the final/coadded sky standard deviation
($\sigma$) will be $\sigma=\sigma_s/\sqrt{N}$.  As a result, the surface
brightness limit between the regions with $N$ exposures and $M$
exposures differs by $2.5\times log_{10}(\sqrt{N/M}) = 1.25\times
log_{10}(N/M)$ magnitudes.  If we set $N=3$ and $M=5$, we get a surface
brightness magnitude difference of 0.28!

   This is a very small difference; given all the other sources of error
that will be present; but how much it improves the calibration
artifacts.  Therefore at the cost of decreasing our surface brightness
limit by 0.28 magnitudes, we are now able to calibrate the individual
exposures much better, and even cover a larger area!

   The argument above didn't involve any image and was primarily
theoretical.  For the more visually-inclined readers, let's add raw
Gaussian noise (with a $\sigma$ of 100 counts) over each simulated
exposure.  We will then instruct ‘astscript-pointing-simulate’ to coadd
them as we would coadd actual data (by taking the sigma-clipped mean).
The command below is identical to the previous call to the pointing
simulation script with the following differences.  Note that this is
just for demonstration, so you should not include this in your script
(unless you want to see the noisy coadd every time; at double the
processing time).

‘--output’
     We are using a different output name, so we can compare the output
     of the new command with the previous one.

‘--coadd-operator’
     This should be one of the Arithmetic program's *note Coadding
     operators::.  By default the value is ‘sum’; because by default,
     each pixel of each exposure is given a value of 1.  When coadding
     is defined through the summation operator, we can obtain the
     exposure map that you have already seen above.

     But in this run, we are adding noise to each input exposure
     (through the hook that is described below) and coadding them (as we
     would coadd actual science images).  Since the purpose differs
     here, we are using this option to change the operator.

‘--hook-warp-after’
     This is the most visible difference of this command the previous
     one.  Through a "hook", you can give any arbitrarily long (series
     of) command(s) that will be added to the processing of this script
     at a certain location.  This particular hook gets applied "after"
     the "warp"ing phase of each exposure (when the pixels of each
     exposure are mapped to the final pixel grid; but not yet coadded).

     Since the script runs in parallel (the actual work-horse is a
     Makefile!), you can't assume any fixed file name for the input(s)
     and output.  Therefore the inputs to, and output(s) of, hooks are
     some pre-defined shell variables that you should use in the
     command(s) that you hook into the processing.  They are written in
     full-caps to be clear and separate from your own variables.  In
     this case, they are the ‘$WARPED’ (input file of the hook) and
     ‘$TARGET’ (output name that next steps in the script will operate
     on).  As you see from the command below, through this hook we are
     calling the Arithmetic program to add noise to all non-zero pixels
     in the warped image.  For more on the noise-adding operators, see
     *note Random number generators::.

     $ center_ra=192.721250
     $ center_dec=41.120556
     $ astscript-pointing-simulate build/pointing.txt --img=input/ref.fits \
                --center=$center_ra,$center_dec \
                --width=2 --coadd-operator="3 0.2 sigclip-mean swap free" \
                --output=build/coadd-noised.fits \
                --hook-warp-after='astarithmetic $WARPED set-i \
                                               i i 0 uint8 eq nan where \
                                               100 mknoise-sigma \
                                                --output=$TARGET'

     $ astscript-fits-view build/coadd.fits build/coadd-noised.fits

   When you visually compare the two images of the last command above,
you will see that (at least by eye) it is almost impossible to
distinguish the differing noise pattern in the regions with 3 exposures
from the regions with 5 exposures.  But the regions with a single
exposure are clearly visible!  This is because the surface brightness
limit in the single-exposure regions is $1.25\times\log_{10}(1/5)=-0.87$
magnitudes brighter.  This almost one magnitude difference in surface
brightness is significant and clearly visible in the coadded image
(recall that magnitudes are measured in a logarithmic scale).

   Thanks to the argument above, we can now have a sufficiently large
area with a usable depth.  However, each the center of each pointing
will still contain the central part of the galaxy.  In other words, M94
will be present in all the exposures while doing the calibrations.  Even
in not-too-deep observations, we already see a large ring around this
galaxy.  When we do a low surface brightness optimized reduction, there
is a good chance that the size of the galaxy is much larger than that
ring.  This very extended structure will make it hard to do the
calibrations on very accurate scales.  Accurate calibration is necessary
if you do not want to loose the faint photons that have been recorded in
your exposures.

*Calibration is very important:* Better calibration can result in a
fainter surface brightness limit than more exposures with poor
calibration; especially for very low surface brightness signal that
covers a large area and is systematically affected by calibration
issues.

   Ideally, you want your target to be on the four edges/corners of each
image.  This will make sure that a large fraction of each exposure will
not be covered by your final target in each exposure, allowing you to
calibrate much more accurately.


File: gnuastro.info,  Node: Pointings that account for sky curvature,  Next: Accounting for non-exposed pixels,  Prev: Larger steps sizes for better calibration,  Up: Pointing pattern design

2.8.5 Pointings that account for sky curvature
----------------------------------------------

In *note Larger steps sizes for better calibration::, we saw how a small
loss in surface brightness limit can allow better calibration and even a
larger area.  Let's extend this by setting ‘step_arcmin=40’ (almost half
the width of the detector) inside your script (see *note Script with
pointing simulation steps so far::).  After running the script with this
change, take a look at ‘build/deep.fits’:

     $ astscript-fits-view build/deep.fits --ds9scale=minmax

   You will see that the region with 5 exposure depth is a horizontally
elongated rectangle now!  Also, the vertical component of the cross with
four exposures is much thicker than the horizontal component!  Where
does this asymmetry come from?  All the steps in our pointing strategy
had the same (fixed) size of 40 arc minutes.

   This happens because the same change in RA and Dec (defined on the
curvature of a sphere) will result in different absolute changes on the
equator.  To visually see this, let's look at the pointing positions in
TOPCAT:

     $ cat build/pointing.txt
     # Column 1: RA  [deg, f64] Right Ascension
     # Column 2: Dec [deg, f64] Declination
     192.721250 41.120556
     193.387917 41.120556
     192.721250 41.787223
     192.054583 41.120556
     192.721250 40.453889

     $ asttable build/pointing.txt -obuild/pointing.fits
     $ astscript-fits-view build/pointing.fits

   After TOPCAT opens, under the "graphics" window, select "Plane Plot".
In the newly opened window, click on the "Axes" item on the bottom-left
list of items.  Then activate the "Aspect lock" box so the vertical and
horizontal axes have the same scaling.  You will see what you expect
from the numbers: we have a beautifully symmetric set of 5 points shaped
like a '+' sign.

   Keep the previous window, and let's go back to the original TOPCAT
window.  In the first TOPCAT window, click on "Graphics" again, but this
time, select "Sky plot".  You will notice that the vertical component of
the cross is now longer than the horizontal component!  If you zoom-out
(by scrolling your mouse over the plot) a lot, you will see that this is
actually on the spherical surface of the sky!  In other words, as you
see here, on the sky, the horizontal points are closer to each other
than the vertical points; causing a larger overlap between them, making
the vertical overlap thicker in ‘build/pointing.fits’.

   On the celestial sphere, only the declination is measured in degrees.
In other words, the difference in declination of two points can be
calculated only with their declination.  However, except for points that
are on the equator, differences in right ascension depend on the
declination.  Therefore, the origin of this problem is that we done the
additions and subtractions for defining the pointing points in a flat
space: based on the step size in arc minutes that was applied similarly
on RA and Dec (in *note Preparing input and generating exposure map::).

   To fix this problem, we need to convert our points from the flat
RA/Dec into the spherical RA/Dec.  In the FITS standard, we have the
"World Coordinate System" (WCS) that defines this type of conversion,
using pre-defined projections in the ‘CTYPEi’ keyword (short for for
"Coordinate TYPE in dimension i").  Let's have a look at the coadd to
see the default projection of our final coadd:

$ astfits build/coadd.fits -h1 | grep CTYPE
CTYPE1  = 'RA---TAN'           / Right ascension, gnomonic projection
CTYPE2  = 'DEC--TAN'           / Declination, gnomonic projection

   We therefore see that the default projection of our final coadd is
the ‘TAN’ (short for "tangential") projection, which is more formally
known as the Gnomonic projection
(https://en.wikipedia.org/wiki/Gnomonic_projection).  This is the most
commonly used projection in optical astronomy.  Now that we know the
final projection, we can do this conversion using Table's column
arithmetic operator ‘eq-j2000-from-flat’ like below:

$ pointingcat=build/pointing.txt
$ pointingonsky=build/pointing-on-sky.fits
$ asttable $pointingcat --output=$pointingonsky \
           -c'arith RA          set-r \
                    DEC         set-d \
                    r meanvalue set-ref-r \
                    d meanvalue set-ref-d \
                    r d ref-r ref-d TAN eq-j2000-from-flat' \
           --colmetadata=1,RA,deg,"Right ascension" \
           --colmetadata=2,Dec,deg,"Declination"

$ astscript-fits-view build/pointing-on-sky.fits

   Here is a break-down of the first command above: to do the
flat-to-sky conversion, we need a reference point (where the two are
equal).  We have used the mean RA and mean Dec (through the ‘meanvalue’
operator in Arithmetic) as our reference point (which are placed in the
‘ref-r’ and ‘red-d’ variables.  After calling the ‘eq-j2000-from-flat’
operator, we have just added metadata to the two columns.

   To confirm that this operator done the job correctly, after the
second command above, repeat the same experiment as before with TOPCAT
(where you viewed the pointing positions on a flat and spherical
coordinate system).  You will see that indeed, on the sphere you have a
'+' shape, but on the flat plot, it looks stretched.

*Script update 1:* you should now add the ‘pointingonsky’ definition and
the ‘asttable’ command above into the script of *note Script with
pointing simulation steps so far::.  They should be placed before the
call to ‘astscript-pointing-simulate’.  Also, in the call to
‘astscript-pointing-simulate’, ‘$pointingcat’ should be replaced with
‘$pointingonsky’ (so it doesn't use the flat RA, Dec pointings).

   After implementing this change in your script and running it, open
‘deep.fits’ and you will see that the widths of both the horizontal and
vertical regions are much more similar.  The top of the vertical overlap
is slightly wider than the bottom, but that is something you can't fix
by just pointing (your camera's field of view is fixed on the sky!).  It
can be correctly by slightly rotating some of the exposures, but that
will result in different PSFs from one exposure to another; and this can
cause more important problems for your final science.

*Plotting the spherical RA and Dec in your papers:* The inverse of the
‘eq-j2000-from-flat’ operator above is the ‘eq-j2000-to-flat’.
‘eq-j2000-to-flat’ can be used when you want to plot a set points with
spherical RA and Dec in a paper.  When the minimum and maximum RA and
Dec differ by larger than half a degree, you'll clearly see the
difference.  For more, see the description of these operators in *note
Column arithmetic::.

   Try to slightly increase ‘step_arcmin’ to make the cross-like region
with 4 exposures as thin as possible.  For example, set it to
‘step_arcmin=42’.  When you open ‘deep.fits’, you will see that the
depth across this image is almost contiguous (which is another positive
factor!).  Try increasing it to 43 arc minutes to see that the central
cross will become almost fully NaN in ‘deep.fits’ (which is bad!).

   You will notice that the vertical region of 4 exposure depth is
thinner in the bottom than on the top.  This is due to the RA/Dec change
above, but across the width of the image.  We can't therefore change
this by just changing the position of the pointings, we need to rotate
some of the exposures if we want it to be removed.  But rotation is not
yet implemented in this script.

   You can construct any complex pointing pattern (with more than 5
points and in any shape) based on the logic and reasoning above to help
extract the most science from the valuable telescope time that you will
be getting.  Since the output is a FITS file, you can easily download
another FITS file of your target, open it with DS9 (and "lock" the
"WCS") with the coadd produced by this simulation to make sure that the
deep parts correspond to the area of interest for your science case.

   Factors like the optimal exposure time are also critical for the
final result(1), but is was beyond the scope of this tutorial.  One
relevant factor however is the effect of vignetting: the pixels on the
outer extremes of the field of view that are not exposed to light and
should be removed from your final coadd.  They effect your pointing
pattern: by decreasing your total area, they act like a larger spacing
between your points, causing similar shallow crosses as you saw when you
set ‘step_arcmin’ to 43 arc minutes.  In *note Accounting for
non-exposed pixels::, we will show how this can be done within the same
test concept that we done here.

   ---------- Footnotes ----------

   (1) The exposure time will determine the Signal-to-noise ration on a
single exposure level.


File: gnuastro.info,  Node: Accounting for non-exposed pixels,  Prev: Pointings that account for sky curvature,  Up: Pointing pattern design

2.8.6 Accounting for non-exposed pixels
---------------------------------------

At the end of *note Pointings that account for sky curvature:: we were
able to maximize the region of same depth in our coadd.  But we noticed
that issues like strong vignetting
(https://en.wikipedia.org/wiki/Vignetting) can create discontinuity in
our final coadded data product.  In this section, we'll review the steps
to account for such effects.  Generally, the full area of a detector is
not usually used in the final coadd.  Vignetting is one cause, it can be
due to other problems also.  For example due to baffles in the optical
path (to block stray light), or large regions of bad (unusable or
"dead") pixels that may be in any place on the detector(1).

   Without accounting for these pixels that do not receive any light,
the deep area we measured in the sections above will be over-estimated.
In this sub-section, let's review the necessary additions to account for
such artifacts.  Therefore, before continuing, please make sure that you
have already read and applied the steps of the previous sections (this
sub-section builds upon that section).

   Vignetting strongly depends on the optical design of the instrument
you are using.  It can be a constant number of pixels on all the edges
the detector, or it can have a more complex shape.  For example on
cameras that have multiple detectors on the field of view, in this case,
the regions to exclude on each detector can be very different and will
not be symmetric!

   Therefore, within Gnuastro's ‘astscript-pointing-simulate’ script
there is no parameter for pre-defined vignetting shapes.  Instead, you
should define a mask that you can apply on each exposure through the
provided hook (‘--hook-warp-before’; recall that we previously used
another hook in *note Larger steps sizes for better calibration::).
Through the mask, you are free to set any vignetted or bad pixel to NaN
(thus ignoring them in the coadd) and applying it in any way that best
suites your instrument and detector.

   The mask image should be same size as the reference image, but only
containing two values: 0 or 1.  Pixels in each exposure that have a
value of 1 in the mask will be set to NaN before the coadding process
and will not contribute to the final coadd.  Ideally, you can use the
master flat field image of the previous reductions to create this mask:
any pixel that has a low sensitivity in the master flat (for any reason)
can be set to 1, and the rest of the pixels to 0.

   Let's build a simple mask by assuming that we only have strong
vignetting that is affecting the outer 30 arc seconds of the individual
exposures.  To mask the outer edges of an image we can use Gnuastro's
Arithmetic program; and in particular, the ‘indexonly’ operator.  To
learn more about this operator, see *note Size and position operators::.

   But before doing that, we need convert this angular distance to
pixels on the detector.  In *note Pointing pattern design::, we used an
undersampled version of the input image, so we should do this conversion
on that image:

     $ margin_arcsec=30
     $ margin_pix=$(astfits input/ref.fits --pixelscale --quiet \
                          | awk '{print int('$margin_arcsec'/($1*3600))}')
     $ echo $margin_pix
     5

   To build the mask, we can now follow the recipe under "Image: masking
margins" of the ‘index’ operator in Arithmetic (for a full description
of what this command is doing(2), see *note Size and position
operators::).  Finally, in the last command, let's look at the mask
image in the "red" color map of DS9 (which will shows the thin 1-valued
pixels to mask on the border more clearly).

     $ width=$(astfits  input/ref.fits --keyvalue=NAXIS1 -q)
     $ height=$(astfits input/ref.fits --keyvalue=NAXIS2 -q)

     $ astarithmetic input/ref.fits indexonly     set-i \
                     $width         uint16        set-w \
                     $height        uint16        set-h \
                     $margin_pix    uint16        set-m \
                     i w %          uint16        set-X \
                     i w /          uint16        set-Y \
                     X m lt         X w m - gt       or \
                     Y m lt         Y h m - gt       or \
                     or --output=build/mask.fits

     $ astscript-fits-view build/mask.fits --ds9extra="-cmap red"

   We are now ready to run the main pointing simulate script.  With the
command below, we will use the ‘--hook-warp-before’ to apply this mask
on the image of each exposure just before warping.  The concept of this
hook is very similar to that of ‘--hook-warp-after’ in *note Pointing
pattern design::.  As the name suggests, this hook is applied "before"
the warping.  The input to the command given to this hook should be
called with ‘$EXPOSURE’ and the output should be called with ‘$TOWARP’.
With the second command, let's compare the two outputs:

     $ astscript-pointing-simulate build/pointing-on-sky.fits \
                --output=build/coadd-with-trim.fits --img=input/ref.fits \
                --center=$center_ra,$center_dec --width=2 \
                --hook-warp-before='astarithmetic $EXPOSURE build/mask.fits \
                                                  nan where -g1 -o$TOWARP'

     $ astscript-fits-view build/coadd.fits build/coadd-with-trim.fits

   As expected, due to the smaller area of the detector that is exposed
to photons, the regions with 4 exposures have become much thinner and on
the bottom, it has been removed.  To have contiguous depth in the deeper
region, use this new call in your script and decrease the
‘step_arcmin=41’.

   You can use the same command on a mask that is created in any way and
as realistic as possible.  More generically, you can use the before and
after hooks for any other operation; for example to insert objects from
a catalog using *note MakeProfiles:: as well as adding noise as we did
in *note Pointing pattern design::.

   Therefore it is also good to add the mask and its application in your
script.  This should be pretty easy by now (following *note Script with
pointing simulation steps so far:: and the "Script update 1" box of
*note Pointings that account for sky curvature::).  So we will leave
this as an exercise.

   ---------- Footnotes ----------

   (1) For an example of bad pixels over the detector, see Figures 4 and
6 of Instrument Science Report WFC3 2019-03
(https://www.stsci.edu/files/live/sites/www/files/home/hst/instrumentation/wfc3/documentation/instrument-science-reports-isrs/_documents/2019/WFC3-2019-03.pdf)
by the Space Telescope Science Institute.

   (2) By learning how this command works, you can customize it.  For
example, to mask different widths along each separate edge: it often
happens that the left/right or top/bottom edges are affected differently
by vignetting.


File: gnuastro.info,  Node: Moire pattern in coadding and its correction,  Next: Clipping outliers,  Prev: Pointing pattern design,  Up: Tutorials

2.9 Moiré pattern in coadding and its correction
================================================

After warping some images with the default mode of Warp (see *note Align
pixels with WCS considering distortions::) you may notice that the
background noise is no longer flat.  Some regions will be smoother and
some will be sharper; depending on the orientation and distortion of the
input/output pixel grids.  This is due to the Moiré pattern
(https://en.wikipedia.org/wiki/Moir%C3%A9_pattern), which is especially
noticeable/significant when two slightly different grids are
super-imposed.

   With the commands below, we'll download a single exposure image from
the J-PLUS survey (https://www.j-plus.es) and run Warp (on a $8\times8$
arcmin$^2$ region to speed it up the demos here).  Finally, we'll open
the image to visually see the artificial Moiré pattern on the warped
image.

     ## Download the image (73.7 MB containing an 9216x9232 pixel image)
     $ jplusdr2=http://archive.cefca.es/catalogues/vo/siap/jplus-dr2/reduced
     $ wget $jplusdr2/get_fits?id=771463 -Ojplus-exp1.fits.fz

     ## Align a small part of it with the sky coordinates.
     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits

     ## Open the aligned region with DS9
     $ astscript-fits-view jplus-e1.fits

   In the opened DS9 window, you can see the Moiré pattern as wave-like
patterns in the noise: some parts of the noise are more smooth and some
parts are more sharp.  Right in the center of the image is a blob of
sharp noise.  Warp has the ‘--checkmaxfrac’ option for direct inspection
of the Moiré pattern (described with the other options in *note Align
pixels with WCS considering distortions::).  When run with this option,
an extra HDU (called ‘MAX-FRAC’) will be added to the output.  The image
in this HDU has the same size as the output.  However, each output pixel
will contain the largest (maximum) fraction of area that it covered over
the input pixel grid.  So if an output pixel has a value of 0.9, this
shows that it covered $90\%$ of an input pixel.  Let's run Warp with
‘--checkmaxfrac’ and see the output (after DS9 opens, in the "Cube"
window, flip between the first and second HDUs):

     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --checkmaxfrac

     $ astscript-fits-view jplus-e1.fits

   By comparing the first and second HDUs/extensions, you will clearly
see that the regions with a sharp noise pattern fall exactly on parts of
the ‘MAX-FRAC’ extension with values larger than 0.5.  In other words,
output pixels where one input pixel contributed more than half of the
its value.  As this fraction increases, the sharpness also increases
because a single input pixel's value dominates the value of the output
pixel.  On the other hand, when this value is small, we see that many
input pixels contribute to that output pixel.  Since many input pixels
contribute to an output pixel, it acts like a convolution, hence that
output pixel becomes smoother (see *note Spatial domain convolution::).
Let's have a look at the distribution of the ‘MAX-FRAC’ pixel values:

     $ aststatistics jplus-e1.fits -hMAX-FRAC
     Statistics (GNU Astronomy Utilities) 0.24
     -------
     Input: jplus-e1.fits (hdu: MAX-FRAC)
     -------
       Number of elements:                      744769
       Minimum:                                 0.250213461
       Maximum:                                 0.9987495374
       Mode:                                    0.5034223567
       Mode quantile:                           0.3773819498
       Median:                                  0.5520805544
       Mean:                                    0.5693956458
       Standard deviation:                      0.1554693738
     -------
     Histogram:
      |                      ***
      |                   **********
      |                 *****************
      |              ************************
      |           *******************************
      |         **************************************
      |       *********************************************
      |     ****************************************************
      |   ***********************************************************
      | ******************************************************************
      |**********************************************************************
      |----------------------------------------------------------------------

   The smallest value is 0.25 (=1/4), showing that 4 input pixels
contributed to the output pixels value.  While the maximum is almost
1.0, showing that a single input pixel defined the output pixel value.
You can also see that the most probable value (the mode) is 0.5, and
that the distribution is positively skewed.

   This is a well-known problem in astronomical imaging and professional
photography.  If you only have a single image (that is already taken!),
you can undersample the input: set the angular size of the output pixels
to be larger than the input.  This will decrease the resolution of your
image, but will ensure that pixel-mixing will always happen.  In the
example below we are setting the output pixel scale (which is known as
‘CDELT’ in the FITS standard) to $1/0.5=2$ of the input's.  In other
words each output pixel edge will cover double the input pixel's edge on
the sky, and the output's number of pixels in each dimension will be
half of the previous output.

     $ cdelt=$(astfits jplus-exp1.fits.fz --pixelscale -q \
                       | awk '{print $1}')
     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --cdelt=$cdelt/0.5 \
               --checkmaxfrac

   In the first extension, you can hardly see any Moiré pattern in the
noise.  When you go to the next (‘MAX-FRAC’) extension, you will see
that almost all the pixels have a value of 1.  Of course, decreasing the
resolution by half is a little too drastic.  Depending on your image,
you may be able to reach a sufficiently good result without such a
drastic degrading of the input image.  For example, if you want an
output pixel scale that is just 1.5 times larger than the input, you can
divide the original coordinate-delta (or "cdelt") by $1/1.5=0.6666$ and
try again.  In the ‘MAX-FRAC’ extension, you will see that the range of
pixel values is now between 0.56 to 1.0 (recall that originally, this
was between 0.25 and 1.0).  This shows that the pixels are more
similarly mixed and in fact, when you look at the actual warped image,
you can hardly distinguish any Moiré pattern in the noise.

   However, deep astronomical data are usually built by several
exposures (images), not a single one.  Each image is also taken by
(slightly) shifting the telescope compared to the previous exposure.
This shift is known as "dithering" or a "pointing pattern", see *note
Pointing pattern design::.  We do this for many reasons (for example
tracking errors in the telescope, high background values, removing the
effect of bad pixels or those affected by cosmic rays, robust flat
pattern measurement, etc.(1)).  One of those "etc."  reasons is to
correct the Moiré pattern in the final coadded deep image.

   The Moiré pattern is fixed to the grid of the image, slightly
shifting the telescope will result in the pattern appearing in different
parts of the sky.  Therefore when we later coadd, or coadd, the separate
exposures into a deep image, the Moiré pattern will be decreased there.
However, dithering has possible drawbacks based on the scientific goal.
For example when observing time-variable phenomena where cutting the
exposures to several shorter ones is not feasible.  If this is not the
case for you (for example in galaxy evolution), continue with the rest
of this section.

   Because we have multiple exposures that are slightly (sub-pixel)
shifted, we can also increase the spatial resolution of the output.  For
example, let's set the output coordinate-delta (‘--cdelt’, or pixel
scale) to be 1/2 of the input.  In other words, the number of pixels in
each dimension of the output is double the first Warp command of this
section:

     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --cdelt=$cdelt/2 \
               --checkmaxfrac

     $ aststatistics jplus-e1.fits -hMAX-FRAC --minimum --maximum
     6.26360438764095e-02 2.50680270139128e-01

     $ astscript-fits-view jplus-e1.fits

   From the last command, you see that like the previous change in
‘--cdelt’, the range of ‘MAX-FRAC’ has decreased.  However, when you
look at the warped image and the ‘MAX-FRAC’ image with the last command,
you still visually see the Moiré pattern in the noise (although it has
significantly decreased compared to the original resolution).  It is
still present because 2 is an exact multiple of 1.  Let's try increasing
the resolution (oversampling) by a factor of 1.25 (which isn't an exact
multiple of 1):

     $ astwarp jplus-exp1.fits.fz --center=107.62920,39.72472 \
               --width=8/60 -ojplus-e1.fits --cdelt=$cdelt/1.25 \
               --checkmaxfrac
     $ astscript-fits-view jplus-e1.fits

   You don't see any Moiré pattern in the noise any more, but when you
look at the ‘MAX-FRAC’ extension, you see it is very different from the
ones you had seen before.  In the previous ‘MAX-FRAC’ image, you could
see large blobs of similar values.  But here, you see that the variation
is almost on a pixel scale, and the difference between one pixel to the
next is not significant.  This is why you don't see any Moiré pattern in
the warped image.

   In J-PLUS, each part of the sky was observed with a three-point
pointing pattern (very small shifts in each pointing).  Let's download
the other two exposures and warp the same region of the sky to the same
pixel grid (using the ‘--gridfile’ feature).  Then, let's open all three
warped images in one DS9 instance:

     $ wget $jplusdr2/get_fits?id=771465 -Ojplus-exp2.fits.fz
     $ wget $jplusdr2/get_fits?id=771467 -Ojplus-exp3.fits.fz

     $ astwarp jplus-exp2.fits.fz --gridfile jplus-e1.fits \
               -o jplus-e2.fits --checkmaxfrac
     $ astwarp jplus-exp3.fits.fz --gridfile jplus-e1.fits \
               -o jplus-e3.fits --checkmaxfrac

     $ astscript-fits-view jplus-e*.fits

In the three warped images, you don't see any Moiré pattern, so far so
good...  now, take the following steps:
  1. In the small "Cube" window, click the "Next" button so you see the
     ‘MAX-FRAC’ extension/HDU.
  2. Click on the "Frame" button (in the top row of buttons just on top
     of the image), and select the "Single" button in the bottom row.
  3. Open the "Zoom" menu (not button), and select "Zoom 16".
  4. Press the <TAB> key to flip through each exposure.
  5. Focus your eyes on the pixels with the largest value (white colored
     pixels), while pressing <TAB> to flip between the exposures.  You
     will see that in each exposure they cover different pixels (nicely
     getting averaged out after coadding).

   The exercise above shows that the Moiré pattern (that had already
decreased significantly) will be further decreased after we coadd the
images.  So let's coadd these three images with the commands below.
First, we need to remove the sky-level from each image using *note
NoiseChisel::, then we'll coadd the ‘INPUT-NO-SKY’ extensions using
filled MAD-clipping (to reject outliers, and especially diffuse
outliers, robustly, see *note Clipping outliers::).

     $ for i in $(seq 3); do \
        astnoisechisel jplus-e$i.fits -ojplus-nc$i.fits; \
       done

     $ astarithmetic jplus-nc*.fits 3 5 0.2 sigclip-mean swap free \
                     -gINPUT-NO-SKY -ojplus-coadd.fits

     $ astscript-fits-view jplus-nc*.fits jplus-coadd.fits

After opening the individual exposures and the final coadd with the last
command, take the following steps to see the comparisons properly:
  1. Click on the coadd image so it is selected.
  2. Go to the "Frame" menu, then the "Lock" item, then activate "Scale
     and Limits".
  3. Scroll your mouse or touchpad to zoom into the image.

You clearly see that the coadded image is deeper and that there is no
Moiré pattern, while you have slightly _improved_ the spatial resolution
of the output compared to the input.

   For optimal results, the oversampling should be determined by the
dithering pattern of the observation: For example if you only have two
dither points, you want the pixels with maximum value in the ‘MAX-FRAC’
image of one exposure to fall on those with a minimum value in the other
exposure.  Ideally, many more dither points should be chosen when you
are planning your observation (not just for the Moiré pattern, but also
for all the other reasons mentioned above).  Based on the dithering
pattern, you want to select the increased resolution such that the
maximum ‘MAX-FRAC’ values fall on every different pixel of the output
grid for each exposure.  Note that this discussion is on small shifts
between pointings (dithers), not large ones like offsets); see *note
Pointing pattern design::.

   ---------- Footnotes ----------

   (1) E.g.,
<https://www.stsci.edu/hst/instrumentation/wfc3/proposing/dithering-strategies>


File: gnuastro.info,  Node: Clipping outliers,  Prev: Moire pattern in coadding and its correction,  Up: Tutorials

2.10 Clipping outliers
======================

Outliers occur often in data sets.  For example cosmic rays in
astronomical imaging: the image of your target galaxy can be affected by
a cosmic ray in one of the five exposures you took in one night.  As a
result, when you compare the measured magnitude of your target galaxy in
all the exposures, you will get measurements like this (all in
magnitudes) 19.8, 20.1, 20.5, 17.0, 19.9 (all fluctuating around
magnitude 20, except the much brighter 17th magnitude measurement).

   Normally, you would simply take the mean of these measurements to
estimate the magnitude of your target with more precision.  However, the
17th magnitude measurement above is clearly wrong and will significantly
affect the mean: without it, the mean magnitude is 20.07, but with it,
the mean is 19.46:

     $ echo " 19.8 20.1 20.5 17 19.9" \
            | tr ' ' '\n' \
            | aststatistics --mean
     1.94600000000000e+01

     $ echo " 19.8 20.1 20.5 19.9" \
            | tr ' ' '\n' \
            | aststatistics --mean
     2.00750000000000e+01

   This difference of 0.61 magnitudes (or roughly 1.75 times) is
significant (for the definition of magnitudes in astronomy, see *note
Brightness flux magnitude::).  In the simple example above, you can
visually identify the "outlier" and manually remove it.  But in most
common situations you will not be so lucky!  For example when you want
to coadd the five images of the five exposures above, and each image has
$4000\times4000$ (or 16 million!)  pixels and not possible by hand in a
reasonable time (an average human's lifetime!).

   This tutorial reviews the effect of outliers and different available
ways to remove them.  In particular, we will be looking at coadding of
multiple datasets and collapsing one dataset along one of its
dimensions.  But the concepts and methods are applicable to any analysis
that is affected by outliers.

* Menu:

* Building inputs and analysis without clipping::  Building a dataset for demonstration below.
* Sigma clipping::              Standard deviation (STD) clipping.
* MAD clipping::                Median Absolute Deviation (MAD) clipping.
* Contiguous outliers::         Two clips with holes filled in the middle.


File: gnuastro.info,  Node: Building inputs and analysis without clipping,  Next: Sigma clipping,  Prev: Clipping outliers,  Up: Clipping outliers

2.10.1 Building inputs and analysis without clipping
----------------------------------------------------

As described in *note Clipping outliers::, the goal of this tutorial is
to demonstrate the effects of outliers and show how to "clip" them from
basic statistics measurements.  This is best done on an actual dataset
(rather than pure theory).  In this section we will build nine noisy
images with the script below, such that one of the images has a circle
in the middle.  We will then coadd the 9 images into one final image
based on different statistical measurements: the mean, median, standard
deviation (STD), median absolute deviation (MAD) and number of inputs
used in each pixel.  We will then analyze the resulting coadds to
demonstrate the problem with outliers.

   Put the script below into a plain-text file (assuming it is called
‘script.sh’), and run it with ‘bash ./script.sh’.  For more on writing
and good practices in shell scripts, see *note Writing scripts to
automate the steps::.  The last command of the script above calls DS9 to
visualize the five output coadded images mentioned above.

# Constants
list=""
sigma=10
number=9
radius=30
width=201
bdir=build
profsum=3e5
background=10
random_seed=1699270427


# Clipping parameters (will be set later when we start clipping).
# clip_multiple:  3   for sigma; 4.5  for MAD
# clip_tolerance: 0.1 for sigma; 0.01 for MAD
clip_operator=""
clip_multiple=""
clip_tolerance=""


# Stop if there is any error.
set -e


# If the build directory does not exist, build it.
if ! [ -d $bdir ]; then mkdir $bdir; fi


# The final image (with largest number) will contain the outlier:
# we'll put a flat circle in the center of the image as the outlier
# structure.
outlier=$bdir/in-$number.fits
nn=$bdir/$number-no-noise.fits
export GSL_RNG_SEED=$random_seed
center=$(echo $width | awk '{print int($1/2)+1}')
echo "1 $center $center 5 $radius 0 0 1 $profsum 1" \
    | astmkprof --mode=img --mergedsize=$width,$width \
                --oversample=1 --output=$nn --mcolissum
astarithmetic $nn $background + $sigma mknoise-sigma \
	      --envseed -o$outlier


# Build pure noise and add elements to the list of images to coadd.
list=$outlier
numnoise=$(echo $number | awk '{print $1-1}')
for i in $(seq 1 $numnoise); do
    img="$bdir/in-$i.fits"
    if ! [ -f $img ]; then
	export GSL_RNG_SEED=$(echo $random_seed | awk '{print $1+'$i'}')
	astarithmetic $width $width 2 makenew float32 $background + \
		      $sigma mknoise-sigma --envseed --output=$img
    fi
    list="$list $img"
done


# Coadd the images.
for op in mean median std mad; do
    if [ x"$clip_operator" = x ]; then    # No clipping.
	out=$bdir/coadd-$op.fits
	astarithmetic $list $number $op -g1 --output=$out
    else			          # With clipping.
	operator=$clip_operator-$op
	out=$bdir/coadd-$operator.fits
	astarithmetic $list $number $clip_multiple $clip_tolerance \
		      $operator -g1 --writeall --output=$out
    fi
done


# Collapse the first and last image along the 2nd dimension.
for i in 1 $number; do
    if [ x"$clip_operator" = x ]; then    # No clipping.
	out=$bdir/collapsed-$i.fits
	astarithmetic $bdir/in-$i.fits 2 collapse-median counter \
		      --writeall --output=$out
    else                                  # With clipping.
	out=$bdir/collapsed-$clip_operator-$i.fits
	astarithmetic $bdir/in-$i.fits $clip_multiple $clip_tolerance \
		      2 collapse-$clip_operator-median counter \
		      --writeall --output=$out
    fi
done

After the script finishes, you can see the generated input images with
the first command below.  The second command shows the coadded images
with the different statistics:

     $ astscript-fits-view build/in-*.fits --ds9extra="-lock scalelimits yes"
     $ astscript-fits-view build/coadd-*.fits

   Color-blind readers may not clearly see the issue in the opened
images with this color bar.  In this case, please choose the "color"
menu at the top of the DS9 and select "gray" or any other color that
makes the noisy circle (in the noise) most visible.

   The effect of an outlier on the different measurements above can be
visually seen (and quantitatively measured) through the visibility of
the circle (that was only present in one image, of nine).  Let's look at
them one by one (from the one that is most affected to the least):

‘std.fits’
     The standard deviation (third image in DS9) is the most strongly
     affected statistic by an outlier.  This is so strong that the edge
     of the circle is also clearly visible!  The standard deviation is
     calculated by first finding the mean, and estimating the difference
     of each element from the mean.  Those differences are then taken to
     the power of two and finally the square root is taken (after a
     division by the number).  It is the power-of-two component that
     amplifies the effect of the single outlier as you see here.

‘mean.fits’
     The mean (first image in DS9) is also affected by the outlier in
     such a way that the circle footprint is clearly visible.  This is
     because the nine images have the same importance in the combination
     with a simple mean.  Therefore, the outlier value pushes the result
     to higher values and the circle is printed.

‘median.fits’
     The median (second image in DS9) is also affected by the outlier;
     although much less significantly than the standard deviation or
     mean.  At first sight the circle may not be too visible!  To see it
     more clearly, click on the "Analysis" menu in DS9 and then the
     "smooth" item.  After smoothing, you will see how the single
     outlier has leaked into the median coadd.

     Intuitively, we would think that since the median is calculated
     from the middle element after sorting, the outlier goes to the end
     and won't affect the result.  However, this is not the case as we
     see here: with 9 elements, the "central" element is the 5th
     (counting from 1; after sorting).  Since the pixels covered by the
     circle only have 8 pure noise elements; the "true" median should
     have been the average of the 4th and 5th elements (after sorting).
     By definition, the 5th element is always larger than the mean of
     the 4th and 5th (because the 4th element after sorting has a
     smaller value than the 5th element).  Therefore, using the 5th
     element (after sorting), we are systematically choosing higher
     noise values in regions that are covered by the circle!

     With larger datasets, the difference between the central elements
     will be less.  However, the improved precision (in the elements
     without an outlier) will also be more.  A detailed analysis of the
     effect of a single outlier on the median based on the number of
     inputs can be done as an exercise; but in general, as this argument
     shows, the median is not immune to outliers; especially when you
     care about low signal-to-noise regimes (as we do in astronomy: low
     surface brightness science).

‘mad.fits’
     The median absolute deviation (fourth image in DS9) is affected by
     outliers in a similar fashion to the median.

   The example above included a single outlier.  But we are not usually
that lucky: there are usually more outliers!  For example, the last
‘for’ loop in the script above collapsed ‘1.fits’ (that was pure noise,
without the circle) and ‘9.fits’ (with the circle) along their second
dimension (the vertical).  The output of collapsing has one less
dimension; in this case, producing a 1D table (with the same number of
rows as the image's horizontal axis).  Collapsing was done by taking the
median along all the pixels in the vertical dimension.  To easily plot
the output afterwards, we have also used the ‘counter’ operator.  With
the command below, you can open both tables and compare them:

     $ astscript-fits-view build/collapsed-*.fits

   After TOPCAT has opened, select ‘collapsed-1.fits’ in the "Table
List" side-bar.  In the "Graphics" menu, select "Plane plot" and you
will see all the values fluctuating around 10 (with a maximum/minimum
around $\pm2$).  Afterwards, click on the "Layers" menu of the new
window (with a plot) and click on "Add position control".  At the bottom
of the window (where the scroll bar in front of "Table" is empty),
select ‘collapsed-9.fits’.  In the regions that there was no circle in
any of the vertical axes, the two match nicely (the noise level is the
same).  However, you see that the regions that were partly covered by
the outlying circle gradually get more affected as the width of the
circle in that column increases (the full diameter of the circle was in
the middle of the image).  This shows how the median is biased by
outliers as their number increases.

   To see the problem more prominently, use the ‘collapse-mean’ operator
instead of the median.  The reason that the mean is more strongly
affected by the outlier is exactly the same as above for the coadding of
the input images.  In the subsections below, we will describe some of
the available ways (in Gnuastro) to reject the effect of these outliers
(and have better coadds or collapses).  But the methodology is not
limited to these types of data and can be generically applied; unless
specified explicitly.


File: gnuastro.info,  Node: Sigma clipping,  Next: MAD clipping,  Prev: Building inputs and analysis without clipping,  Up: Clipping outliers

2.10.2 Sigma clipping
---------------------

Let's assume that you have pure noise (centered on zero) with a clear
Gaussian distribution
(https://en.wikipedia.org/wiki/Normal_distribution), or see *note Photon
counting noise::.  Now let's assume you add very bright objects (signal)
on the image which have a very sharp boundary.  By a sharp boundary, we
mean that there is a clear cutoff (from the noise) at the pixels the
objects finish.  In other words, at their boundaries, the objects do not
fade away into the noise.

   In optical astronomical imaging, cosmic rays (when they collide at a
near normal incidence angle) are a very example of such outliers.  The
tracks they leave behind in the image are perfectly immune to the
blurring caused by the atmosphere on images of stars or galaxies and
they have a very high signal-to-noise ratio.  They are also very
energetic and so their borders are usually clearly separated from the
surrounding noise.  See Figure 15 in Akhlaghi and Ichikawa, 2015
(https://arxiv.org/abs/1505.01664).

   In such a case, when you plot the histogram (see *note Histogram and
Cumulative Frequency Plot::) of the distribution, the pixels relating to
those objects will be clearly separate from pixels that belong to parts
of the image that did not have any signal (were just noise).  In the
cumulative frequency plot, after a steady rise (due to the noise), you
would observe a long flat region were for a certain range of the dynamic
range (horizontal axis), there is no increase in the index (vertical
axis).

   In the previous section (*note Building inputs and analysis without
clipping::) we created one such dataset (‘in-9.fits’).  With the command
below, let's have a look at its histogram and cumulative frequency plot
(in simple ASCII format; we are decreasing the default number of bins
with ‘--numasciibins’ to show them easily within the width of the print
version of this manual; feel free to change this).

     $ aststatistics build/in-9.fits --asciihist --asciicfp \
                     --numasciibins=65

     ASCII Histogram:
     Number: 40401
     Y: (linear: 0 to 4191)
     X: (linear: -31.9714 -- 150.323, in 65 bins)
      |              **
      |             ****
      |            ******
      |            ******
      |           ********
      |           ********
      |          **********
      |         ************
      |        **************
      |      ******************                          ******
      |*******************************         *************************
      |-----------------------------------------------------------------


     ASCII Cumulative frequency plot:
     Y: (linear: 0 to 40401)
     X: (linear: -31.9714 -- 150.323, in 65 bins)
      |                                                  ***************
      |                   **********************************************
      |                  ***********************************************
      |                *************************************************
      |               **************************************************
      |              ***************************************************
      |             ****************************************************
      |            *****************************************************
      |           ******************************************************
      |         ********************************************************
      |*****************************************************************
      |-----------------------------------------------------------------

   We see a clear bimodal
(https://en.wikipedia.org/w/index.php?title=Bimodal) distribution in the
histogram.  Such outliers can significantly bias any single measurement
over the whole dataset.  For example let's compare the median, mean and
standard deviation of the image above with ‘1.fits’:

     $ aststatistics build/in-1.fits --median --mean --std
     9.90529778313248e+00 9.96143102101206e+00 1.00137568561776e+01

     $ aststatistics build/in-9.fits --median --mean --std
     1.09305819367634e+01 1.74470443173776e+01 2.88895986970341e+01

   The effect of the outliers is obvious in all three measures: the
median has become 1.10 times larger, the mean 1.75 times and the
standard deviation about 2.88 times!  The differing effect of outliers
in different statistics was already discussed in *note Building inputs
and analysis without clipping::; also see *note Quantifying signal in a
tile::.

   $\sigma$-clipping is one commonly used way to remove/clip the effect
of such very strong outliers in measures like the above (although not
the most robust, continue reading to the end of this tutorial in the
next sections).  $\sigma$-clipping is defined as the very simple
iteration below.  In each iteration, the number of input values used
might decrease.  When the outliers are as strong as above, the outliers
will be removed through this iteration.

  1. Calculate the standard deviation ($\sigma$) and median ($m$) of a
     distribution.  The median is used because, as shown above, the mean
     is too significantly affected by the presence of outliers.
  2. Remove all points that are smaller or larger than
     $m\pm\alpha\sigma$.
  3. Go back to step 1, unless the selected exit criteria is reached.
     There are commonly two types of exit criteria (to stop the
     $\sigma$-clipping iteration).  Within Gnuastro's programs that use
     sigma-clipping, the exit criteria is the second value to the
     ‘--sclipparams’ option (the first value is the $\alpha$ above):

        • When a certain number of iterations has taken place (exit
          criteria is an integer, larger than 1).  For example
          ‘--sclipparams=5,3’ means that the $5\sigma$ clipping will
          stop after 3 clips.
        • When the new measured standard deviation is within a certain
          tolerance level of the previous iteration (exit criteria is
          floating point and less than 1.0).  The tolerance level is
          defined by:

           $$\sigma_{old}-\sigma_{new} \over \sigma_{new}$$

          In each clipping, the dispersion in the distribution is either
          less or equal.  So $\sigma_{old}\geq\sigma_{new}$.  For
          example ‘--sclipparams=5,0.2’ means that the $5\sigma$
          clipping will stop the old and new standard deviations are
          equal within a factor of $0.2$.

   Let's see the algorithm in practice with the ‘--sigmaclip’ option of
Gnuastro's Statistics program (using the default configuration of
$3\sigma$ clipping and tolerance of 0.1):

     $ aststatistics build/in-9.fits --sigmaclip
     Statistics (GNU Astronomy Utilities) 0.24
     -------
     Input: build/in-9.fits (hdu: 1)
     -------
     3-sigma clipping steps until relative change in STD is less than 0.1:

     round number     median       STD
     1     40401      1.09306e+01  2.88896e+01
     2     37660      1.00306e+01  1.07153e+01
     3     37539      1.00080e+01  9.93741e+00
     -------
     Statistics (after clipping):
       Number of input elements:                40401
       Number of clips:                         2
       Final number of elements:                37539
       Median:                                  1.000803e+01
       Mean:                                    1.001822e+01
       Standard deviation:                      9.937410e+00
       Median Absolute Deviation:               6.772760e+00

   After the basic information about the input and settings, the
Statistics program has printed the information for each round
(iteration) of clipping.  Initially, there was 40401 elements (the image
is $201\times201$ pixels).  After the first round of clipping, only
37660 elements remained and because the difference in standard deviation
was larger than the tolerance level, a third clipping was one.  But the
change in standard deviation after the third clip (in relation to the
second) was smaller than the tolerance level, so the exit criteria was
activated and the clipping finished with 37539 elements.  In the end, we
see that the final median, mean and standard deviation are very similar
to the data without any outlier (‘build/1.fits’ in the example above).
Therefore, through clipping we were able to remove the second "outlier"
distribution from the bimodal histogram above (because it was so nicely
separated from the main/noise).

   The example above provided a single statistic from a single dataset.
Other scenarios where sigma-clipping becomes necessary are coadding and
collapsing (that was the main goal of the script in *note Building
inputs and analysis without clipping::).  To generate $\sigma$-clipped
coadds and collapsed tables, you just need to change the values of the
three variables of the script (shown below).  After making this change
in your favorite text editor, have a look at the outputs.  By the way,
if you have still not read (and understood) the commands in that script,
this is a good time to do it so the steps below do not appear as a black
box to you (for more on writing shell scripts, see *note Writing scripts
to automate the steps::).

     $ grep ^clip_ script.sh
     clip_operator=sigclip    # These variables will be used more
     clip_multiple=3          # effectively with the clipping
     clip_tolerance=0.1       # operators of the next sections.

     $ bash ./script.sh

     $ astscript-fits-view build/coadd-std.fits \
                           build/coadd-sigclip-std.fits \
                           build/coadd-*mean.fits \
                           build/coadd-*median.fits \
                           --ds9extra="-tile grid layout 2 3 -scale minmax"

   You will see 6 images arranged in two columns: the first column is
the normal coadd (without $\sigma$-clipping) and the second column is
the $\sigma$-clipped coadd of the same statistic (first row: standard
deviation, second row: mean, third row: median).

   It is clear that the $\sigma$-clipped (right column in DS9) results
have improved in all three measures (compared to the left column).  This
was achieved by clipping/removing outliers.  To see how many input
images were used in each pixel of the clipped coadd, you should look
into the second HDU of any clipping output which shows the number of
inputs that were used for each pixel:

     $ astscript-fits-view build/coadd-sigclip-median.fits \
                           --ds9extra="-scale minmax"

   In the "Cube" window of opened DS9, click on the "Next" button.  The
pixels in this image only have two values: 8 or 9.  Over the footprint
of the circle, most pixels have a value of 8: only 8 inputs were used
for these (one of the inputs was clipped out).  In the other regions of
the image, you see that the pixels almost consistently have a value of 9
(except for some noisy pixels here and there).

   It is the "holes" (with value 9) within the footprint of the circle
that keep the circle visible in the final coadd of the output (as we saw
previously in the 2-column DS9 command before).  Spoiler alert: in a
later section of this tutorial (*note Contiguous outliers::) you will
see how we fix this problem.  But please be patient and continue reading
and running the commands for now.

   So far, $\sigma$-clipping seems to have preformed nicely.  However,
there are important caveats to $\sigma$-clipping that are listed in the
box below and further elaborated (with examples) afterwards.

*Caveats of $\sigma$-clipping*: continue this section to visually see
the effect of both these caveats:
   • The standard deviation is itself heavily influenced by the presence
     of outliers (as we have shown above).  Therefore a sufficiently
     small number of outliers can cause an over-estimation of the
     standard deviation.  This can be strong enough to keep those from
     getting clipped!

   • When the outliers do not constitute a clearly distinct distribution
     like the example here, $\sigma$-clipping will not be able to
     separate them (see the bimodal histogram above for situations that
     $\sigma$-clipping is useful).

   To demonstrate the caveats above, let's decrease the brightness
(total sum of values) in the circle by decreasing the value of the
‘profsum’ variable in the script:

     $ grep ^profsum script.sh
     profsum=1e5

     $ bash ./script.sh

   First, let's have a look at the new circle in noise with the first
command below.  With the second command, let's view its pixel value
histogram (recall that previously, the circle had a clearly separate
distribution):

     $ astscript-fits-view build/in-9.fits

     $ aststatistics build/in-9.fits --asciihist --numasciibins=65
     ASCII Histogram:
     Number: 40401
     Y: (linear: 0 to 2654)
     X: (linear: -31.9714 -- 79.4266, in 65 bins)
      |                        **
      |                      *****
      |                    *********
      |                    **********
      |                  *************
      |                  **************
      |                *****************
      |               *******************
      |             ***********************
      |          ****************************************
      |*****************************************************************
      |-----------------------------------------------------------------

   We see that even tough the circle is still clearly visible in the
noise in DS9, we don't have a bimodal histogram any more; the circle's
pixels have blended into the noise, and just caused a skewness in the
(otherwise symmetric) noise distribution.  Let's try running the
‘--sigmaclip’ option as above:

     $ aststatistics build/in-9.fits --sigmaclip
     Statistics (GNU Astronomy Utilities) 0.24
     -------
     Input: build/in-9.fits (hdu: 1)
     -------
     3-sigma clipping steps until relative change in STD is less than 0.1:

     round number     median       STD
     1     40401      1.09295e+01  1.34784e+01
     2     39618      1.06762e+01  1.19852e+01
     3     39126      1.05265e+01  1.12983e+01
     -------
     Statistics (after clipping):
       Number of input elements:                40401
       Number of clips:                         2
       Final number of elements:                39126
       Median:                                  1.052652e+01
       Mean:                                    1.114819e+01
       Standard deviation:                      1.129831e+01
       Median Absolute Deviation:               7.106166e+00

   We see that the median, mean and standard deviation are over
estimated (each worse than the time when the circle was brighter!).
Let's look at the $\sigma$-clipping coadd:

     $ astscript-fits-view build/coadd-std.fits \
                           build/coadd-sigclip-std.fits \
                           build/coadd-*mean.fits \
                           build/coadd-*median.fits \
                           --ds9extra="-tile grid layout 2 3 -scale minmax"

   Compared to the previous run (where the outlying circle was
brighter), we see that $\sigma$-clipping is now less successful in
removing the outlying circle from the coadds; or in the single value
measurements.  To see the reason, we can have a look at the numbers
image (note that here, we are using ‘-h2’ to only see the numbers image)

     $ astscript-fits-view build/coadd-sigclip-median.fits -h2 \
                           --ds9extra="-scale minmax"

   Unlike before (where the density of pixels with 8 images was very
high over the circle's footprint), the circle is barely visible in the
numbers image!  There is only a very weak clustering of pixels with a
value of 8 over the circle's footprint.  This has happened because the
outliers have biased the standard deviation itself to a level that
includes them with this multiple of the standard deviation.

   To gauge if $\sigma$-clipping will be useful for your dataset, you
should inspect the bimodality of its histogram like we did above.  But
you can't do this manually in every case (as in the coadding which
involved more than forty thousand separate $\sigma$-clippings: one for
every output)!  Clipping outliers should be based on a different (from
standard deviation) measure of scatter/dispersion, one that is more
robust (less affected by outliers).  Therefore, in Gnuastro we also have
median absolute deviation (MAD) clipping which is described in the next
section (*note MAD clipping::).


File: gnuastro.info,  Node: MAD clipping,  Next: Contiguous outliers,  Prev: Sigma clipping,  Up: Clipping outliers

2.10.3 MAD clipping
-------------------

When clipping outliers, it is important that the used measure of
dispersion is itself not strongly affected by the outliers.  Previously
(in *note Sigma clipping::), we saw that the standard deviation is not a
good measure of dispersion because of its strong dependency on outliers.
In this section, we'll introduce clipping operators that are based on
the median absolute deviation
(https://en.wikipedia.org/wiki/Median_absolute_deviation) (MAD).

   The median absolute deviation is defined as the median of the
differences from the median (MAD requires taking two rounds of medians).
As mathematically derived in the Wikipedia page above, for a pure
Gaussian distribution, the median absolute deviation will be roughly
$0.67449\sigma$.  We can confirm this numerically from the images with
pure noise that we created previously in *note Building inputs and
analysis without clipping::.  With the first command below we can see
the raw standard deviation and median absolute deviation values and the
second command shows their division:

     $ aststatistics build/in-1.fits --std --mad
     1.00137568561776e+01 6.74662296703343e+00

     $ aststatistics build/in-1.fits --std --mad | awk '{print $2/$1}'
     0.673735

   The algorithm of MAD-clipping is identical to $\sigma$-clipping,
except that instead of $\sigma$, it uses the median absolute deviation
(also see the next paragraph).  Since the median absolute deviation is
smaller than the standard deviation by roughly 0.67, if you regularly
use $3\sigma$ there, you should use $(3/0.67)\rm{MAD}=(4.48)\rm{MAD}$
when doing MAD-clipping.  The usual tolerance should also be changed due
to the differing (discrete) nature of the median absolute deviation
(based on sorted differences) in relation to the standard deviation
(based on the sum of squared differences; which is more smooth).  A
tolerance of 0.01 is better suited to the termination criteria of
MAD-clipping.

   Another difference of the MAD-clipping algorithm in Gnuastro with
$\sigma$-clipping is a special condition when the MAD becomes zero (will
be more common in integer datasets).  This can happen when the median
value is repeated in the input dataset.  In such cases, we will find the
two elements before and after the median that have a different value
than the median; it will then use their difference as the MAD(1).

   To demonstrate the steps in practice, let's assume you have the
original script in *note Building inputs and analysis without clipping::
with the changes shown in the first command below and With the second
command we'll execute the script.

     $ grep '^clip_\|^profsum' script.sh
     profsum=1e5
     clip_operator=madclip
     clip_multiple=4.5
     clip_tolerance=0.01

     $ bash ./script.sh

Let's start by applying MAD-clipping on the image with the bright
circle:

     $ aststatistics build/in-9.fits --madclip
     Statistics (GNU Astronomy Utilities) 0.24
     -------
     Input: build/in-9.fits (hdu: 1)
     -------
     4.5-MAD clipping steps until relative change in MAD
     (median absolute deviation) is less than 0.01:

     round number     median       MAD
     1     40401      1.09295e+01  7.38609e+00
     2     38812      1.04313e+01  7.04036e+00
     3     38567      1.03497e+01  6.98680e+00
     -------
     Statistics (after clipping):
       Number of input elements:                40401
       Number of clips:                         2
       Final number of elements:                38567
       Median:                                  1.034968e+01
       Mean:                                    1.070246e+01
       Standard deviation:                      1.063998e+01
       Median Absolute Deviation:               6.986797e+00

   We see that the median, mean and standard deviation after
MAD-clipping are much better than the $\sigma$-clipping (see *note Sigma
clipping::): the median is now 10.3 (was 10.5 in $\sigma$-clipping),
mean is 10.7 (was 10.11) and the standard deviation is 10.6 (was 10.12).

   Let's compare the MAD-clipped coadds with the results of the previous
section.  Since we want the images shown in a certain order, we'll first
construct the list of images (with a ‘for’ loop that will fill the
‘imgs’ variable).  Note that this assumes you have ran and carefully
read/understand all the commands in the previous sections (*note
Building inputs and analysis without clipping:: and *note Sigma
clipping::).

     $ imgs=""
     $ p=build/coadd   # 'p' is short for "prefix"
     $ for m in std mean median mad; do \
        imgs="$imgs $p-$m.fits $p-sigclip-$m.fits $p-madclip-$m.fits"; \
       done
     $ astscript-fits-view $imgs --ds9extra="-tile grid layout 3 4"

   The first column shows the non-clipped coadds for each statistic
(generated in *note Building inputs and analysis without clipping::),
the second column are $\sigma$-clipped coadds (generated in *note Sigma
clipping::), and the third column shows the newly created MAD-clipped
coadds.  We see that the circle is much more weaker in the MAD-clipped
coadds than in the $\sigma$-clipped coadds in all rows (different
statistics).  Let's confirm this with the numbers images of the two
clipping methods:

     $ astscript-fits-view -g2 \
                 build/coadd-sigclip-median.fits \
                 build/coadd-madclip-median.fits -g2 \
                 --ds9extra="-scale limits 1 9 -lock scalelimits yes"

   In the numbers image of the MAD-clipped coadd, you see the circle
much more clearly.  However, you also see that in the regions outside
the circle, many random pixels have also been coadded with less than 9
input images!  This is a caveat of MAD clipping and is expected: by
nature MAD clipping is much more "noisy".  With the command below, let's
have a look at the statistics of the numbers image of the MAD-clipping.
With the second, let's see how many pixels used fewer than 5 input
images:

     $ aststatistics build/coadd-madclip-median.fits -h2 \
                     --numasciibins=60
     Statistics (GNU Astronomy Utilities) 0.24
     -------
     Input: build/coadd-madclip-median.fits (hdu: 2)
     -------
       Number of elements:                      40401
       Minimum:                                 2
       Maximum:                                 9
       Median:                                  9
       Mean:                                    8.500284646
       Standard deviation:                      1.1275244
     -------
      |                                                           *
      |                                                           *
      |                                                           *
      |                                                           *
      |                                                           *
      |                                                           *
      |                                                           *
      |                                                           *
      |                                                   *       *
      |                                          *        *       *
      |*       *        *       *        *       *        *       *
      |------------------------------------------------------------

     $ aststatistics build/coadd-madclip-median.fits -h2 \
                     --lessthan=5 --number
     686

   Almost 700 pixels were made with less than 5 inputs!  The large
number of random pixels that have been clipped is not good and can make
it hard to understand the noise statistics of the coadd.

   Ultimately, even MAD-clipping is not perfect and even though the
circle is weaker, we still see the circle in all four cases, even with
the MAD-clipped median (more clearly: after smoothing/blocking).  The
reason is similar to what was described in $\sigma$-clipping (using the
original ‘profsum=3e5’: the "holes" in the numbers image.  Because the
circle's pixel values are not too distant from the noise and the noisy
nature of the MAD, some of its elements do not get clipped, and their
coadded value gets systematically higher than the rest of the image.

   Fortunately all is not lost!  In Gnuastro, we have a fix for such
contiguous outliers that is described fully in the next section (*note
Contiguous outliers::).

   ---------- Footnotes ----------

   (1) For more specialized conditions in Gnuastro's implementation of
MAD-clipping, see the heavily commented code of ‘MAD_ALTERNATIVE’ in
‘lib/statistics.c’ of Gnuastro's source.


File: gnuastro.info,  Node: Contiguous outliers,  Prev: MAD clipping,  Up: Clipping outliers

2.10.4 Contiguous outliers
--------------------------

When source of the outlier covers more than one element, and its flux is
close to the noise level, not all of its elements will be clipped:
because of noise, some of its elements will remain un-clipped; and thus
affecting the output.

   Examples of this were created and thoroughly discussed in previous
sections with $\sigma$-clipping and MAD-clipping (see *note Sigma
clipping:: and *note MAD clipping::).  $\sigma$-clipping had good purity
(very few non-outliers were clipped) but at the cost of bad completeness
(many outliers remained).  MAD-clipping was the opposite: it masked many
outliers (good completeness), but at the cost of clipping many pixels
that shouldn't have been clipped (bad purity).

   Fortunately there is a good way to benefit from the best of both
worlds.  Recall that in the numbers image of the MAD-clipping output,
the wrongly clipped pixels were randomly distributed are barely
connected.  On the other hand, those that covered the circle were nicely
connected, with unclipped pixels scattered within it.  Therefore, using
their spatial distribution, we can improve the completeness (not have
any "holes" within the masked circle) and purity (remove the false
clips).  This is done through the ‘madclip-maskfilled’ operator:

  1. MAD-clipping is applied ($\sigma$-clipping is also possible, but
     less effective).
  2. A binary image is created for each input: any outlying pixel of
     each input is set to 1 (foreground); the rest are set to 0
     (background).  Mathematical morphology operators are then used in
     preparation to filling the holes (to close the boundary of the
     contiguous outlier):
        • For 2D images (were each pixel has 8 neighbors) the foreground
          pixels are dilated with a "connectivity" of 1 (only the
          nearest neighbors: 4-connectivity in a 2D image).
        • For 1D arrays (where each element only has two neighbors), the
          foreground is eroded.  This is necessary because the next step
          (where the holes are filled), two pixels that have been
          clipped purely due to noise with a large distance between them
          can wrongly mask a very large range of the input data.
  3. Any background pixel that is fully surrounded by the foreground (or
     a "hole") is filled (given a value of 1: becoming a foreground
     pixel).
  4. One step of 8-connected opening (an erosion, followed by a
     dilation) is applied to remove (set to background) any
     non-contiguous foreground pixel of each input.
  5. All the foreground pixels of the binary images are set to NaN in
     the respective input data (that the binary image corresponds to).

   Let's have a look at the output of this process with the first
command below.  Note that because ‘madclip-maskfilled’ returns its main
input operands back to the coadd, we need to call Arithmetic with
‘--writeall’ (which will produce a multi-HDU output file).  With the
second, open the output in DS9:

     $ astarithmetic build/in-*.fits 9 4.5 0.01 madclip-maskfilled \
                     -g1 --writeall --output=inputs-masked.fits

     $ astscript-fits-view inputs-masked.fits

   In the small "Cube" window, you will see that 9 HDUs are present in
‘inputs-masked.fits’.  Click on the "Next" button to see each input.
When you get to the last (9th) HDU, you will notice that the circle has
been masked there (it is filled with NaN values).  Now that all the
contiguous outlier(s) of the inputs are masked, we can use more pure
coadding operators (like $\sigma$-clipping) to remove any strong,
single-pixel outlier:

     $ astarithmetic build/in-*.fits 9 4.5 0.01 madclip-maskfilled \
                     9 5 0.1 sigclip-mean \
                     -g1 --writeall --output=coadd-good.fits

     $ astscript-fits-view coadd-good.fits --ds9scale=minmax

   You see a clean noisy coadd in the first HDU (note that we used the
$\sigma$-clipped mean here; which was strongly affected by outliers as
we saw before in *note Sigma clipping::).  When you go to the next HDU,
you see that over the circle only 8 images were used and that there are
no "holes" there.  But the operator that was most affected by outliers
was the standard deviation.  To test it, repeat the first command above
and use ‘sigclip-std’ instead of ‘sigclip-mean’ and have a look at the
output: again, you don't see any footprint of the circle.

   Of course, if the potential outlier signal can become weaker, there
are some solutions: use more inputs if you can (to decrease the noise),
or decrease the multiple MAD in the ‘madclip-maskfilled’ call above: it
will decrease your purity, but to some level, this may be fine (depends
on your usage of the coadd).


File: gnuastro.info,  Node: Installation,  Next: Common program behavior,  Prev: Tutorials,  Up: Top

3 Installation
**************

The latest released version of Gnuastro source code is always available
at the following URL:

   <http://ftpmirror.gnu.org/gnuastro/gnuastro-latest.tar.gz>

*note Quick start:: describes the commands necessary to configure,
build, and install Gnuastro on your system.  This chapter will be useful
in cases where the simple procedure above is not sufficient, for
example, your system lacks a mandatory/optional dependency (in other
words, you cannot pass the ‘$ ./configure’ step), or you want greater
customization, or you want to build and install Gnuastro from other
random points in its history, or you want a higher level of control on
the installation.  Thus if you were happy with downloading the tarball
and following *note Quick start::, then you can safely ignore this
chapter and come back to it in the future if you need more
customization.

   *note Dependencies:: describes the mandatory, optional and
bootstrapping dependencies of Gnuastro.  Only the first group are
required/mandatory when you are building Gnuastro using a tarball (see
*note Release tarball::), they are very basic and low-level tools used
in most astronomical software, so you might already have them installed,
if not they are very easy to install as described for each.  *note
Downloading the source:: discusses the two methods you can obtain the
source code: as a tarball (a significant snapshot in Gnuastro's
history), or the full history(1).  The latter allows you to build
Gnuastro at any random point in its history (for example, to get bug
fixes or new features that are not released as a tarball yet).

   The building and installation of Gnuastro is heavily customizable, to
learn more about them, see *note Build and install::.  This section is
essentially a thorough explanation of the steps in *note Quick start::.
It discusses ways you can influence the building and installation.  If
you encounter any problems in the installation process, it is probably
already explained in *note Known issues::.  In *note Other useful
software:: the installation and usage of some other free software that
are not directly required by Gnuastro but might be useful in conjunction
with it is discussed.

* Menu:

* Dependencies::                Necessary packages for Gnuastro.
* Downloading the source::      Ways to download the source code.
* Build and install::           Configure, build and install Gnuastro.

   ---------- Footnotes ----------

   (1) *note Bootstrapping dependencies:: are required if you clone the
full history.


File: gnuastro.info,  Node: Dependencies,  Next: Downloading the source,  Prev: Installation,  Up: Installation

3.1 Dependencies
================

A minimal set of dependencies are mandatory for building Gnuastro from
the standard tarball release.  If they are not present you cannot pass
Gnuastro's configuration step.  The mandatory dependencies are therefore
very basic (low-level) tools which are easy to obtain, build and
install, see *note Mandatory dependencies:: for a full discussion.

   If you have the packages of *note Optional dependencies::, Gnuastro
will have additional functionality (for example, converting FITS images
to JPEG or PDF). If you are installing from a tarball as explained in
*note Quick start::, you can stop reading after this section.  If you
are cloning the version controlled source (see *note Version controlled
source::), an additional bootstrapping step is required before
configuration and its dependencies are explained in *note Bootstrapping
dependencies::.

   Your operating system's package manager is an easy and convenient way
to download and install the dependencies that are already pre-built for
your operating system.  In *note Dependencies from package managers::,
we will list some common operating system package manager commands to
install the optional and mandatory dependencies.

* Menu:

* Mandatory dependencies::      Gnuastro will not install without these.
* Optional dependencies::       Adding more functionality.
* Bootstrapping dependencies::  If you have the version controlled source.
* Dependencies from package managers::  Installing from OS package managers.


File: gnuastro.info,  Node: Mandatory dependencies,  Next: Optional dependencies,  Prev: Dependencies,  Up: Dependencies

3.1.1 Mandatory dependencies
----------------------------

The mandatory Gnuastro dependencies are very basic and low-level tools.
They all follow the same basic GNU based build system (like that shown
in *note Quick start::), so even if you do not have them, installing
them should be pretty straightforward.  In this section we explain each
program and any specific note that might be necessary in the
installation.

* Menu:

* GNU Scientific Library::      Installing GSL.
* CFITSIO::                     C interface to the FITS standard.
* WCSLIB::                      C interface to the WCS standard of FITS.


File: gnuastro.info,  Node: GNU Scientific Library,  Next: CFITSIO,  Prev: Mandatory dependencies,  Up: Mandatory dependencies

3.1.1.1 GNU Scientific Library
..............................

The GNU Scientific Library (http://www.gnu.org/software/gsl/), or GSL,
is a large collection of functions that are very useful in scientific
applications, for example, integration, random number generation, and
Fast Fourier Transform among many others.  To download and install GSL
from source, you can run the following commands.

     $ wget https://ftp.gnu.org/gnu/gsl/gsl-latest.tar.gz
     $ tar -xf gsl-latest.tar.gz
     $ cd gsl-X.X                     # Replace X.X with version number.
     $ ./configure CFLAGS="$CFLAGS -g0 -O3"
     $ make -j8                       # Replace 8 with no. CPU threads.
     $ make check
     $ sudo make install


File: gnuastro.info,  Node: CFITSIO,  Next: WCSLIB,  Prev: GNU Scientific Library,  Up: Mandatory dependencies

3.1.1.2 CFITSIO
...............

CFITSIO (http://heasarc.gsfc.nasa.gov/fitsio/) is the closest you can
get to the pixels in a FITS image while remaining faithful to the FITS
standard (http://fits.gsfc.nasa.gov/fits_standard.html).  It is written
by William Pence, the principal author of the FITS standard(1), and is
regularly updated.  Setting the definitions for all other software
packages using FITS images.

   Some GNU/Linux distributions have CFITSIO in their package managers,
if it is available and updated, you can use it.  One problem that might
occur is that CFITSIO might not be configured with the
‘--enable-reentrant’ option by the distribution.  This option allows
CFITSIO to open a file in multiple threads, it can thus provide great
speed improvements.  If CFITSIO was not configured with this option, any
program which needs this capability will warn you and abort when you ask
for multiple threads (see *note Multi-threaded operations::).

   To install CFITSIO from source, we strongly recommend that you have a
look through Chapter 2 (Creating the CFITSIO library) of the CFITSIO
manual and understand the options you can pass to ‘$ ./configure’ (they
are not too much).  This is a very basic package for most astronomical
software and it is best that you configure it nicely with your system.
Once you download the source and unpack it, the following configure
script should be enough for most purposes.  Do not forget to read
chapter two of the manual though, for example, the second option is only
for 64bit systems.  The manual also explains how to check if it has been
installed correctly.

   CFITSIO comes with two executable files called ‘fpack’ and ‘funpack’.
From their manual: they "are standalone programs for compressing and
uncompressing images and tables that are stored in the FITS (Flexible
Image Transport System) data format.  They are analogous to the gzip and
gunzip compression programs except that they are optimized for the types
of astronomical images that are often stored in FITS format".  The
commands below will compile and install them on your system along with
CFITSIO. They are not essential for Gnuastro, since they are just
wrappers for functions within CFITSIO, but they can come in handy.  The
‘make utils’ command is only available for versions above 3.39, it will
build these executable files along with several other executable test
files which are deleted in the following commands before the
installation (otherwise the test files will also be installed).

   The commands necessary to download the source, decompress, build and
install CFITSIO from source are described below.

     $ urlbase=http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c
     $ wget $urlbase/cfitsio_latest.tar.gz
     $ tar -xf cfitsio_latest.tar.gz
     $ cd cfitsio-X.XX                   # Replace X.XX with version
     $ ./configure --prefix=/usr/local --enable-sse2 --enable-reentrant \
                   CFLAGS="$CFLAGS -g0 -O3"
     $ make
     $ make utils
     $ ./testprog > testprog.lis         # See below if this has an error
     $ diff testprog.lis testprog.out    # Should have no output
     $ cmp testprog.fit testprog.std     # Should have no output
     $ rm cookbook fitscopy imcopy smem speed testprog
     $ sudo make install

   In the ‘./testprog > testprog.lis’ step, you may confront an error,
complaining that it cannot find ‘libcfitsio.so.AAA’ (where ‘AAA’ is an
integer).  This is the library that you just built and have not yet
installed.  But unfortunately some versions of CFITSIO do not account
for this on some OSs.  To fix the problem, you need to tell your OS to
also look into current CFITSIO build directory with the first command
below, afterwards, the problematic command (second below) should run
properly.

     $ export LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH"
     $ ./testprog > testprog.lis

   Recall that the modification above is ONLY NECESSARY FOR THIS STEP.
_Do not_ put the ‘LD_LIBRARY_PATH’ modification command in a permanent
place (like your bash startup file).  After installing CFITSIO, close
your terminal and continue working on a new terminal (so
‘LD_LIBRARY_PATH’ has its default value).  For more on
‘LD_LIBRARY_PATH’, see *note Installation directory::.

   ---------- Footnotes ----------

   (1) Pence, W.D. et al.  Definition of the Flexible Image Transport
System (FITS), version 3.0.  (2010) Astronomy and Astrophysics, Volume
524, id.A42, 40 pp.


File: gnuastro.info,  Node: WCSLIB,  Prev: CFITSIO,  Up: Mandatory dependencies

3.1.1.3 WCSLIB
..............

WCSLIB (http://www.atnf.csiro.au/people/mcalabre/WCS/) is written and
maintained by one of the authors of the World Coordinate System (WCS)
definition in the FITS standard
(http://fits.gsfc.nasa.gov/fits_standard.html)(1), Mark Calabretta.  It
might be already built and ready in your distribution's package
management system.  However, here the installation from source is
explained, for the advantages of installation from source please see
*note Mandatory dependencies::.  To install WCSLIB you will need to have
CFITSIO already installed, see *note CFITSIO::.

   WCSLIB also has plotting capabilities which use PGPLOT (a plotting
library for C). If you wan to use those capabilities in WCSLIB, *note
PGPLOT:: provides the PGPLOT installation instructions.  However PGPLOT
is old(2), so its installation is not easy, there are also many great
modern WCS plotting tools (mostly in written in Python).  Hence, if you
will not be using those plotting functions in WCSLIB, you can configure
it with the ‘--without-pgplot’ option as shown below.

   If you have the cURL library (3) on your system and you installed
CFITSIO version 3.42 or later, you will need to also link with the cURL
library at configure time (through the ‘-lcurl’ option as shown below).
CFITSIO uses the cURL library for its HTTPS (or HTTP Secure(4)) support
and if it is present on your system, CFITSIO will depend on it.
Therefore, if ‘./configure’ command below fails (you do not have the
cURL library), then remove this option and rerun it.

   To download, configure, build, check and install WCSLIB from source,
you can follow the steps below.
     ## Download and unpack the source tarball
     $ wget ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2
     $ tar -xf wcslib.tar.bz2

     ## In the `cd' command, replace `X.X' with version number.
     $ cd wcslib-X.X

     ## If `./configure' fails, remove `-lcurl' and run again.
     $ ./configure LIBS="-pthread -lcurl -lm" --without-pgplot     \
                   --disable-fortran CFLAGS="$CFLAGS -g0 -O3"
     $ make
     $ make check
     $ sudo make install

   ---------- Footnotes ----------

   (1) Greisen E.W., Calabretta M.R. (2002) Representation of world
coordinates in FITS. Astronomy and Astrophysics, 395, 1061-1075.

   (2) As of early June 2016, its most recent version was uploaded in
February 2001.

   (3) <https://curl.haxx.se>

   (4) <https://en.wikipedia.org/wiki/HTTPS>


File: gnuastro.info,  Node: Optional dependencies,  Next: Bootstrapping dependencies,  Prev: Mandatory dependencies,  Up: Dependencies

3.1.2 Optional dependencies
---------------------------

The libraries listed here are only used for very specific applications,
therefore they are optional and Gnuastro can be built without them (with
only those specific features disabled).  Since these are pretty
low-level tools, they are not too hard to install from source, but you
can also use your operating system's package manager to easily install
all of them.  For more, see *note Dependencies from package managers::.

   If the ‘./configure’ script cannot find any of these optional
dependencies, it will notify you of the operation(s) you cannot do due
to not having them.  If you continue the build and request an operation
that uses a missing library, Gnuastro's programs will warn that the
optional library was missing at build-time and abort.  Since Gnuastro
was built without that library, installing the library afterwards will
not help.  The only way is to rebuild Gnuastro from scratch (after the
library has been installed).  However, for program dependencies (like
cURL or Ghostscript) things are easier: you can install them after
building Gnuastro also.  This is because libraries are used to build the
internal structure of Gnuastro's executables.  However, a program
dependency is called by Gnuastro's programs at run-time and has no
effect on their internal structure.  So if a dependency program becomes
available later, it will be used next time it is requested.

GNU Libtool
     Libtool is a program to simplify managing of the libraries to build
     an executable (a program).  GNU Libtool has some added
     functionality compared to other implementations.  If GNU Libtool is
     not present on your system at configuration time, a warning will be
     printed and *note BuildProgram:: will not be built or installed.
     The configure script will look into your search path (‘PATH’) for
     GNU Libtool through the following executable names: ‘libtool’
     (acceptable only if it is the GNU implementation) or ‘glibtool’.
     See *note Installation directory:: for more on ‘PATH’.

     GNU Libtool (the binary/executable file) is a low-level program
     that is probably already present on your system, and if not, is
     available in your operating system package manager(1).  If you want
     to install GNU Libtool's latest version from source, please visit
     its web page (https://www.gnu.org/software/libtool/).

     Gnuastro's tarball is shipped with an internal implementation of
     GNU Libtool.  Even if you have GNU Libtool, Gnuastro's internal
     implementation is used for the building and installation of
     Gnuastro.  As a result, you can still build, install and use
     Gnuastro even if you do not have GNU Libtool installed on your
     system.  However, this internal Libtool does not get installed.
     Therefore, after Gnuastro's installation, if you want to use *note
     BuildProgram:: to compile and link your own C source code which
     uses the *note Gnuastro library::, you need to have GNU Libtool
     available on your system (independent of Gnuastro).  See *note
     Review of library fundamentals:: to learn more about libraries.

GNU Make extension headers
     GNU Make is a workflow management system that can be used to run a
     series of commands in a specific order, and in parallel if you
     want.  GNU Make offers special features to extend it with custom
     functions within a dynamic library.  They are defined in the
     ‘gnumake.h’ header.  If ‘gnumake.h’ can be found on your system at
     configuration time, Gnuastro will build a custom library that GNU
     Make can use for extended functionality in (astronomical) data
     analysis scenarios.

libgit2
     Git is one of the most common version control systems (see *note
     Version controlled source::).  When ‘libgit2’ is present, and
     Gnuastro's programs are run within a version controlled directory,
     outputs will contain the version number of the working directory's
     repository for future reproducibility.  See the ‘COMMIT’ keyword
     header in *note Output FITS files:: for a discussion.

libjpeg
     libjpeg is only used by ConvertType to read from and write to JPEG
     images, see *note Recognized file formats::.  libjpeg
     (http://www.ijg.org/) is a very basic library that provides tools
     to read and write JPEG images, most Unix-like graphic programs and
     libraries use it.  Therefore you most probably already have it
     installed.  libjpeg-turbo (http://libjpeg-turbo.virtualgl.org/) is
     an alternative to libjpeg.  It uses Single instruction, multiple
     data (SIMD) instructions for ARM based systems that significantly
     decreases the processing time of JPEG compression and decompression
     algorithms.

libtiff
     libtiff is used by ConvertType and the libraries to read TIFF
     images, see *note Recognized file formats::.  libtiff
     (http://www.simplesystems.org/libtiff/) is a very basic library
     that provides tools to read and write TIFF images, most Unix-like
     operating system graphic programs and libraries use it.  Therefore
     even if you do not have it installed, it must be easily available
     in your package manager.

cURL
     cURL's executable (‘curl’) is called by *note Query:: for
     submitting queries to remote datasets and retrieving the results.
     It is not necessary for the build of Gnuastro from source (only a
     warning will be printed if it cannot be found at configure time),
     so if you do not have it at build-time there is no problem.  Just
     be sure to have it when you run ‘astquery’, otherwise you'll get an
     error about not finding ‘curl’.

GPL Ghostscript
     GPL Ghostscript's executable (‘gs’) is called by ConvertType to
     compile a PDF file from a source PostScript file, see *note
     ConvertType::.  Therefore its headers (and libraries) are not
     needed.

Python3 with Numpy
     Python is a high-level programming language and Numpy is the most
     commonly used library within Python to add multi-dimensional arrays
     and matrices.  If you configure Gnuastro with ‘--with-python’ _and_
     version 3 of Python is available with a corresponding Numpy
     Library, Gnuastro's library will be built with some Python-related
     helper functions.  Python wrappers for Gnuastro's library (for
     example, 'pyGnuastro') can use these functions when being built
     from source.  For more on Gnuastro's Python helper functions, see
     *note Python interface::.

     This Python interface is only relevant if you want to build the
     Python wrappers (like 'pyGnuastro') from source.  If you install
     the Gnuastro Python wrapper from a pre-built repository like PyPI,
     this feature of your Gnuastro library won't be used.  Pre-built
     libraries contain the full Gnuastro library that they need within
     them (you don't even need to have Gnuastro at all!).

     *Can't find the Python3 and Numpy of a virtual environment:* make
     sure to set the ‘$PYTHON’ variable to point to the ‘python3’
     command of the virtual environment before running ‘./configure’.
     Note that you don't need to activate the virtual env, just point
     ‘PYTHON’ to its Python3 executable, like the example below:

          $ python3 -m venv test-env    # Setting up the virtual env.
          $ export PYTHON="$(pwd)/test-env/bin/python3"
          $ ./configure                 # Gnuastro's configure script.

SAO DS9
     SAO DS9 (‘ds9’) is a visualization tool for FITS images.
     Gnuastro's ‘astscript-fits-view’ program calls DS9 to visualize
     FITS images.  We have a full appendix on it and how to install it
     in *note SAO DS9::.  Since it is a run-time dependency, it can be
     installed at any later time (after building and installing
     Gnuastro).

TOPCAT
     TOPCAT (‘topcat’) is a visualization tool for astronomical tables
     (most commonly: plotting).  Gnuastro's ‘astscript-fits-view’
     program calls TOPCAT it to visualize tables.  We have a full
     appendix on it and how to install it in *note TOPCAT::.  Since it
     is a run-time dependency, it can be installed at any later time
     (after building and installing Gnuastro).

   ---------- Footnotes ----------

   (1) Note that we want the binary/executable Libtool program which can
be run on the command-line.  In Debian-based operating systems which
separate various parts of a package, you want want ‘libtool-bin’, the
‘libtool’ package will not contain the executable program.


File: gnuastro.info,  Node: Bootstrapping dependencies,  Next: Dependencies from package managers,  Prev: Optional dependencies,  Up: Dependencies

3.1.3 Bootstrapping dependencies
--------------------------------

Bootstrapping is only necessary if you have decided to obtain the full
version controlled history of Gnuastro, see *note Version controlled
source:: and *note Bootstrapping::.  Using the version controlled source
enables you to always be up to date with the most recent development
work of Gnuastro (bug fixes, new functionalities, improved algorithms,
etc.).  If you have downloaded a tarball (see *note Downloading the
source::), then you can ignore this subsection.

   To successfully run the bootstrapping process, there are some
additional dependencies to those discussed in the previous subsections.
These are low level tools that are used by a large collection of
Unix-like operating systems programs, therefore they are most probably
already available in your system.  If they are not already installed,
you should be able to easily find them in any GNU/Linux distribution
package management system (‘apt-get’, ‘yum’, ‘pacman’, etc.).  The short
names in parenthesis in ‘typewriter’ font after the package name can be
used to search for them in your package manager.  For the GNU
Portability Library, GNU Autoconf Archive and TeX Live, it is
recommended to use the instructions here, not your operating system's
package manager.

GNU Portability Library (Gnulib)
     To ensure portability for a wider range of operating systems (those
     that do not include GNU C library, namely glibc), Gnuastro depends
     on the GNU portability library, or Gnulib.  Gnulib keeps a copy of
     all the functions in glibc, implemented (as much as possible) to be
     portable to other operating systems.  The ‘bootstrap’ script can
     automatically clone Gnulib (as a ‘gnulib/’ directory inside
     Gnuastro), however, as described in *note Bootstrapping:: this is
     not recommended.

     The recommended way to bootstrap Gnuastro is to first clone Gnulib
     and the Autoconf archives (see below) into a local directory
     outside of Gnuastro.  Let's call it ‘DEVDIR’(1) (which you can set
     to any directory; preferentially where you keep your other
     development projects).  Currently in Gnuastro, both Gnulib and
     Autoconf archives have to be cloned in the same top directory(2)
     like the case here(3):

          $ DEVDIR=/home/yourname/Development  ## Select any location.
          $ mkdir $DEVDIR                      ## If it doesn't exist!
          $ cd $DEVDIR
          $ git clone https://git.sv.gnu.org/git/gnulib.git
          $ git clone https://git.sv.gnu.org/git/autoconf-archive.git

     Gnulib is a source-based dependency of Gnuastro's bootstrapping
     process, so simply having it is enough on your computer, there is
     no need to install, and thus check anything.

     You now have the full version controlled source of these two
     repositories in separate directories.  Both these packages are
     regularly updated, so every once in a while, you can run ‘$ git
     pull’ within them to get any possible updates.

GNU Automake (‘automake’)
     GNU Automake will build the ‘Makefile.in’ files in each
     sub-directory using the (hand-written) ‘Makefile.am’ files.  The
     ‘Makefile.in’s are subsequently used to generate the ‘Makefile’s
     when the user runs ‘./configure’ before building.

     To check that you have a working GNU Automake in your system, you
     can try this command:

          $ automake --version

GNU Autoconf (‘autoconf’)
     GNU Autoconf will build the ‘configure’ script using the
     configurations we have defined (hand-written) in ‘configure.ac’.

     To check that you have a working GNU Autoconf in your system, you
     can try this command:

          $ autoconf --version

GNU Autoconf Archive
     These are a large collection of tests that can be called to run at
     ‘./configure’ time.  See the explanation under GNU Portability
     Library (Gnulib) above for instructions on obtaining it and keeping
     it up to date.

     GNU Autoconf Archive is a source-based dependency of Gnuastro's
     bootstrapping process, so simply having it is enough on your
     computer, there is no need to install, and thus check anything.
     Just do not forget that it has to be in the same directory as
     Gnulib (described above).

GNU Texinfo (‘texinfo’)
     GNU Texinfo is the tool that formats this manual into the various
     output formats.  To bootstrap Gnuastro you need all of Texinfo's
     command-line programs.  However, some operating systems package
     them separately, for example, in Fedora, ‘makeinfo’ is packaged in
     the ‘texinfo-tex’ package.

     To check that you have a working GNU Texinfo in your system, you
     can try this command:

          $ makeinfo --version

GNU Libtool (‘libtool’)
     GNU Libtool is in charge of building all the libraries in Gnuastro.
     The libraries contain functions that are used by more than one
     program and are installed for use in other programs.  They are thus
     put in a separate directory (‘lib/’).

     To check that you have a working GNU Libtool in your system, you
     can try this command (and from the output, make sure it is GNU's
     libtool)

          $ libtool --version

GNU help2man (‘help2man’)
     GNU help2man is used to convert the output of the ‘--help’ option
     (*note --help::) to the traditional Man page (*note Man pages::).

     To check that you have a working GNU Help2man in your system, you
     can try this command:

          $ help2man --version

LaTeX and some TeX packages
     Some of the figures in this book are built by LaTeX (using the
     PGF/TikZ package).  The LaTeX source for those figures is version
     controlled for easy maintenance not the actual figures.  So the
     ‘./boostrap’ script will run LaTeX to build the figures.  The best
     way to install LaTeX and all the necessary packages is through TeX
     live (https://www.tug.org/texlive/) which is a package manager for
     TeX related tools that is independent of any operating system.  It
     is thus preferred to the TeX Live versions distributed by your
     operating system.

     To install TeX Live, first download its installer, unpack it, enter
     the unpacked directory (which includes the date it was generated!),

          $ baseurl=http://mirrors.rit.edu/CTAN/systems/texlive/tlnet
          $ wget $baseurl/install-tl-unx.tar.gz
          $ tar -xf install-tl-unx.tar.gz
          $ cd install-tl-20*
          $ ./install-tl

     The output of the last command above is an interactive shell that
     will allow you to customize the installation.  There are two
     important parts which you need to edit:

     Basic scheme
          By default the full package repository will be downloaded and
          installed (around 9 Gigabytes!)  which can take _very_ long to
          download and to update later.  However, most packages are not
          needed by everyone!  So it is easier, faster and better to
          install only the "Basic scheme" (consisting of only the most
          basic TeX and LaTeX packages, which is almost 300 Megabytes).
          To do this, from the top interactive installer environment
          press the following keys:
            1. '<S>' to select the installation scheme.
            2. '<d>' to select the basic scheme.
            3. '<R>' to return to the main menu.

     Installation path
          By default, TeXLive will try to install in a system-wide
          location which will require root permissions.  A better
          solution is to install TeXLive in your user-accessible
          directories to avoid the need to become root when installing
          packages or updating.  This also helps in using servers where
          you do not have root access at all.  To install it in another
          directory take the following steps from the main menu:
            1. '<D>' to enter the directory settings.
            2. '<1>' to select the "main tree" (base of all the other
               directories by default).
            3. You will be prompted to enter a new directory.

               To get the directory name, let's open a new/different
               terminal.  Run the following commands to make and get the
               absolute location of the directory (no problem if the
               first command says that ‘$HOME/.local’ already exists).
               If you already have a different place to host
               custom-built software, you can modify this step
               accordingly.
                    $ mkdir $HOME/.local; mkdir $HOME/.local/texlive
                    $ cd $HOME/.local/texlive
                    $ pwd
               Copy the result of the last command above, come back to
               the terminal with the TeXLive interactive installation
               and paste the directory there.  Afterwards, you can close
               the temporary directory.
            4. '<R>' to return to the main menu.

     After the customizations above are implemented simply press the
     '<I>' key to start the installation of TeXLive.  After the
     installation finishes, be sure to set the environment variables as
     suggested in the end of the outputs.  For more on how to insert the
     given directories for the given environment PATHs, see *note
     Installation directory::.

     After the installation and setup of the PATHs are complete, you
     need install all the necessary TeX packages for a successful
     Gnuastro bootstrap.  To do that, run the command below (in case it
     complains about not finding ‘tlmgr’, there is a problem in the
     PATHs above).

          $ tlmgr install epsf jknapltx caption biblatex biber \
                          logreq xstring xkeyval pgf xcolor \
                          pgfplots times rsfs ps2eps epspdf

     Generally (outside of Gnuastro's development), any time you
     confront (need) a package you do not have(4), simply install it
     with the ‘tlmgr’ command above.  It is very similar to how you
     install software from your operating system's package manager).

ImageMagick (‘imagemagick’)
     ImageMagick is a wonderful and robust program for image
     manipulation on the command-line.  ‘bootstrap’ uses it to convert
     the book images into the formats necessary for the various book
     formats.

     Since ImageMagick version 7, it is necessary to edit the policy
     file (‘/etc/ImageMagick-7/policy.xml’) to have the following line
     (it maybe present, but commented, in this case un-comment it):

          <policy domain="coder" rights="read|write" pattern="{PS,PDF,XPS}"/>

     If the following line is present, it is also necessary to
     comment/remove it.

          <policy domain="delegate" rights="none" pattern="gs" />

     To learn more about the ImageMagick security policy please see:
     <https://imagemagick.org/script/security-policy.php>.

     To check that you have a working ImageMagick in your system, you
     can try this command:

          $ convert --version

   ---------- Footnotes ----------

   (1) If you are not a developer in Gnulib or Autoconf archives,
‘DEVDIR’ can be a directory that you do not backup.  In this way the
large number of files in these projects will not slow down your backup
process or take bandwidth (if you backup to a remote server).

   (2) If you already have the Autoconf archives in a separate
directory, or cannot clone it in the same directory as Gnulib, or you
have it with another directory name (not ‘autoconf-archive/’), you can
follow this short step.  Set ‘AUTOCONFARCHIVES’ to your desired address.
Then define a symbolic link in ‘DEVDIR’ with the following command so
Gnuastro's bootstrap script can find it:
‘$ ln -s $AUTOCONFARCHIVES $DEVDIR/autoconf-archive’.

   (3) If your internet connection is active, but Git complains about
the network, it might be due to your network setup not recognizing the
git protocol.  In that case use the following URL for the HTTP protocol
instead (for Autoconf archives, replace the name):
‘http://git.sv.gnu.org/r/gnulib.git’

   (4) After running TeX, or LaTeX, you might get a warning complaining
about a ‘missingfile’.  Run '‘tlmgr info missingfile’' to see the
package(s) containing that file which you can install.


File: gnuastro.info,  Node: Dependencies from package managers,  Prev: Bootstrapping dependencies,  Up: Dependencies

3.1.4 Dependencies from package managers
----------------------------------------

The most basic way to install a package on your system is to build the
packages from source yourself.  Alternatively, you can use your
operating system's package manager to download pre-compiled files and
install them.  The latter choice is easier and faster.  However, we
recommend that you build the *note Mandatory dependencies:: yourself
from source (all necessary commands and links are given in the
respective section).  Here are some basic reasons behind this
recommendation.

  1. Your operating system's pre-built software might not be the most
     recent release.  For example, Gnuastro itself is also packaged in
     some package managers.  For the list see:
     <https://repology.org/project/gnuastro/versions>.  You will notice
     that Gnuastro's version in some operating systems is more than 10
     versions old!  It is the same for all the dependencies of Gnuastro.

  2. For each package, Gnuastro might preform better (or require)
     certain configuration options that your distribution's package
     managers did not add for you.  If present, these configuration
     options are explained during the installation of each in the
     sections below (for example, in *note CFITSIO::).  When the proper
     configuration has not been set, the programs should complain and
     inform you.

  3. For the libraries, they might separate the binary file from the
     header files which can cause confusion, see *note Known issues::.

  4. Like any other tool, the science you derive from Gnuastro's tools
     highly depend on these lower level dependencies, so generally it is
     much better to have a close connection with them.  By reading their
     manuals, installing them and staying up to date with changes/bugs
     in them, your scientific results and understanding (of what is
     going on, and thus how you interpret your scientific results) will
     also correspondingly improve.

   Based on your package manager, you can use any of the following
commands to install the mandatory and optional dependencies.  If your
package manager is not included in the list below, please send us the
respective command, so we add it.  For better archivability and
compression ratios, Gnuastro's recommended tarball compression format is
with the Lzip (http://lzip.nongnu.org/lzip.html) program, see *note
Release tarball::.  Therefore, the package manager commands below also
contain Lzip.

‘apt-get’ (Debian-based OSs: Debian, Ubuntu, Linux Mint, etc.)
     Debian (https://en.wikipedia.org/wiki/Debian) is one of the oldest
     GNU/Linux distributions(1).  It thus has a very extended user
     community and a robust internal structure and standards.  All of it
     is free software and based on the work of volunteers around the
     world.  Many distributions are thus derived from it, for example,
     Ubuntu and Linux Mint.  This arguably makes Debian-based OSs the
     largest, and most used, class of GNU/Linux distributions.  All of
     them use Debian's Advanced Packaging Tool (APT, for example,
     ‘apt-get’) for managing packages.

     Development features (Ubuntu or derivatives)
          By default, a newly installed Ubuntu does not contain the
          low-level tools that are necessary for building a software
          from source.  Therefore, if you are using Ubuntu, please run
          the following command.
               $ sudo apt-get install gcc make zlib1g-dev lzip

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo apt-get install libgsl-dev libcfitsio-dev \
                                      wcslib-dev

     Optional dependencies
          If present, these libraries can be used in Gnuastro's build
          for extra features, see *note Optional dependencies::.
               $ sudo apt-get install ghostscript libtool-bin \
                                      libjpeg-dev libtiff-dev \
                                      libgit2-dev curl

     Programs to view FITS images or tables
          These are not used in Gnuastro's build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ sudo apt-get install saods9 topcat

     Gnuastro is packaged (https://tracker.debian.org/pkg/gnuastro) in
     Debian (and thus some of its derivate operating systems).  Just
     make sure it is the most recent version.

‘dnf’
‘yum’ (Red Hat-based OSs: Red Hat, Fedora, CentOS, Scientific Linux, etc.)
     Red Hat Enterprise Linux (https://en.wikipedia.org/wiki/Red_Hat)
     (RHEL) is released by Red Hat Inc.  RHEL requires paid
     subscriptions for use of its binaries and support.  But since it is
     free software, many other teams use its code to spin-off their own
     distributions based on RHEL. Red Hat-based GNU/Linux distributions
     initially used the "Yellowdog Updated, Modifier" (YUM) package
     manager, which has been replaced by "Dandified yum" (DNF). If the
     latter is not available on your system, you can use ‘yum’ instead
     of ‘dnf’ in the command below.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo dnf install gsl-devel cfitsio-devel \
                                  wcslib-devel

     Optional dependencies
          If present, these libraries can be used in Gnuastro's build
          for extra features, see *note Optional dependencies::.
               $ sudo dnf install ghostscript libtool \
                                  libjpeg-devel libtiff-devel \
                                  libgit2-devel lzip curl

     Programs to view FITS images or tables
          These are not used in Gnuastro's build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ sudo dnf install saods9 topcat

‘brew’ (macOS)
     macOS (https://en.wikipedia.org/wiki/MacOS) is the operating system
     used on Apple devices.  macOS does not come with a package manager
     pre-installed, but several widely used, third-party package
     managers exist, such as Homebrew or MacPorts.  Both are free
     software.  Currently we have only tested Gnuastro's installation
     with Homebrew as described below.  If not already installed, first
     obtain Homebrew by following the instructions at <https://brew.sh>.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!

          Homebrew manages packages in different 'taps'.  To install
          WCSLIB via Homebrew you will need to ‘tap’ into
          ‘brewsci/science’ first (the tap may change in the future, but
          can be found by calling ‘brew search wcslib’).
               $ brew tap brewsci/science
               $ brew install wcslib gsl cfitsio

     Optional dependencies
          If present, these libraries can be used in Gnuastro's build
          for extra features, see *note Optional dependencies::.
               $ brew install ghostscript libtool libjpeg \
                              libtiff libgit2 curl lzip

     Programs to view FITS images or tables
          These are not used in Gnuastro's build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ brew install saoimageds9 topcat

‘pacman’ (Arch Linux)
     Arch Linux (https://en.wikipedia.org/wiki/Arch_Linux) is a smaller
     GNU/Linux distribution, which follows the KISS principle ("keep it
     simple, stupid") as a general guideline.  It "focuses on elegance,
     code correctness, minimalism and simplicity, and expects the user
     to be willing to make some effort to understand the system's
     operation".  Arch GNU/Linux uses "Package manager" (Pacman) to
     manage its packages/components.

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo pacman -S gsl cfitsio wcslib

     Optional dependencies
          If present, these libraries can be used in Gnuastro's build
          for extra features, see *note Optional dependencies::.
               $ sudo pacman -S ghostscript libtool libjpeg \
                                libtiff libgit2 curl lzip

     Programs to view FITS images or tables
          These are not used in Gnuastro's build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!

          SAO DS9 and TOPCAT are not available in the standard Arch
          GNU/Linux repositories.  However, installing and using both is
          very easy from their own web pages, as described in *note SAO
          DS9:: and *note TOPCAT::.

‘zypper’ (openSUSE and SUSE Linux Enterprise Server)
     SUSE Linux Enterprise Server(2) (SLES) is the commercial offering
     which shares code and tools.  Many additional packages are offered
     in the Build Service(3).  openSUSE and SLES use ‘zypper’ (cli) and
     YaST (GUI) for managing repositories and packages.

     Configuration
          When building Gnuastro, run the configure script with the
          following ‘CPPFLAGS’ environment variable:

               $ ./configure CPPFLAGS="-I/usr/include/cfitsio"

     Mandatory dependencies
          Without these, Gnuastro cannot be built, they are necessary
          for input/output and low-level mathematics (see *note
          Mandatory dependencies::)!
               $ sudo zypper install gsl-devel cfitsio-devel \
                                     wcslib-devel

     Optional dependencies
          If present, these libraries can be used in Gnuastro's build
          for extra features, see *note Optional dependencies::.
               $ sudo zypper install ghostscript_any libtool \
                                     pkgconfig libcurl-devel \
                                     libgit2-devel \
                                     libjpeg62-devel \
                                     libtiff-devel curl

     Programs to view FITS images or tables
          These are not used in Gnuastro's build.  They can just help in
          viewing the inputs/outputs independent of Gnuastro!
               $ sudo zypper install ds9 topcat

   Usually, when libraries are installed by operating system package
managers, there should be no problems when configuring and building
other programs from source (that depend on the libraries: Gnuastro in
this case).  However, in some special conditions, problems may pop-up
during the configuration, building, or checking/running any of
Gnuastro's programs.  The most common of such problems and their
solution are discussed below.

*Not finding library during configuration:* If a library is installed,
but during Gnuastro's ‘configure’ step the library is not found, then
configure Gnuastro like the command below (correcting ‘/path/to/lib’).
For more, see *note Known issues:: and *note Installation directory::.
     $ ./configure LDFLAGS="-L/path/to/lib"

*Not finding header (.h) files while building:* If a library is
installed, but during Gnuastro's ‘make’ step, the library's header (file
with a ‘.h’ suffix) is not found, then configure Gnuastro like the
command below (correcting ‘/path/to/include’).  For more, see *note
Known issues:: and *note Installation directory::.
     $ ./configure CPPFLAGS="-I/path/to/include"

*Gnuastro's programs do not run during check or after install:* If a
library is installed, but the programs do not run due to linking
problems, set the ‘LD_LIBRARY_PATH’ variable like below (assuming
Gnuastro is installed in ‘/path/to/installed’).  For more, see *note
Known issues:: and *note Installation directory::.
     $ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/installed/lib"

   ---------- Footnotes ----------

   (1) 
<https://en.wikipedia.org/wiki/List_of_Linux_distributions#Debian-based>

   (2) <https://www.suse.com/products/server>

   (3) <https://build.opensuse.org>


File: gnuastro.info,  Node: Downloading the source,  Next: Build and install,  Prev: Dependencies,  Up: Installation

3.2 Downloading the source
==========================

Gnuastro's source code can be downloaded in two ways.  As a tarball,
ready to be configured and installed on your system (as described in
*note Quick start::), see *note Release tarball::.  If you want official
releases of stable versions this is the best, easiest and most common
option.  Alternatively, you can clone the version controlled history of
Gnuastro, run one extra bootstrapping step and then follow the same
steps as the tarball.  This will give you access to all the most recent
work that will be included in the next release along with the full
project history.  The process is thoroughly introduced in *note Version
controlled source::.

* Menu:

* Release tarball::             Download a stable official release.
* Version controlled source::   Get and use the version controlled source.


File: gnuastro.info,  Node: Release tarball,  Next: Version controlled source,  Prev: Downloading the source,  Up: Downloading the source

3.2.1 Release tarball
---------------------

A release tarball (commonly compressed) is the most common way of
obtaining free and open source software.  A tarball is a snapshot of one
particular moment in the Gnuastro development history along with all the
necessary files to configure, build, and install Gnuastro easily (see
*note Quick start::).  It is very straightforward and needs the least
set of dependencies (see *note Mandatory dependencies::).  Gnuastro has
tarballs for official stable releases and pre-releases for testing.  See
*note Version numbering:: for more on the two types of releases and the
formats of the version numbers.  The URLs for each type of release are
given below.

Official stable releases (<http://ftp.gnu.org/gnu/gnuastro>):
     This URL hosts the official stable releases of Gnuastro.  Always
     use the most recent version (see *note Version numbering::).  By
     clicking on the "Last modified" title of the second column, the
     files will be sorted by their date which you can also use to find
     the latest version.  It is recommended to use a mirror to download
     these tarballs, please visit <http://ftpmirror.gnu.org/gnuastro/>
     and see below.

Pre-release tarballs (<http://alpha.gnu.org/gnu/gnuastro>):
     This URL contains unofficial pre-release versions of Gnuastro.  The
     pre-release versions of Gnuastro here are for enthusiasts to try
     out before an official release.  If there are problems, or bugs
     then the testers will inform the developers to fix before the next
     official release.  See *note Version numbering:: to understand how
     the version numbers here are formatted.  If you want to remain even
     more up-to-date with the developing activities, please clone the
     version controlled source as described in *note Version controlled
     source::.

   Gnuastro's official/stable tarball is released with two formats: Gzip
(with suffix ‘.tar.gz’) and Lzip (with suffix ‘.tar.lz’).  The
pre-release tarballs (after version 0.3) are released only as an Lzip
tarball.  Gzip is a very well-known and widely used compression program
created by GNU and available in most systems.  However, Lzip provides a
better compression ratio and more robust archival capacity.  For
example, Gnuastro 0.3's tarball was 2.9MB and 4.3MB with Lzip and Gzip
respectively, see the Lzip web page
(http://www.nongnu.org/lzip/lzip.html) for more.  Lzip might not be
pre-installed in your operating system, if so, installing it from your
operating system's package manager or from source is very easy and fast
(it is a very small program).

   The GNU FTP server is mirrored (has backups) in various locations on
the globe (<http://www.gnu.org/order/ftp.html>).  You can use the
closest mirror to your location for a more faster download.  Note that
only some mirrors keep track of the pre-release (alpha) tarballs.  Also
note that if you want to download immediately after and announcement
(see *note Announcements::), the mirrors might need some time to
synchronize with the main GNU FTP server.


File: gnuastro.info,  Node: Version controlled source,  Prev: Release tarball,  Up: Downloading the source

3.2.2 Version controlled source
-------------------------------

The publicly distributed Gnuastro tarball (for example,
‘gnuastro-X.X.tar.gz’) does not contain the revision history, it is only
a snapshot of the source code at one significant instant of Gnuastro's
history (specified by the version number, see *note Version
numbering::), ready to be configured and built.  To be able to develop
successfully, the revision history of the code can be very useful to
track when something was added or changed, also some updates that are
not yet officially released might be in it.

   We use Git for the version control of Gnuastro.  For those who are
not familiar with it, we recommend the ProGit book
(https://git-scm.com/book/en).  The whole book is publicly available for
online reading and downloading and does a wonderful job at explaining
the concepts and best practices.

   Let's assume you want to keep Gnuastro in the ‘TOPGNUASTRO’ directory
(can be any directory, change the value below).  The full version
controlled history of Gnuastro can be cloned in ‘TOPGNUASTRO/gnuastro’
by running the following commands(1):

     $ TOPGNUASTRO=/home/yourname/Research/projects/
     $ cd $TOPGNUASTRO
     $ git clone git://git.sv.gnu.org/gnuastro.git

The ‘$TOPGNUASTRO/gnuastro’ directory will contain hand-written (version
controlled) source code for Gnuastro's programs, libraries, this book
and the tests.  All are divided into sub-directories with standard and
very descriptive names.  The version controlled files in the top cloned
directory are either mainly in capital letters (for example, ‘THANKS’
and ‘README’) or mainly written in small-caps (for example,
‘configure.ac’ and ‘Makefile.am’).  The former are non-programming,
standard writing for human readers containing high-level information
about the whole package.  The latter are instructions to customize the
GNU build system for Gnuastro.  For more on Gnuastro's source code
structure, please see *note Developing::.  We will not go any deeper
here.

   The cloned Gnuastro source cannot immediately be configured,
compiled, or installed since it only contains hand-written files, not
automatically generated or imported files which do all the hard work of
the build process.  See *note Bootstrapping:: for the process of
generating and importing those files (it is not too hard!).  Once you
have bootstrapped Gnuastro, you can run the standard procedures (in
*note Quick start::).  Very soon after you have cloned it, Gnuastro's
main ‘master’ branch will be updated on the main repository (since the
developers are actively working on Gnuastro), for the best practices in
keeping your local history in sync with the main repository see *note
Synchronizing::.

* Menu:

* Bootstrapping::               Adding all the automatically generated files.
* Synchronizing::               Keep your local clone up to date.

   ---------- Footnotes ----------

   (1) If your internet connection is active, but Git complains about
the network, it might be due to your network setup not recognizing the
Git protocol.  In that case use the following URL which uses the HTTP
protocol instead: ‘http://git.sv.gnu.org/r/gnuastro.git’


File: gnuastro.info,  Node: Bootstrapping,  Next: Synchronizing,  Prev: Version controlled source,  Up: Version controlled source

3.2.2.1 Bootstrapping
.....................

The version controlled source code lacks the source files that we have
not written or are automatically built.  These automatically generated
files are included in the distributed tarball for each distribution (for
example, ‘gnuastro-X.X.tar.gz’, see *note Version numbering::) and make
it easy to immediately configure, build, and install Gnuastro.  However
from the perspective of version control, they are just bloatware and
sources of confusion (since they are not changed by Gnuastro
developers).

   The process of automatically building and importing necessary files
into the cloned directory is known as _bootstrapping_.  After
bootstrapping is done you are ready to follow the default GNU build
steps that you normally run on the tarball (‘./configure && make’ for
example, described more in *note Quick start::).  Some known issues with
bootstrapping may occur during the process, to see how to fix them,
please see *note Known issues::.

   All the instructions for an automatic bootstrapping are available in
‘bootstrap’ and configured using ‘bootstrap.conf’.  ‘bootstrap’ and
‘COPYING’ (which contains the software copyright notice) are the only
files not written by Gnuastro developers but under version control to
enable simple bootstrapping and legal information on usage immediately
after cloning.  ‘bootstrap.conf’ is maintained by the GNU Portability
Library (Gnulib) and this file is an identical copy, so do not make any
changes in this file since it will be replaced when Gnulib releases an
update.  Make all your changes in ‘bootstrap.conf’.

   The bootstrapping process has its own separate set of dependencies,
the full list is given in *note Bootstrapping dependencies::.  They are
generally very low-level and used by a very large set of commonly used
programs, so they are probably already installed on your system.  The
simplest way to bootstrap Gnuastro is to simply run the bootstrap script
within your cloned Gnuastro directory as shown below.  However, please
read the next paragraph before doing so (see *note Version controlled
source:: for ‘TOPGNUASTRO’).

     $ cd TOPGNUASTRO/gnuastro
     $ ./bootstrap                      # Requires internet connection

   Without any options, ‘bootstrap’ will clone Gnulib within your cloned
Gnuastro directory (‘TOPGNUASTRO/gnuastro/gnulib’) and download the
necessary Autoconf archives macros.  So if you run bootstrap like this,
you will need an internet connection every time you decide to bootstrap.
Also, Gnulib is a large package and cloning it can be slow.  It will
also keep the full Gnulib repository within your Gnuastro repository, so
if another one of your projects also needs Gnulib, and you insist on
running bootstrap like this, you will have two copies.  In case you
regularly backup your important files, Gnulib will also slow down the
backup process.  Therefore while the simple invocation above can be used
with no problem, it is not recommended.  To do better, see the next
paragraph.

   The recommended way to get these two packages is thoroughly discussed
in *note Bootstrapping dependencies:: (in short: clone them in the
separate ‘DEVDIR/’ directory).  The following commands will take you
into the cloned Gnuastro directory and run the ‘bootstrap’ script, while
telling it to copy some files (instead of making symbolic links, with
the ‘--copy’ option, this is not mandatory(1)) and where to look for
Gnulib (with the ‘--gnulib-srcdir’ option).  Please note that the
address given to ‘--gnulib-srcdir’ has to be an absolute address (so do
not use ‘~’ or ‘../’ for example).

     $ cd $TOPGNUASTRO/gnuastro
     $ ./bootstrap --copy --gnulib-srcdir=$DEVDIR/gnulib

   Since Gnulib and Autoconf archives are now available in your local
directories, you do not need an internet connection every time you
decide to remove all un-tracked files and redo the bootstrap (see box
below).  You can also use the same command on any other project that
uses Gnulib.  All the necessary GNU C library functions, Autoconf macros
and Automake inputs are now available along with the book figures.  The
standard GNU build system (*note Quick start::) will do the rest of the
job.

*Undoing the bootstrap:* During the development, it might happen that
you want to remove all the automatically generated and imported files.
In other words, you might want to reverse the bootstrap process.
Fortunately Git has a good program for this job: ‘git clean’.  Run the
following command and every file that is not version controlled will be
removed.

     git clean -fxd

It is best to commit any recent change before running this command.  You
might have created new files since the last commit and if they have not
been committed, they will all be gone forever (using ‘rm’).  To get a
list of the non-version controlled files instead of deleting them, add
the ‘n’ option to ‘git clean’, so it becomes ‘-fxdn’.

   Besides the ‘bootstrap’ and ‘bootstrap.conf’, the ‘bootstrapped/’
directory and ‘README-hacking’ file are also related to the
bootstrapping process.  The former hosts all the imported (bootstrapped)
directories.  Thus, in the version controlled source, it only contains a
‘README’ file, but in the distributed tarball it also contains
sub-directories filled with all bootstrapped files.  ‘README-hacking’
contains a summary of the bootstrapping process discussed in this
section.  It is a necessary reference when you have not built this book
yet.  It is thus not distributed in the Gnuastro tarball.

   ---------- Footnotes ----------

   (1) The ‘--copy’ option is recommended because some backup systems
might do strange things with symbolic links.


File: gnuastro.info,  Node: Synchronizing,  Prev: Bootstrapping,  Up: Version controlled source

3.2.2.2 Synchronizing
.....................

The bootstrapping script (see *note Bootstrapping::) is not regularly
needed: you mainly need it after you have cloned Gnuastro (once) and
whenever you want to re-import the files from Gnulib, or Autoconf
archives(1) (not too common).  However, Gnuastro developers are
constantly working on Gnuastro and are pushing their changes to the
official repository.  Therefore, your local Gnuastro clone will soon be
out-dated.  Gnuastro has two mailing lists dedicated to its developing
activities (see *note Developing mailing lists::).  Subscribing to them
can help you decide when to synchronize with the official repository.

   To pull all the most recent work in Gnuastro, run the following
command from the top Gnuastro directory.  If you do not already have a
built system, ignore ‘make distclean’.  The separate steps are described
in detail afterwards.

     $ make distclean && git pull && autoreconf -f

You can also run the commands separately:

     $ make distclean
     $ git pull
     $ autoreconf -f

   If Gnuastro was already built in this directory, you do not want some
outputs from the previous version being mixed with outputs from the
newly pulled work.  Therefore, the first step is to clean/delete all the
built files with ‘make distclean’.  Fortunately the GNU build system
allows the separation of source and built files (in separate
directories).  This is a great feature to keep your source directory
clean and you can use it to avoid the cleaning step.  Gnuastro comes
with a script with some useful options for this job.  It is useful if
you regularly pull recent changes, see *note Separate build and source
directories::.

   After the pull, we must re-configure Gnuastro with ‘autoreconf -f’
(part of GNU Autoconf).  It will update the ‘./configure’ script and all
the ‘Makefile.in’(2) files based on the hand-written configurations (in
‘configure.ac’ and the ‘Makefile.am’ files).  After running ‘autoreconf
-f’, a warning about ‘TEXI2DVI’ might show up, you can ignore that.

   The most important reason for rebuilding Gnuastro's build system is
to generate/update the version number for your updated Gnuastro
snapshot.  This generated version number will include the commit
information (see *note Version numbering::).  The version number is
included in nearly all outputs of Gnuastro's programs, therefore it is
vital for reproducing an old result.

   As a summary, be sure to run '‘autoreconf -f’' after every change in
the Git history.  This includes synchronization with the main server or
even a commit you have made yourself.

   If you would like to see what has changed since you last synchronized
your local clone, you can take the following steps instead of the simple
command above (do not type anything after ‘#’):

     $ git checkout master             # Confirm if you are on master.
     $ git fetch origin                # Fetch all new commits from server.
     $ git log master..origin/master   # See all the new commit messages.
     $ git merge origin/master         # Update your master branch.
     $ autoreconf -f                   # Update the build system.

By default ‘git log’ prints the most recent commit first, add the
‘--reverse’ option to see the changes chronologically.  To see exactly
what has been changed in the source code along with the commit message,
add a ‘-p’ option to the ‘git log’.

   If you want to make changes in the code, have a look at *note
Developing:: to get started easily.  Be sure to commit your changes in a
separate branch (keep your ‘master’ branch to follow the official
repository) and re-run ‘autoreconf -f’ after the commit.  If you intend
to send your work to us, you can safely use your commit since it will be
ultimately recorded in Gnuastro's official history.  If not, please
upload your separate branch to a public hosting service, for example,
Codeberg (https://codeberg.org), and link to it in your report/paper.
Alternatively, run ‘make distcheck’ and upload the output
‘gnuastro-X.X.X.XXXX.tar.gz’ to a publicly accessible web page so your
results can be considered scientific (reproducible) later.

   ---------- Footnotes ----------

   (1) <https://savannah.gnu.org/task/index.php?13993> is defined for
you to check if significant (for Gnuastro) updates are made in these
repositories, since the last time you pulled from them.

   (2) In the GNU build system, ‘./configure’ will use the ‘Makefile.in’
files to create the necessary ‘Makefile’ files that are later read by
‘make’ to build the package.


File: gnuastro.info,  Node: Build and install,  Prev: Downloading the source,  Up: Installation

3.3 Build and install
=====================

This section is basically a longer explanation to the sequence of
commands given in *note Quick start::.  If you did not have any problems
during the *note Quick start:: steps, you want to have all the programs
of Gnuastro installed in your system, you do not want to change the
executable names during or after installation, you have root access to
install the programs in the default system wide directory, the Letter
paper size of the print book is fine for you or as a summary you do not
feel like going into the details when everything is working, you can
safely skip this section.

   If you have any of the above problems or you want to understand the
details for a better control over your build and install, read along.
The dependencies which you will need prior to configuring, building and
installing Gnuastro are explained in *note Dependencies::.  The first
three steps in *note Quick start:: need no extra explanation, so we will
skip them and start with an explanation of Gnuastro specific
configuration options and a discussion on the installation directory in
*note Configuring::, followed by some smaller subsections: *note
Tests::, *note A4 print book::, and *note Known issues:: which explains
the solutions to known problems you might encounter in the installation
steps and ways you can solve them.

* Menu:

* Configuring::                 Configure Gnuastro
* Separate build and source directories::  Keeping derivate/build files separate.
* Tests::                       Run tests to see if it is working.
* A4 print book::               Customize the print book.
* Known issues::                Issues you might encounter.


File: gnuastro.info,  Node: Configuring,  Next: Separate build and source directories,  Prev: Build and install,  Up: Build and install

3.3.1 Configuring
-----------------

The ‘$ ./configure’ step is the most important step in the build and
install process.  All the required packages, libraries, headers and
environment variables are checked in this step.  The behaviors of make
and make install can also be set through command-line options to this
command.

   The configure script accepts various arguments and options which
enable the final user to highly customize whatever she is building.  The
options to configure are generally very similar to normal program
options explained in *note Arguments and options::.  Similar to all GNU
programs, you can get a full list of the options along with a short
explanation by running

     $ ./configure --help

A complete explanation is also included in the ‘INSTALL’ file.  Note
that this file was written by the authors of GNU Autoconf (which builds
the ‘configure’ script), therefore it is common for all programs which
use the ‘$ ./configure’ script for building and installing, not just
Gnuastro.  Here we only discuss cases where you do not have superuser
access to the system and if you want to change the executable names.
But before that, a review of the options to configure that are
particular to Gnuastro are discussed.

* Menu:

* Gnuastro configure options::  Configure options particular to Gnuastro.
* Installation directory::      Specify the directory to install.
* Executable names::            Changing executable names.
* Configure and build in RAM::  For minimal use of HDD or SSD, and clean source.


File: gnuastro.info,  Node: Gnuastro configure options,  Next: Installation directory,  Prev: Configuring,  Up: Configuring

3.3.1.1 Gnuastro configure options
..................................

Most of the options to configure (which are to do with building) are
similar for every program which uses this script.  Here the options that
are particular to Gnuastro are discussed.  The next topics explain the
usage of other configure options which can be applied to any program
using the GNU build system (through the configure script).

‘--enable-debug’
     Compile/build Gnuastro with debugging information, no optimization
     and without shared libraries.

     In order to allow more efficient programs when using Gnuastro
     (after the installation), by default Gnuastro is built with a 3rd
     level (a very high level) optimization and no debugging
     information.  By default, libraries are also built for static _and_
     shared linking (see *note Linking::).  However, when there are
     crashes or unexpected behavior, these three features can hinder the
     process of localizing the problem.  This configuration option is
     identical to manually calling the configuration script with
     ‘CFLAGS="-g -O0" --disable-shared’.

     In the (rare) situations where you need to do your debugging on the
     shared libraries, do not use this option.  Instead run the
     configure script by explicitly setting ‘CFLAGS’ like this:
          $ ./configure CFLAGS="-g -O0"

‘--enable-check-with-valgrind’
     Do the ‘make check’ tests through Valgrind.  Therefore, if any
     crashes or memory-related issues (segmentation faults in
     particular) occur in the tests, the output of Valgrind will also be
     put in the ‘tests/test-suite.log’ file without having to manually
     modify the check scripts.  This option will also activate
     Gnuastro's debug mode (see the ‘--enable-debug’ configure-time
     option described above).

     Valgrind is free software.  It is a program for easy checking of
     memory-related issues in programs.  It runs a program within its
     own controlled environment and can thus identify the exact
     line-number in the program's source where a memory-related issue
     occurs.  However, it can significantly slow-down the tests.  So
     this option is only useful when a segmentation fault is found
     during ‘make check’.

‘--enable-progname’
     Only build and install ‘progname’ along with any other program that
     is enabled in this fashion.  ‘progname’ is the name of the
     executable without the ‘ast’, for example, ‘crop’ for Crop (with
     the executable name of ‘astcrop’).

     Note that by default all the programs will be installed.  This
     option (and the ‘--disable-progname’ options) are only relevant
     when you do not want to install all the programs.  Therefore, if
     this option is called for any of the programs in Gnuastro, any
     program which is not explicitly enabled will not be built or
     installed.

‘--disable-progname’
‘--enable-progname=no’
     Do not build or install the program named ‘progname’.  This is very
     similar to the ‘--enable-progname’, but will build and install all
     the other programs except this one.

     *Note:* If some programs are enabled and some are disabled, it is
     equivalent to simply enabling those that were enabled.  Listing the
     disabled programs is redundant.

‘--enable-gnulibcheck’
     Enable checks on the GNU Portability Library (Gnulib).  Gnulib is
     used by Gnuastro to enable users of non-GNU based operating systems
     (that do not use GNU C library or glibc) to compile and use the
     advanced features that this library provides.  We make extensive
     use of such functions.  If you give this option to ‘$ ./configure’,
     when you run ‘$ make check’, first the functions in Gnulib will be
     tested, then the Gnuastro executables.  If your operating system
     does not support glibc or has an older version of it and you have
     problems in the build process (‘$ make’), you can give this flag to
     configure to see if the problem is caused by Gnulib not supporting
     your operating system or Gnuastro, see *note Known issues::.

‘--disable-guide-message’
‘--enable-guide-message=no’
     Do not print a guiding message during the GNU Build process of
     *note Quick start::.  By default, after each step, a message is
     printed guiding the user what the next command should be.
     Therefore, after ‘./configure’, it will suggest running ‘make’.
     After ‘make’, it will suggest running ‘make check’ and so on.  If
     Gnuastro is configured with this option, for example
          $ ./configure --disable-guide-message
     Then these messages will not be printed after any step (like most
     programs).  For people who are not yet fully accustomed to this
     build system, these guidelines can be very useful and encouraging.
     However, if you find those messages annoying, use this option.

‘--without-libgit2’
     Build Gnuastro without libgit2 (for including Git commit hashes in
     output files), see *note Optional dependencies::.  libgit2 is an
     optional dependency, with this option, Gnuastro will ignore any
     possibly existing libgit2 that may already be on the system.

‘--without-libjpeg’
     Build Gnuastro without libjpeg (for reading/writing to JPEG files),
     see *note Optional dependencies::.  libjpeg is an optional
     dependency, with this option, Gnuastro will ignore any possibly
     existing libjpeg that may already be on the system.

‘--without-libtiff’
     Build Gnuastro without libtiff (for reading/writing to TIFF files),
     see *note Optional dependencies::.  libtiff is an optional
     dependency, with this option, Gnuastro will ignore any possibly
     existing libtiff that may already be on the system.

‘--with-python’
     Build the Python interface within Gnuastro's dynamic library.  This
     interface can be used for easy communication with Python wrappers
     (for example, the pyGnuastro package).

     When you install the pyGnuastro package from PyPI, the correct
     configuration of the Gnuastro Library is already packaged with it
     (with the Python interface) and that is independent of your
     Gnuastro installation.  The Python interface is only necessary if
     you want to build pyGnuastro from source (which is only necessary
     for developers).  Therefore it has to be explicitly activated at
     configure time with this option.  For more on the interface
     functions, see *note Python interface::.

   The tests of some programs might depend on the outputs of the tests
of other programs.  For example, MakeProfiles is one the first programs
to be tested when you run ‘$ make check’.  MakeProfiles' test outputs
(FITS images) are inputs to many other programs (which in turn provide
inputs for other programs).  Therefore, if you do not install
MakeProfiles for example, the tests for many the other programs will be
skipped.  To avoid this, in one run, you can install all the programs
and run the tests but not install.  If everything is working correctly,
you can run configure again with only the programs you want.  However,
do not run the tests and directly install after building.


File: gnuastro.info,  Node: Installation directory,  Next: Executable names,  Prev: Gnuastro configure options,  Up: Configuring

3.3.1.2 Installation directory
..............................

One of the most commonly used options to ‘./configure’ is ‘--prefix’, it
is used to define the directory that will host all the installed files
(or the "prefix" in their final absolute file name).  For example, when
you are using a server and you do not have administrator or root access.
In this example scenario, if you do not use the ‘--prefix’ option, you
will not be able to install the built files and thus access them from
anywhere without having to worry about where they are installed.
However, once you prepare your startup file to look into the proper
place (as discussed thoroughly below), you will be able to easily use
this option and benefit from any software you want to install without
having to ask the system administrators or install and use a different
version of a software that is already installed on the server.

   The most basic way to run an executable is to explicitly write its
full file name (including all the directory information) and run it.
One example is running the configuration script with the ‘$ ./configure’
command (see *note Quick start::).  By giving a specific directory (the
current directory or ‘./’), we are explicitly telling the shell to look
in the current directory for an executable file named '‘configure’'.
Directly specifying the directory is thus useful for executables in the
current (or nearby) directories.  However, when the program (an
executable file) is to be used a lot, specifying all those directories
will become a significant burden.  For example, the ‘ls’ executable
lists the contents in a given directory and it is (usually) installed in
the ‘/usr/bin/’ directory by the operating system maintainers.
Therefore, if using the full address was the only way to access an
executable, each time you wanted a listing of a directory, you would
have to run the following command (which is very inconvenient, both in
writing and in remembering the various directories).

     $ /usr/bin/ls

   To address this problem, we have the ‘PATH’ environment variable.  To
understand it better, we will start with a short introduction to the
shell variables.  Shell variable values are basically treated as strings
of characters.  For example, it does not matter if the value is a name
(string of _alphabetic_ characters), or a number (string of _numeric_
characters), or both.  You can define a variable and a value for it by
running
     $ myvariable1=a_test_value
     $ myvariable2="a test value"
As you see above, if the value contains white space characters, you have
to put the whole value (including white space characters) in double
quotes (<">).  You can see the value it represents by running
     $ echo $myvariable1
     $ echo $myvariable2
If a variable has no value or it was not defined, the last command will
only print an empty line.  A variable defined like this will be known as
long as this shell or terminal is running.  Other terminals will have no
idea it existed.  The main advantage of shell variables is that if they
are exported(1), subsequent programs that are run within that shell can
access their value.  So by changing their value, you can change the
"environment" of a program which uses them.  The shell variables which
are accessed by programs are therefore known as "environment
variables"(2).  You can see the full list of exported variables that
your shell recognizes by running:

     $ printenv

   ‘HOME’ is one commonly used environment variable, it is any user's
(the one that is logged in) top directory.  Try finding it in the
command above.  It is used so often that the shell has a special
expansion (alternative) for it: '‘~’'.  Whenever you see file names
starting with the tilde sign, it actually represents the value to the
‘HOME’ environment variable, so ‘~/doc’ is the same as ‘$HOME/doc’.

   Another one of the most commonly used environment variables is
‘PATH’, it is a list of directories to search for executable names.  Its
value is a list of directories (separated by a colon, or '<:>').  When
the address of the executable is not explicitly given (like
‘./configure’ above), the system will look for the executable in the
directories specified by ‘PATH’.  If you have a computer nearby, try
running the following command to see which directories your system will
look into when it is searching for executable (binary) files, one
example is printed here (notice how ‘/usr/bin’, in the ‘ls’ example
above, is one of the directories in ‘PATH’):

     $ echo $PATH
     /usr/local/sbin:/usr/local/bin:/usr/bin

   By default ‘PATH’ usually contains system-wide directories, which are
readable (but not writable) by all users, like the above example.
Therefore if you do not have root (or administrator) access, you need to
add another directory to ‘PATH’ which you actually have write access to.
The standard directory where you can keep installed files (not just
executables) for your own user is the ‘~/.local/’ directory.  The names
of hidden files start with a '<.>' (dot), so it will not show up in your
common command-line listings, or on the graphical user interface.  You
can use any other directory, but this is the most recognized.

   The top installation directory will be used to keep all the package's
components: programs (executables), libraries, include (header) files,
shared data (like manuals), or configuration files (see *note Review of
library fundamentals:: for a thorough introduction to headers and
linking).  So it commonly has some of the following sub-directories for
each class of installed components respectively: ‘bin/’, ‘lib/’,
‘include/’ ‘man/’, ‘share/’, ‘etc/’.  Since the ‘PATH’ variable is only
used for executables, you can add the ‘~/.local/bin’ directory (which
keeps the executables/programs or more generally, "binary" files) to
‘PATH’ with the following command.  As defined below, first the existing
value of ‘PATH’ is used, then your given directory is added to its end
and the combined value is put back in ‘PATH’ (run '‘$ echo $PATH’'
afterwards to check if it was added).

     $ PATH=$PATH:~/.local/bin

   Any executable that you installed in ‘~/.local/bin’ will now be
usable without having to remember and write its full address.  However,
as soon as you leave/close your current terminal session, this modified
‘PATH’ variable will be forgotten.  Adding the directories which contain
executables to the ‘PATH’ environment variable each time you start a
terminal is also very inconvenient and prone to errors.  Fortunately,
there are standard 'startup files' defined by your shell precisely for
this (and other) purposes.  There is a special startup file for every
significant starting step:

‘/etc/profile’ and everything in ‘/etc/profile.d/’
     These startup scripts are called when your whole system starts (for
     example, after you turn on your computer).  Therefore you need
     administrator or root privileges to access or modify them.

‘~/.bash_profile’
     If you are using (GNU) Bash as your shell, the commands in this
     file are run, when you log in to your account _through Bash_.  Most
     commonly when you login through the virtual console (where there is
     no graphic user interface).

‘~/.bashrc’
     If you are using (GNU) Bash as your shell, the commands here will
     be run each time you start a terminal and are already logged in.
     For example, when you open your terminal emulator in the graphic
     user interface.

   For security reasons, it is highly recommended to directly type in
your ‘HOME’ directory value by hand in startup files instead of using
variables.  So in the following, let's assume your user name is '‘name’'
(so ‘~’ may be replaced with ‘/home/name’).  To add ‘~/.local/bin’ to
your ‘PATH’ automatically on any startup file, you have to "export" the
new value of ‘PATH’ in the startup file that is most relevant to you by
adding this line:

     export PATH=$PATH:/home/name/.local/bin

   Now that you know your system will look into ‘~/.local/bin’ for
executables, you can tell Gnuastro's configure script to install
everything in the top ‘~/.local’ directory using the ‘--prefix’ option.
When you subsequently run ‘$ make install’, all the install-able files
will be put in their respective directory under ‘~/.local/’ (the
executables in ‘~/.local/bin’, the compiled library files in
‘~/.local/lib’, the library header files in ‘~/.local/include’ and so
on, to learn more about these different files, please see *note Review
of library fundamentals::).  Note that tilde ('<~>') expansion will not
happen if you put a '<=>' between ‘--prefix’ and ‘~/.local’(3), so we
have avoided the <=> character here which is optional in GNU-style
options, see *note Options::.

     $ ./configure --prefix ~/.local

   You can install everything (including libraries like GSL, CFITSIO, or
WCSLIB which are Gnuastro's mandatory dependencies, see *note Mandatory
dependencies::) locally by configuring them as above.  However, recall
that ‘PATH’ is only for executable files, not libraries and that
libraries can also depend on other libraries.  For example, WCSLIB
depends on CFITSIO and Gnuastro needs both.  Therefore, when you
installed a library in a non-recognized directory, you have to guide the
program that depends on them to look into the necessary library and
header file directories.  To do that, you have to define the ‘LDFLAGS’
and ‘CPPFLAGS’ environment variables respectively.  This can be done
while calling ‘./configure’ as shown below:

     $ ./configure LDFLAGS=-L/home/name/.local/lib            \
                   CPPFLAGS=-I/home/name/.local/include       \
                   --prefix ~/.local

   It can be annoying/buggy to do this when configuring every software
that depends on such libraries.  Hence, you can define these two
variables in the most relevant startup file (discussed above).  The
convention on using these variables does not include a colon to separate
values (as ‘PATH’-like variables do).  They use white space characters
and each value is prefixed with a compiler option(4).  Note the ‘-L’ and
‘-I’ above (see *note Options::), for ‘-I’ see *note Headers::, and for
‘-L’, see *note Linking::.  Therefore we have to keep the value in
double quotation signs to keep the white space characters and adding the
following two lines to the startup file of choice:

     export LDFLAGS="$LDFLAGS -L/home/name/.local/lib"
     export CPPFLAGS="$CPPFLAGS -I/home/name/.local/include"

   Dynamic libraries are linked to the executable every time you run a
program that depends on them (see *note Linking:: to fully understand
this important concept).  Hence dynamic libraries also require a special
path variable called ‘LD_LIBRARY_PATH’ (same formatting as ‘PATH’).  To
use programs that depend on these libraries, you need to add
‘~/.local/lib’ to your ‘LD_LIBRARY_PATH’ environment variable by adding
the following line to the relevant start-up file:

     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/name/.local/lib

   If you also want to access the Info (see *note Info::) and man pages
(see *note Man pages::) documentations add ‘~/.local/share/info’ and
‘~/.local/share/man’ to your ‘INFOPATH’(5) and ‘MANPATH’ environment
variables respectively.

   A final note is that order matters in the directories that are
searched for all the variables discussed above.  In the examples above,
the new directory was added after the system specified directories.  So
if the program, library or manuals are found in the system wide
directories, the user directory is no longer searched.  If you want to
search your local installation first, put the new directory before the
already existing list, like the example below.

     export LD_LIBRARY_PATH=/home/name/.local/lib:$LD_LIBRARY_PATH

This is good when a library, for example, CFITSIO, is already present on
the system, but the system-wide install was not configured with the
correct configuration flags (see *note CFITSIO::), or you want to use a
newer version and you do not have administrator or root access to update
it on the whole system/server.  If you update ‘LD_LIBRARY_PATH’ by
placing ‘~/.local/lib’ first (like above), the linker will first find
the CFITSIO you installed for yourself and link with it.  It thus will
never reach the system-wide installation.

   There are important security problems with using local installations
first: all important system-wide executables and libraries (important
executables like ‘ls’ and ‘cp’, or libraries like the C library) can be
replaced by non-secure versions with the same file names and put in the
customized directory (‘~/.local’ in this example).  So if you choose to
search in your customized directory first, please _be sure_ to keep it
clean from executables or libraries with the same names as important
system programs or libraries.

*Summary:* When you are using a server which does not give you
administrator/root access AND you would like to give priority to your
own built programs and libraries, not the version that is (possibly
already) present on the server, add these lines to your startup file.
See above for which startup file is best for your case and for a
detailed explanation on each.  Do not forget to replace
'‘/YOUR-HOME-DIR’' with your home directory (for example,
'‘/home/your-id’'):

     export PATH="/YOUR-HOME-DIR/.local/bin:$PATH"
     export LDFLAGS="-L/YOUR-HOME-DIR/.local/lib $LDFLAGS"
     export MANPATH="/YOUR-HOME-DIR/.local/share/man/:$MANPATH"
     export CPPFLAGS="-I/YOUR-HOME-DIR/.local/include $CPPFLAGS"
     export INFOPATH="/YOUR-HOME-DIR/.local/share/info/:$INFOPATH"
     export LD_LIBRARY_PATH="/YOUR-HOME-DIR/.local/lib:$LD_LIBRARY_PATH"

Afterwards, you just need to add an extra
‘--prefix=/YOUR-HOME-DIR/.local’ to the ‘./configure’ command of the
software that you intend to install.  Everything else will be the same
as a standard build and install, see *note Quick start::.

   ---------- Footnotes ----------

   (1) By running ‘$ export myvariable=a_test_value’ instead of the
simpler case in the text

   (2) You can use shell variables for other actions too, for example,
to temporarily keep some names or run loops on some files.

   (3) If you insist on using '<=>', you can use
‘--prefix=$HOME/.local’.

   (4) These variables are ultimately used as options while building the
programs.  Therefore every value has be an option name followed be a
value as discussed in *note Options::.

   (5) Info has the following convention: "If the value of ‘INFOPATH’
ends with a colon [or it is not defined] ..., the initial list of
directories is constructed by appending the build-time default to the
value of ‘INFOPATH’."  So when installing in a non-standard directory
and if ‘INFOPATH’ was not initially defined, add a colon to the end of
‘INFOPATH’ as shown below.  Otherwise Info will not be able to find
system-wide installed documentation:
‘echo 'export INFOPATH=$INFOPATH:/home/name/.local/share/info:' >>
~/.bashrc’
Note that this is only an internal convention of Info: do not use it for
other ‘*PATH’ variables.


File: gnuastro.info,  Node: Executable names,  Next: Configure and build in RAM,  Prev: Installation directory,  Up: Configuring

3.3.1.3 Executable names
........................

At first sight, the names of the executables for each program might seem
to be uncommonly long, for example, ‘astnoisechisel’ or ‘astcrop’.  We
could have chosen terse (and cryptic) names like most programs do.  We
chose this complete naming convention (something like the commands in
TeX) so you do not have to spend too much time remembering what the name
of a specific program was.  Such complete names also enable you to
easily search for the programs.

   To facilitate typing the names in, we suggest using the shell
auto-complete.  With this facility you can find the executable you want
very easily.  It is very similar to file name completion in the shell.
For example, simply by typing the letters below (where <[TAB]> stands
for the Tab key on your keyboard)

     $ ast[TAB][TAB]

you will get the list of all the available executables that start with
‘ast’ in your ‘PATH’ environment variable directories.  So, all the
Gnuastro executables installed on your system will be listed.  Typing
the next letter for the specific program you want along with a Tab, will
limit this list until you get to your desired program.

   In case all of this does not convince you and you still want to type
short names, some suggestions are given below.  You should have in mind
though, that if you are writing a shell script that you might want to
pass on to others, it is best to use the standard name because other
users might not have adopted the same customization.  The long names
also serve as a form of documentation in such scripts.  A similar
reasoning can be given for option names in scripts: it is good practice
to always use the long formats of the options in shell scripts, see
*note Options::.

   The simplest solution is making a symbolic link to the actual
executable.  For example, let's assume you want to type ‘ic’ to run Crop
instead of ‘astcrop’.  Assuming you installed Gnuastro executables in
‘/usr/local/bin’ (default) you can do this simply by running the
following command as root:

     # ln -s /usr/local/bin/astcrop /usr/local/bin/ic

In case you update Gnuastro and a new version of Crop is installed, the
default executable name is the same, so your custom symbolic link still
works.

   The installed executable names can also be set using options to ‘$
./configure’, see *note Configuring::.  GNU Autoconf (which configures
Gnuastro for your particular system), allows the builder to change the
name of programs with the three options ‘--program-prefix’,
‘--program-suffix’ and ‘--program-transform-name’.  The first two are
for adding a fixed prefix or suffix to all the programs that will be
installed.  This will actually make all the names longer!  You can use
it to add versions of program names to the programs in order to
simultaneously have two executable versions of a program.

   The third configure option allows you to set the executable name at
install time using the SED program.  SED is a very useful 'stream
editor'.  There are various resources on the internet to use it
effectively.  However, we should caution that using configure options
will change the actual executable name of the installed program and on
every re-install (an update for example), you have to also add this
option to keep the old executable name updated.  Also note that the
documentation or configuration files do not change from their standard
names either.

   For example, let's assume that typing ‘ast’ on every invocation of
every program is really annoying you!  You can remove this prefix from
all the executables at configure time by adding this option:

     $ ./configure --program-transform-name='s/ast/ /'


File: gnuastro.info,  Node: Configure and build in RAM,  Prev: Executable names,  Up: Configuring

3.3.1.4 Configure and build in RAM
..................................

Gnuastro's configure and build process (the GNU build system) involves
the creation, reading, and modification of a large number of files
(input/output, or I/O). Therefore file I/O issues can directly affect
the work of developers who need to configure and build Gnuastro numerous
times.  Some of these issues are listed below:

   • I/O will cause wear and tear on both the HDDs (mechanical failures)
     and SSDs (decreasing the lifetime).

   • Having the built files mixed with the source files can greatly
     affect backing up (synchronization) of source files (since it
     involves the management of a large number of small files that are
     regularly changed.  Backup software can of course be configured to
     ignore the built files and directories.  However, since the built
     files are mixed with the source files and can have a large variety,
     this will require a high level of customization.

   One solution to address both these problems is to use the tmpfs file
system (https://en.wikipedia.org/wiki/Tmpfs).  Any file in tmpfs is
actually stored in the RAM (and possibly SWAP), not on HDDs or SSDs.
The RAM is built for extensive and fast I/O. Therefore the large number
of file I/Os associated with configuring and building will not harm the
HDDs or SSDs.  Due to the volatile nature of RAM, files in the tmpfs
file-system will be permanently lost after a power-off.  Since all
configured and built files are derivative files (not files that have
been directly written by hand) there is no problem in this and this
feature can be considered as an automatic cleanup.

   The modern GNU C library (and thus the Linux kernel) defines the
‘/dev/shm’ directory for this purpose in the RAM (POSIX shared memory).
To build in it, you can use the GNU build system's ability to build in a
separate directory (not necessarily in the source directory) as shown
below.  Just set ‘SRCDIR’ as the address of Gnuastro's top source
directory (for example, where there is the unpacked tarball).

     $ SRCDIR=/home/username/gnuastro
     $ mkdir /dev/shm/tmp-gnuastro-build
     $ cd /dev/shm/tmp-gnuastro-build
     $ $SRCDIR/configure --srcdir=$SRCDIR
     $ make

   Gnuastro comes with a script to simplify this process of configuring
and building in a different directory (a "clean" build), for more see
*note Separate build and source directories::.


File: gnuastro.info,  Node: Separate build and source directories,  Next: Tests,  Prev: Configuring,  Up: Build and install

3.3.2 Separate build and source directories
-------------------------------------------

The simple steps of *note Quick start:: will mix the source and built
files.  This can cause inconvenience for developers or enthusiasts
following the most recent work (see *note Version controlled source::).
The current section is mainly focused on this later group of Gnuastro
users.  If you just install Gnuastro on major releases (following *note
Announcements::), you can safely ignore this section.

   When it is necessary to keep the source (which is under version
control), but not the derivative (built) files (after checking or
installing), the best solution is to keep the source and the built files
in separate directories.  One application of this is already discussed
in *note Configure and build in RAM::.

   To facilitate this process of configuring and building in a separate
directory, Gnuastro comes with the ‘developer-build’ script.  It is
available in the top source directory and is _not_ installed.  It will
make a directory under a given top-level directory (given to
‘--top-build-dir’) and build Gnuastro there.  It thus keeps the source
completely separated from the built files.  For easy access to the built
files, it also makes a symbolic link to the built directory in the top
source files called ‘build’.

   When running the developer-build script without any options in the
Gnuastro's top source directory, default values will be used for its
configuration.  As with Gnuastro's programs, you can inspect the default
values with ‘-P’ (or ‘--printparams’, the output just looks a little
different here).  The default top-level build directory is ‘/dev/shm’:
the shared memory directory in RAM on GNU/Linux systems as described in
*note Configure and build in RAM::.

   Besides these, it also has some features to facilitate the job of
developers or bleeding edge users like the ‘--debug’ option to do a fast
build, with debug information, no optimization, and no shared libraries.
Here is the full list of options you can feed to this script to
configure its operations.

*Not all Gnuastro's common program behavior usable here:*
‘developer-build’ is just a non-installed script with a very limited
scope as described above.  It thus does not have all the common option
behaviors or configuration files for example.

*White space between option and value:* ‘developer-build’ does not
accept an <=> sign between the options and their values.  It also needs
at least one character between the option and its value.  Therefore ‘-n
4’ or ‘--numthreads 4’ are acceptable, while ‘-n4’, ‘-n=4’, or
‘--numthreads=4’ are not.  Finally multiple short option names cannot be
merged: for example, you can say ‘-c -n 4’, but unlike Gnuastro's
programs, ‘-cn4’ is not acceptable.

*Reusable for other packages:* This script can be used in any software
which is configured and built using the GNU Build System.  Just copy it
in the top source directory of that software and run it from there.

*Example usage:* See *note Forking tutorial:: for an example usage of
this script in some scenarios.

‘-b STR’
‘--top-build-dir STR’
     The top build directory to make a directory for the build.  If this
     option is not called, the top build directory is ‘/dev/shm’ (only
     available in GNU/Linux operating systems, see *note Configure and
     build in RAM::).

‘-V’
‘--version’
     Print the version string of Gnuastro that will be used in the
     build.  This string will be appended to the directory name
     containing the built files.

‘-a’
‘--autoreconf’
     Run ‘autoreconf -f’ before building the package.  In Gnuastro, this
     is necessary when a new commit has been made to the project
     history.  In Gnuastro's build system, the Git description will be
     used as the version, see *note Version numbering:: and *note
     Synchronizing::.

‘-c’
‘--clean’
     Delete the contents of the build directory (clean it) before
     starting the configuration and building of this run.

     This is useful when you have recently pulled changes from the main
     Git repository, or committed a change yourself and ran ‘autoreconf
     -f’, see *note Synchronizing::.  After running GNU Autoconf, the
     version will be updated and you need to do a clean build.

‘-d’
‘--debug’
     Build with debugging flags (for example, to use in GNU Debugger,
     also known as GDB, or Valgrind), disable optimization and also the
     building of shared libraries.  Similar to running the configure
     script of below

          $ ./configure --enable-debug

     Besides all the debugging advantages of building with this option,
     it will also be significantly speed up the build (at the cost of
     slower built programs).  So when you are testing something small or
     working on the build system itself, it will be much faster to test
     your work with this option.

‘-v’
‘--valgrind’
     Build all ‘make check’ tests within Valgrind.  For more, see the
     description of ‘--enable-check-with-valgrind’ in *note Gnuastro
     configure options::.

‘-j INT’
‘--jobs INT’
     The maximum number of threads/jobs for Make to build at any moment.
     As the name suggests (Make has an identical option), the number
     given to this option is directly passed on to any call of Make with
     its ‘-j’ option.

‘-C’
‘--check’
     After finishing the build, also run ‘make check’.  By default,
     ‘make check’ is not run because the developer usually has their own
     checks to work on (for example, defined in ‘tests/during-dev.sh’).

‘-i’
‘--install’
     After finishing the build, also run ‘make install’.

‘-D’
‘--dist’
     Run ‘make dist-lzip pdf’ to build a distribution tarball (in
     ‘.tar.lz’ format) and a PDF manual.  This can be useful for
     archiving, or sending to colleagues who do not use Git for an easy
     build and manual.

‘-u STR’
‘--upload STR’
     Activate the ‘--dist’ (‘-D’) option, then use secure copy (‘scp’,
     part of the SSH tools) to copy the tarball and PDF to the ‘src’ and
     ‘pdf’ sub-directories of the specified server and its directory
     (value to this option).  For example, ‘--upload my-server:dir’,
     will copy the tarball in the ‘dir/src’, and the PDF manual in
     ‘dir/pdf’ of ‘my-server’ server.  It will then make a symbolic link
     in the top server directory to the tarball that is called
     ‘gnuastro-latest.tar.lz’.

‘-p STR’
‘--publish=STR’
     Clean, bootstrap, build, check and upload the checked tarball and
     PDF of the book to the URL given as ‘STR’.  This option is just a
     wrapper for ‘--autoreconf --clean --debug --check --upload STR’.
     ‘--debug’ is added because it will greatly speed up the build.
     ‘--debug’ will have no effect on the produced tarball (people who
     later download will be building with the default optimized, and
     non-debug mode).  This option is good when you have made a commit
     and are ready to publish it on your server (if nothing crashes).
     Recall that if any of the previous steps fail the script aborts.

‘-I’
‘--install-archive’
     Short for ‘--autoreconf --clean --check --install --dist’.  This is
     useful when you actually want to install the commit you just made
     (if the build and checks succeed).  It will also produce a
     distribution tarball and PDF manual for easy access to the
     installed tarball on your system at a later time.

     Ideally, Gnuastro's Git version history makes it easy for a
     prepared system to revert back to a different point in history.
     But Gnuastro also needs to bootstrap files and also your
     collaborators might (usually do!)  find it too much of a burden to
     do the bootstrapping themselves.  So it is convenient to have a
     tarball and PDF manual of the version you have installed (and are
     using in your research) handily available.

‘-h’
‘--help’
‘-P’
‘--printparams’
     Print a description of this script along with all the options and
     their current values.


File: gnuastro.info,  Node: Tests,  Next: A4 print book,  Prev: Separate build and source directories,  Up: Build and install

3.3.3 Tests
-----------

After successfully building (compiling) the programs with the ‘$ make’
command you can check the installation before installing.  To run the
tests, run

     $ make check

   For every program some tests are designed to check some possible
operations.  Running the command above will run those tests and give you
a final report.  If everything is OK and you have built all the
programs, all the tests should pass.  In case any of the tests fail,
please have a look at *note Known issues:: and if that still does not
fix your problem, look that the ‘./tests/test-suite.log’ file to see if
the source of the error is something particular to your system or more
general.  If you feel it is general, please contact us because it might
be a bug.  Note that the tests of some programs depend on the outputs of
other program's tests, so if you have not installed them they might be
skipped or fail.  Prior to releasing every distribution all these tests
are checked.  If you have a reasonably modern terminal, the outputs of
the successful tests will be colored green and the failed ones will be
colored red.

   These scripts can also act as a good set of examples for you to see
how the programs are run.  All the tests are in the ‘tests/’ directory.
The tests for each program are shell scripts (ending with ‘.sh’) in a
sub-directory of this directory with the same name as the program.  See
*note Test scripts:: for more detailed information about these scripts
in case you want to inspect them.


File: gnuastro.info,  Node: A4 print book,  Next: Known issues,  Prev: Tests,  Up: Build and install

3.3.4 A4 print book
-------------------

The default print version of this book is provided in the letter paper
size.  If you would like to have the print version of this book on paper
and you are living in a country which uses A4, then you can rebuild the
book.  The great thing about the GNU build system is that the book
source code which is in Texinfo is also distributed with the program
source code, enabling you to do such customization (hacking).

   In order to change the paper size, you will need to have GNU Texinfo
installed.  Open ‘doc/gnuastro.texi’ with any text editor.  This is the
source file that created this book.  In the first few lines you will see
this line:

     @c@afourpaper

In Texinfo, a line is commented with ‘@c’.  Therefore, un-comment this
line by deleting the first two characters such that it changes to:

     @afourpaper

Save the file and close it.  You can now run the following command

     $ make pdf

and the new PDF book will be available in ‘SRCdir/doc/gnuastro.pdf’.  By
changing the ‘pdf’ in ‘$ make pdf’ to ‘ps’ or ‘dvi’ you can have the
book in those formats.  Note that you can do this for any book that is
in Texinfo format, they might not have ‘@afourpaper’ line, so you can
add it close to the top of the Texinfo source file.


File: gnuastro.info,  Node: Known issues,  Prev: A4 print book,  Up: Build and install

3.3.5 Known issues
------------------

Depending on your operating system and the version of the compiler you
are using, you might confront some known problems during the
configuration (‘$ ./configure’), compilation (‘$ make’) and tests (‘$
make check’).  Here, their solutions are discussed.

   • ‘$ ./configure’: _Configure complains about not finding a library
     even though you have installed it._  The possible solution is based
     on how you installed the package:

        • From your distribution's package manager.  Most probably this
          is because your distribution has separated the header files of
          a library from the library parts.  Please also install the
          'development' packages for those libraries too.  Just add a
          ‘-dev’ or ‘-devel’ to the end of the package name and re-run
          the package manager.  This will not happen if you install the
          libraries from source.  When installed from source, the
          headers are also installed.

        • From source.  Then your linker is not looking where you
          installed the library.  If you followed the instructions in
          this chapter, all the libraries will be installed in
          ‘/usr/local/lib’.  So you have to tell your linker to look in
          this directory.  To do so, configure Gnuastro like this:

               $ ./configure LDFLAGS="-L/usr/local/lib"

          If you want to use the libraries for your other programming
          projects, then export this environment variable in a start-up
          script similar to the case for ‘LD_LIBRARY_PATH’ explained
          below, also see *note Installation directory::.

   • ‘$ make’: _Complains about an unknown function on a non-GNU based
     operating system._  In this case, please run ‘$ ./configure’ with
     the ‘--enable-gnulibcheck’ option to see if the problem is from the
     GNU Portability Library (Gnulib) not supporting your system or if
     there is a problem in Gnuastro, see *note Gnuastro configure
     options::.  If the problem is not in Gnulib and after all its tests
     you get the same complaint from ‘make’, then please contact us at
     ‘bug-gnuastro@gnu.org’.  The cause is probably that a function that
     we have used is not supported by your operating system and we did
     not included it along with the source tarball.  If the function is
     available in Gnulib, it can be fixed immediately.

   • ‘$ make’: _Cannot find the headers (.h files) of installed
     libraries._  Your C preprocessor (CPP) is not looking in the right
     place.  To fix this, configure Gnuastro with an additional
     ‘CPPFLAGS’ like below (assuming the library is installed in
     ‘/usr/local/include’:

          $ ./configure CPPFLAGS="-I/usr/local/include"

     If you want to use the libraries for your other programming
     projects, then export this environment variable in a start-up
     script similar to the case for ‘LD_LIBRARY_PATH’ explained below,
     also see *note Installation directory::.

   • ‘$ make check’: _Only the first couple of tests pass, all the rest
     fail or get skipped._  It is highly likely that when searching for
     shared libraries, your system does not look into the
     ‘/usr/local/lib’ directory (or wherever you installed Gnuastro or
     its dependencies).  To make sure it is added to the list of
     directories, add the following line to your ‘~/.bashrc’ file and
     restart your terminal.  Do not forget to change ‘/usr/local/lib’ if
     the libraries are installed in other (non-standard) directories.

          export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"

     You can also add more directories by using a colon '‘:’' to
     separate them.  See *note Installation directory:: and *note
     Linking:: to learn more on the ‘PATH’ variables and dynamic linking
     respectively.

   • ‘$ make check’: _The tests relying on external programs (for
     example, ‘fitstopdf.sh’ fail.)_  This is probably due to the fact
     that the version number of the external programs is too old for the
     tests we have preformed.  Please update the program to a more
     recent version.  For example, to create a PDF image, you will need
     GPL Ghostscript, but older versions do not work, we have
     successfully tested it on version 9.15.  Older versions might cause
     a failure in the test result.

   • ‘$ make pdf’: _The PDF book cannot be made._  To make a PDF book,
     you need to have the GNU Texinfo program (like any program, the
     more recent the better).  A working TeX program is also necessary,
     which you can get from Tex Live(1).

   • After ‘make check’: do not copy the programs' executables to
     another (for example, the installation) directory manually (using
     ‘cp’, or ‘mv’ for example).  In the default configuration(2), the
     program binaries need to link with Gnuastro's shared library which
     is also built and installed with the programs.  Therefore, to run
     successfully before and after installation, linking modifications
     need to be made by GNU Libtool at installation time.  ‘make
     install’ does this internally, but a simple copy might give linking
     errors when you run it.  If you need to copy the executables, you
     can do so after installation.

   • ‘$ make’ (when bootstrapping): After you have bootstrapped Gnuastro
     from the version-controlled source, you may confront the following
     (or a similar) error when converting images (for more on
     bootstrapping, see *note Bootstrapping::):

          convert: attempt to perform an operation not allowed by the
          security policy `gs'  error/delegate.c/ExternalDelegateCommand/378.

     This error is a known issue(3) with ‘ImageMagick’ security policies
     in some operating systems.  In short, ‘imagemagick’ uses
     Ghostscript for PDF, EPS, PS and XPS parsing.  However, because
     some security vulnerabilities have been found in Ghostscript(4), by
     default, ImageMagick may be compiled without Ghostscript library.
     In such cases, if allowed, ImageMagick will fall back to the
     external ‘gs’ command instead of the library.  But this may be
     disabled with the following (or a similar) lines in
     ‘/etc/ImageMagick-7/policy.xml’ (anything related to PDF, PS, or
     Ghostscript).

          <policy domain="delegate" rights="none" pattern="gs" />
          <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" />

     To fix this problem, simply comment such lines (by placing a ‘<!--’
     before each statement/line and ‘-->’ at the end of that
     statement/line).

   • ‘$ make dist’: _Complains about "Numeric user ID too large" and
     aborts._  This is a problem related to the way that GNU Automake
     calls GNU Tar on some operating systems.  For more information see
     bug 65578(5).  Until this bug is fixed, you can build the tarball
     by following the steps below:

          ## Generate the build directory; ready to be packaged.
          $ ./developer-build -a -c -d -C

          ## Go to the build directory:
          $ cd build

     Open and edit the ‘Makefile’ to add ‘--format=posix’ to the
     definition of the ‘am__tar’.  So, the line

          am__tar = $${TAR-tar} chof - "$$tardir"

     becomes

          am__tar = $${TAR-tar} --format=posix chof - "$$tardir"

     Finally, create the tarball:
          $ make dist

If your problem was not listed above, please file a bug report (*note
Report a bug::).

   ---------- Footnotes ----------

   (1) <https://www.tug.org/texlive/>

   (2) If you configure Gnuastro with the ‘--disable-shared’ option,
then the libraries will be statically linked to the programs and this
problem will not exist, see *note Linking::.

   (3) <https://wiki.archlinux.org/title/ImageMagick>

   (4) <https://security.archlinux.org/package/ghostscript>

   (5) <https://savannah.gnu.org/bugs/index.php?65578>


File: gnuastro.info,  Node: Common program behavior,  Next: Data containers,  Prev: Installation,  Up: Top

4 Common program behavior
*************************

All the programs in Gnuastro share a set of common behavior mainly to do
with user interaction to facilitate their usage and development.  This
includes how to feed input datasets into the programs, how to configure
them, specifying the outputs, numerical data types, treating columns of
information in tables, etc.  This chapter is devoted to describing this
common behavior in all programs.  Because the behaviors discussed here
are common to several programs, they are not repeated in each program's
description.

   In *note Command-line::, a very general description of running the
programs on the command-line is discussed, like difference between
arguments and options, as well as options that are common/shared between
all programs.  None of Gnuastro's programs keep any internal
configuration value (values for their different operational steps), they
read their configuration primarily from the command-line, then from
specific files in directory, user, or system-wide settings.  Using these
configuration files can greatly help reproducible and robust usage of
Gnuastro, see *note Configuration files:: for more.

   It is not possible to always have the different options and
configurations of each program on the top of your head.  It is very
natural to forget the options of a program, their current default
values, or how it should be run and what it did.  Gnuastro's programs
have multiple ways to help you refresh your memory in multiple levels
(just an option name, a short description, or fast access to the
relevant section of the manual.  See *note Getting help:: for more for
more on benefiting from this very convenient feature.

   Many of the programs use the multi-threaded character of modern CPUs,
in *note Multi-threaded operations:: we will discuss how you can
configure this behavior, along with some tips on making best use of
them.  In *note Numeric data types::, we will review the various types
to store numbers in your datasets: setting the proper type for the usage
context(1) can greatly improve the file size and also speed of reading,
writing or processing them.

   We will then look into the recognized table formats in *note Tables::
and how large datasets are broken into tiles, or mesh grid in *note
Tessellation::.  Finally, we will take a look at the behavior regarding
output files: *note Automatic output:: describes how the programs set a
default name for their output when you do not give one explicitly (using
‘--output’).  When the output is a FITS file, all the programs also
store some very useful information in the header that is discussed in
*note Output FITS files::.

* Menu:

* Command-line::                How to use the command-line.
* Configuration files::         Values for unspecified variables.
* Getting help::                Getting more information on the go.
* Multi-threaded operations::   How threads are managed in Gnuastro.
* Numeric data types::          Different types and how to specify them.
* Memory management::           How memory is allocated (in RAM or HDD/SSD).
* Tables::                      Recognized table formats.
* Tessellation::                Tile the dataset into non-overlapping bins.
* Automatic output::            About automatic output names.
* Output FITS files::           Common properties when outputs are FITS.
* Numeric locale::              Decimal point printed like 0.5 instead of 0,5.

   ---------- Footnotes ----------

   (1) For example, if the values in your dataset can only be integers
between 0 or 65000, store them in a unsigned 16-bit type, not 64-bit
floating point type (which is the default in most systems).  It takes
four times less space and is much faster to process.


File: gnuastro.info,  Node: Command-line,  Next: Configuration files,  Prev: Common program behavior,  Up: Common program behavior

4.1 Command-line
================

Gnuastro's programs are customized through the standard Unix-like
command-line environment and GNU style command-line options.  Both are
very common in many Unix-like operating system programs.  In *note
Arguments and options:: we will start with the difference between
arguments and options and elaborate on the GNU style of options.
Afterwards, in *note Common options::, we will go into the detailed list
of all the options that are common to all the programs in Gnuastro.

* Menu:

* Arguments and options::       Different ways to specify inputs and configuration.
* Common options::              Options that are shared between all programs.
* Shell TAB completion::        Customized TAB completion in Gnuastro.
* Standard input::              Using output of another program as input.
* Shell tips::                  Useful tips and tricks for program usage.


File: gnuastro.info,  Node: Arguments and options,  Next: Common options,  Prev: Command-line,  Up: Command-line

4.1.1 Arguments and options
---------------------------

When you type a command on the command-line, it is passed onto the shell
(a generic name for the program that manages the command-line) as a
string of characters.  As an example, see the "Invoking ProgramName"
sections in this manual for some examples of commands with each program,
like *note Invoking asttable::, *note Invoking astfits::, or *note
Invoking aststatistics::.

   The shell then brakes up your string into separate _tokens_ or
_words_ using any _metacharacters_ (like white-space, tab, ‘|’, ‘>’ or
‘;’) that are in the string.  On the command-line, the first thing you
usually enter is the name of the program you want to run.  After that,
you can specify two types of tokens: _arguments_ and _options_.  In the
GNU-style, arguments are those tokens that are not preceded by any
hyphens (‘-’, see *note Arguments::).  Here is one example:

     $ astcrop --center=53.162551,-27.789676 -w10/3600 --mode=wcs udf.fits

   In the example above, we are running *note Crop:: to crop a region of
width 10 arc-seconds centered at the given RA and Dec from the input
Hubble Ultra-Deep Field (UDF) FITS image.  Here, the argument is
‘udf.fits’.  Arguments are most commonly the input file names containing
your data.  Options start with one or two hyphens, followed by an
identifier for the option (the option's name, for example, ‘--center’,
‘-w’, ‘--mode’ in the example above) and its value (anything after the
option name, or the optional <=> character).  Through options you can
configure how the program runs (interprets the data you provided).

   Arguments can be mandatory and optional and unlike options, they do
not have any identifiers.  Hence, when there multiple arguments, their
order might also matter (for example, in ‘cp’ which is used for copying
one file to another location).  The outputs of ‘--usage’ and ‘--help’
shows which arguments are optional and which are mandatory, see *note
--usage::.

   As their name suggests, _options_ can be considered to be optional
and most of the time, you do not have to worry about what order you
specify them in.  When the order does matter, or the option can be
invoked multiple times, it is explicitly mentioned in the "Invoking
ProgramName" section of each program (this is a very important aspect of
an option).

   If there is only one such character, you can use a backslash (‘\’)
before it.  If there are multiple, it might be easier to simply put your
whole argument or option value inside of double quotes (‘"’).  In such
cases, everything inside the double quotes will be seen as one token or
word.

   For example, let's say you want to specify the header data unit (HDU)
of your FITS file using a complex expression like '‘3; images(exposure >
100)’'.  If you simply add these after the ‘--hdu’ (‘-h’) option, the
programs in Gnuastro will read the value to the HDU option as '‘3’' and
run.  Then, the shell will attempt to run a separate command
'‘images(exposure > 100)’' and complain about a syntax error.  This is
because the semicolon (‘;’) is an 'end of command' character in the
shell.  To solve this problem you can simply put double quotes around
the whole string you want to pass to ‘--hdu’ as seen below:
     $ astcrop --hdu="3; images(exposure > 100)" image.fits

* Menu:

* Arguments::                   For specifying the main input files/operations.
* Options::                     For configuring the behavior of the program.


File: gnuastro.info,  Node: Arguments,  Next: Options,  Prev: Arguments and options,  Up: Arguments and options

4.1.1.1 Arguments
.................

In Gnuastro, arguments are almost exclusively used as the input data
file names.  Please consult the first few paragraph of the "Invoking
ProgramName" section for each program for a description of what it
expects as input, how many arguments, or input data, it accepts, or in
what order.  Everything particular about how a program treats arguments,
is explained under the "Invoking ProgramName" section for that program.

   Generally, if there is a standard file name suffix for a particular
format, that filename extension is checked to identify their format.  In
astronomy (and thus Gnuastro), FITS is the preferred format for inputs
and outputs, so the focus here and throughout this book is on FITS.
However, other formats are also accepted in special cases, for example,
*note ConvertType:: also accepts JPEG or TIFF inputs, and writes JPEG,
EPS or PDF files.  The recognized suffixes for these formats are listed
there.

   The list below shows the recognized suffixes for FITS data files in
Gnuastro's programs.  However, in some scenarios FITS writers may not
append a suffix to the file, or use a non-recognized suffix (not in the
list below).  Therefore if a FITS file is expected, but it does not have
any of these suffixes, Gnuastro programs will look into the contents of
the file and if it does conform with the FITS standard, the file will be
used.  Just note that checking about 5 characters at the end of a name
string is much more efficient than opening and checking the contents of
a file, so it is generally recommended to have a recognized FITS suffix.

   • ‘.fits’: The standard file name ending of a FITS image.

   • ‘.fit’: Alternative (3 character) FITS suffix.

   • ‘.fits.Z’: A FITS image compressed with ‘compress’.

   • ‘.fits.gz’: A FITS image compressed with GNU zip (gzip).

   • ‘.fits.fz’: A FITS image compressed with ‘fpack’.

   • ‘.imh’: IRAF format image file.

   Throughout this book and in the command-line outputs, whenever we
want to generalize all such astronomical data formats in a text
place-holder, we will use ‘ASTRdata’ and assume that the extension is
also part of this name.  Any file ending with these names is directly
passed on to CFITSIO to read.  Therefore you do not necessarily have to
have these files on your computer, they can also be located on an FTP or
HTTP server too, see the CFITSIO manual for more information.

   CFITSIO has its own error reporting techniques, if your input file(s)
cannot be opened, or read, those errors will be printed prior to the
final error by Gnuastro.


File: gnuastro.info,  Node: Options,  Prev: Arguments,  Up: Arguments and options

4.1.1.2 Options
...............

Command-line options allow configuring the behavior of a program in all
GNU/Linux applications for each particular execution on a particular
input data.  A single option can be called in two ways: _long_ or
_short_.  All options in Gnuastro accept the long format which has two
hyphens an can have many characters (for example, ‘--hdu’).  Short
options only have one hyphen (<->) followed by one character (for
example, ‘-h’).  You can see some examples in the list of options in
*note Common options:: or those for each program's "Invoking
ProgramName" section.  Both formats are shown for those which support
both.  First the short is shown then the long.

   Usually, the short options are handy when you are writing on the
command-line and want to save keystrokes and time.  The long options are
good for shell scripts, where you are not usually rushing.  Long options
provide a level of documentation, since they are more descriptive and
less cryptic.  Usually after a few months of not running a program, the
short options will be forgotten and reading your previously written
script will not be easy.

   Some options need to be given a value if they are called and some do
not.  You can think of the latter type of options as on/off options.
These two types of options can be distinguished using the output of the
‘--help’ and ‘--usage’ options, which are common to all GNU software,
see *note Getting help::.  In Gnuastro we use the following strings to
specify when the option needs a value and what format that value should
be in.  More specific tests will be done in the program and if the
values are out of range (for example, negative when the program only
wants a positive value), an error will be reported.

‘INT’
     The value is read as an integer.

‘FLT’
     The value is read as a float.  There are generally two types,
     depending on the context.  If they are for fractions, they will
     have to be less than or equal to unity.

‘STR’
     The value is read as a string of characters.  For example, column
     names in a table, or HDU names in a multi-extension FITS file.
     Other examples include human-readable settings by some programs
     like the ‘--domain’ option of the Convolve program that can be
     either ‘spatial’ or ‘frequency’ (to specify the type of
     convolution, see *note Convolve::).

‘FITS or FITS/TXT’
     The value should be a file (most commonly FITS). In many cases,
     other formats may also be accepted (for example, input tables can
     be FITS or plain-text, see *note Recognized table formats::).

To specify a value in the short format, simply put the value after the
option.  Note that since the short options are only one character long,
you do not have to type anything between the option and its value.  For
the long option you either need white space or an ‘=’ sign, for example,
‘-h2’, ‘-h 2’, ‘--hdu 2’ or ‘--hdu=2’ are all equivalent.

   The short format of on/off options (those that do not need values)
can be concatenated for example, these two hypothetical sequences of
options are equivalent: ‘-a -b -c4’ and ‘-abc4’.  As an example,
consider the following command to run Crop:
     $ astcrop -Dr3 --wwidth 3 catalog.txt --deccol=4 ASTRdata
The ‘$’ is the shell prompt, ‘astcrop’ is the program name.  There are
two arguments (‘catalog.txt’ and ‘ASTRdata’) and four options, two of
them given in short format (‘-D’, ‘-r’) and two in long format
(‘--width’ and ‘--deccol’).  Three of them require a value and one
(‘-D’) is an on/off option.

   If an abbreviation is unique between all the options of a program,
the long option names can be abbreviated.  For example, instead of
typing ‘--printparams’, typing ‘--print’ or maybe even ‘--pri’ will be
enough, if there are conflicts, the program will warn you and show you
the alternatives.  Finally, if you want the argument parser to stop
parsing arguments beyond a certain point, you can use two dashes: ‘--’.
No text on the command-line beyond these two dashes will be parsed.

   Gnuastro has two types of options with values, those that only take a
single value are the most common type.  If these options are repeated or
called more than once on the command-line, the value of the last time it
was called will be assigned to it.  This is very useful when you are
testing/experimenting.  Let's say you want to make a small modification
to one option value.  You can simply type the option with a new value in
the end of the command and see how the script works.  If you are
satisfied with the change, you can remove the original option for human
readability.  If the change was not satisfactory, you can remove the one
you just added and not worry about forgetting the original value.
Without this capability, you would have to memorize or save the original
value somewhere else, run the command and then change the value again
which is not at all convenient and is potentially cause lots of bugs.

   On the other hand, some options can be called multiple times in one
run of a program and can thus take multiple values (for example, see the
‘--column’ option in *note Invoking asttable::.  In these cases, the
order of stored values is the same order that you specified on the
command-line.

   Gnuastro's programs do not keep any internal default values, so some
options are mandatory and if they do not have a value, the program will
complain and abort.  Most programs have many such options and typing
them by hand on every call is impractical.  To facilitate the user
experience, after parsing the command-line, Gnuastro's programs read
special configuration files to get the necessary values for the options
you have not identified on the command-line.  These configuration files
are fully described in *note Configuration files::.

*CAUTION:* In specifying a file address, if you want to use the shell's
tilde expansion (‘~’) to specify your home directory, leave at least one
space between the option name and your value.  For example, use ‘-o
~/test’, ‘--output ~/test’ or ‘--output= ~/test’.  Calling them with
‘-o~/test’ or ‘--output=~/test’ will disable shell expansion.
*CAUTION:* If you forget to specify a value for an option which requires
one, and that option is the last one, Gnuastro will warn you.  But if it
is in the middle of the command, it will take the text of the next
option or argument as the value which can cause undefined behavior.
*NOTE:* In some contexts Gnuastro's counting starts from 0 and in others
1.  You can assume by default that counting starts from 1, if it starts
from 0 for a special option, it will be explicitly mentioned.


File: gnuastro.info,  Node: Common options,  Next: Shell TAB completion,  Prev: Arguments and options,  Up: Command-line

4.1.2 Common options
--------------------

To facilitate the job of the users and developers, all the programs in
Gnuastro share some basic command-line options for the options that are
common to many of the programs.  The full list is classified as *note
Input output options::, *note Processing options::, and *note Operating
mode options::.  In some programs, some of the options are irrelevant,
but still recognized (you will not get an unrecognized option error, but
the value is not used).  Unless otherwise mentioned, these options are
identical between all programs.

* Menu:

* Input output options::        Common input/output options.
* Processing options::          Options for common processing steps.
* Operating mode options::      Common operating mode options.


File: gnuastro.info,  Node: Input output options,  Next: Processing options,  Prev: Common options,  Up: Common options

4.1.2.1 Input/Output options
............................

These options are to do with the input and outputs of the various
programs.

‘--stdintimeout’
     Number of micro-seconds to wait for writing/typing in the _first
     line_ of standard input from the command-line (see *note Standard
     input::).  This is only relevant for programs that also accept
     input from the standard input, _and_ you want to manually
     write/type the contents on the terminal.  When the standard input
     is already connected to a pipe (output of another program), there
     will not be any waiting (hence no timeout, thus making this option
     redundant).

     If the first line-break (for example, with the <ENTER> key) is not
     provided before the timeout, the program will abort with an error
     that no input was given.  Note that this time interval is _only_
     for the first line that you type.  Once the first line is given,
     the program will assume that more data will come and accept rest of
     your inputs without any time limit.  You need to specify the ending
     of the standard input, for example, by pressing <CTRL-D> after a
     new line.

     Note that any input you write/type into a program on the
     command-line with Standard input will be discarded (lost) once the
     program is finished.  It is only recoverable manually from your
     command-line (where you actually typed) as long as the terminal is
     open.  So only use this feature when you are sure that you do not
     need the dataset (or have a copy of it somewhere else).

‘-h STR/INT’
‘--hdu=STR/INT’
     The name or number of the desired Header Data Unit, or HDU, in the
     FITS image.  A FITS file can store multiple HDUs or extensions,
     each with either an image or a table or nothing at all (only a
     header).  Note that counting of the extensions starts from 0(zero),
     not 1(one).  Counting from 0 is forced on us by CFITSIO which
     directly reads the value you give with this option (see *note
     CFITSIO::).  When specifying the name, case is not important so
     ‘IMAGE’, ‘image’ or ‘ImAgE’ are equivalent.

     CFITSIO has many capabilities to help you find the extension you
     want, far beyond the simple extension number and name.  See CFITSIO
     manual's "HDU Location Specification" section for a very complete
     explanation with several examples.  A ‘#’ is appended to the string
     you specify for the HDU(1) and the result is put in square brackets
     and appended to the FITS file name before calling CFITSIO to read
     the contents of the HDU for all the programs in Gnuastro.

     *Default HDU is HDU number 1 (counting from 0):* by default,
     Gnuastro’s programs assume that their (main/first) input is in HDU
     number 1 (counting from zero).  So if you don’t specify the HDU
     number, the program will read the input from this HDU. For programs
     that can take multiple FITS datasets as input (like *note
     Arithmetic::) this default HDU applies to the first input, you
     still need to call ‘--hdu’ for the other inputs.  Generally, all
     Gnuastro's programs write their outputs in HDU number 1 (HDU 0 is
     reserved for metadata like the configuration parameters that the
     program was run with).  For more on this, see *note Fits::.

‘-s STR’
‘--searchin=STR’
     Where to match/search for columns when the column identifier was
     not a number, see *note Selecting table columns::.  The acceptable
     values are ‘name’, ‘unit’, or ‘comment’.  This option is only
     relevant for programs that take table columns as input.

‘-I’
‘--ignorecase’
     Ignore case while matching/searching column meta-data (in the field
     specified by the ‘--searchin’).  The FITS standard suggests to
     treat the column names as case insensitive, which is strongly
     recommended here also but is not enforced.  This option is only
     relevant for programs that take table columns as input.

     This option is not relevant to *note BuildProgram::, hence in that
     program the short option ‘-I’ is used for include directories, not
     to ignore case.

‘-o STR’
‘--output=STR’
     The name of the output file or directory.  With this option the
     automatic output names explained in *note Automatic output:: are
     ignored.

‘-T STR’
‘--type=STR’
     The data type of the output depending on the program context.  This
     option is not applicable to some programs like *note Fits:: and
     will be ignored by them.  The different acceptable values to this
     option are fully described in *note Numeric data types::.

‘-D’
‘--dontdelete’
     By default, if the output file already exists, Gnuastro's programs
     will silently delete it and put their own outputs in its place.
     When this option is activated, if the output file already exists,
     the programs will not delete it, will warn you, and will abort.

‘-K’
‘--keepinputdir’
     In automatic output names, do not remove the directory information
     of the input file names.  As explained in *note Automatic output::,
     if no output name is specified (with ‘--output’), then the output
     name will be made in the existing directory based on your input's
     file name (ignoring the directory of the input).  If you call this
     option, the directory information of the input will be kept and the
     automatically generated output name will be in the same directory
     as the input (usually with a suffix added).  Note that his is only
     relevant if you are running the program in a different directory
     than the input data.

‘-t STR’
‘--tableformat=STR’
     The output table's type.  This option is only relevant when the
     output is a table and its format cannot be deduced from its
     filename.  For example, if a name ending in ‘.fits’ was given to
     ‘--output’, then the program knows you want a FITS table.  But
     there are two types of FITS tables: FITS ASCII, and FITS binary.
     Thus, with this option, the program is able to identify which type
     you want.  The currently recognized values to this option are:

‘--wcslinearmatrix=STR’
     Select the linear transformation matrix of the output's WCS. This
     option only takes two values: ‘pc’ (for the ‘PCi_j’ formalism) and
     ‘cd’ (for ‘CDi_j’).  For more on the different formalisms, please
     see Section 8.1 of the FITS standard(2), version 4.0.

     In short, in the ‘PCi_j’ formalism, we only keep the linear
     rotation matrix in these keywords and put the scaling factor (or
     the pixel scale in astronomical imaging) in the ‘CDELTi’ keywords.
     In the ‘CDi_j’ formalism, we blend the scaling into the rotation
     into a single matrix and keep that matrix in these FITS keywords.
     By default, Gnuastro uses the ‘PCi_j’ formalism, because it greatly
     helps in human readability of the raw keywords and is also the
     default mode of WCSLIB. However, in some circumstances it may be
     necessary to have the keywords in the CD format; for example, when
     you need to feed the outputs into other software that do not follow
     the full FITS standard and only recognize the ‘CDi_j’ formalism.

     ‘txt’
          A plain text table with white-space characters between the
          columns (see *note Gnuastro text table format::).
     ‘fits-ascii’
          A FITS ASCII table (see *note Recognized table formats::).
     ‘fits-binary’
          A FITS binary table (see *note Recognized table formats::).

‘--outfitsnoconfig’
     Do not write any of the program's metadata (option values or
     versions and dates) into the 0-th HDU of the output FITS file, see
     *note Output FITS files::.

‘--outfitsnodate’
     Do not write the ‘DATE’ or ‘DATEUTC’ keywords into the 0-th HDU of
     the output FITS file, see *note Output FITS files::.

‘--outfitsnocommit’
     Do not write the ‘COMMIT’ keyword into the 0-th HDU of the output
     FITS file, see *note Output FITS files::.

‘--outfitsnoversions’
     Do not write the versions of any dependency software into the 0-th
     HDU of the output FITS file, see *note Output FITS files::.

   ---------- Footnotes ----------

   (1) With the ‘#’ character, CFITSIO will only read the desired HDU
into your memory, not all the existing HDUs in the fits file.

   (2) <https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf>


File: gnuastro.info,  Node: Processing options,  Next: Operating mode options,  Prev: Input output options,  Up: Common options

4.1.2.2 Processing options
..........................

Some processing steps are common to several programs, so they are
defined as common options to all programs.  Note that this class of
common options is thus necessarily less common between all the programs
than those described in *note Input output options::, or *note Operating
mode options:: options.  Also, if they are irrelevant for a program,
these options will not display in the ‘--help’ output of the program.

‘--minmapsize=INT’
     The minimum size (in bytes) to memory-map a processing/internal
     array as a file (on the non-volatile HDD/SSD), and not use the
     system's RAM. Before using this option, please read *note Memory
     management::.  By default processing arrays will only be
     memory-mapped to a file when the RAM is full.  With this option,
     you can force the memory-mapping, even when there is enough RAM. To
     ensure this default behavior, the pre-defined value to this option
     is an extremely large value (larger than any existing RAM).

     Please note that using a non-volatile file (in the HDD/SDD) instead
     of RAM can significantly increase the program's running time,
     especially on HDDs (where read/write is slower).  Also, note that
     the number of memory-mapped files that your kernel can support is
     limited.  So when this option is necessary, it is best to give it
     values larger than 1 megabyte (‘--minmapsize=1000000’).  You can
     then decrease it for a specific program's invocation on a large
     input after you see memory issues arise (for example, an error, or
     the program not aborting and fully consuming your memory).  If you
     see randomly named files remaining in this directory when the
     program finishes normally, please send us a bug report so we
     address the problem, see *note Report a bug::.

     *Limited number of memory-mapped files:* The operating system
     kernels usually support a limited number of memory-mapped files.
     Therefore never set ‘--minmapsize’ to zero or a small number of
     bytes (so too many files are created).  If the kernel capacity is
     exceeded, the program will crash.

‘--quietmmap’
     Do not print any message when an array is stored in non-volatile
     memory (HDD/SSD) and not RAM, see the description of ‘--minmapsize’
     (above) for more.

‘-Z INT[,INT[,...]]’
‘--tilesize=[,INT[,...]]’
     The size of regular tiles for tessellation, see *note
     Tessellation::.  For each dimension an integer length (in units of
     data-elements or pixels) is necessary.  If the number of input
     dimensions is different from the number of values given to this
     option, the program will stop with an error.  Values must be
     separated by commas (<,>) and can also be fractions (for example,
     ‘4/2’).  If they are fractions, the result must be an integer,
     otherwise an error will be printed.

‘-M INT[,INT[,...]]’
‘--numchannels=INT[,INT[,...]]’
     The number of channels for larger input tessellation, see *note
     Tessellation::.  The number and types of acceptable values are
     similar to ‘--tilesize’.  The only difference is that instead of
     length, the integers values given to this option represent the
     _number_ of channels, not their size.

‘-F FLT’
‘--remainderfrac=FLT’
     The fraction of remainder size along all dimensions to add to the
     first tile.  See *note Tessellation:: for a complete description.
     This option is only relevant if ‘--tilesize’ is not exactly
     divisible by the input dataset's size in a dimension.  If the
     remainder size is larger than this fraction (compared to
     ‘--tilesize’), then the remainder size will be added with one
     regular tile size and divided between two tiles at the start and
     end of the given dimension.

‘--workoverch’
     Ignore the channel borders for the high-level job of the given
     application.  As a result, while the channel borders are respected
     in defining the small tiles (such that no tile will cross a channel
     border), the higher-level program operation will ignore them, see
     *note Tessellation::.

‘--checktiles’
     Make a FITS file with the same dimensions as the input but each
     pixel is replaced with the ID of the tile that it is associated
     with.  Note that the tile IDs start from 0.  See *note
     Tessellation:: for more on Tiling an image in Gnuastro.

‘--oneelempertile’
     When showing the tile values (for example, with ‘--checktiles’, or
     when the program's output is tessellated) only use one element for
     each tile.  This can be useful when only the relative values given
     to each tile compared to the rest are important or need to be
     checked.  Since the tiles usually have a large number of pixels
     within them the output will be much smaller, and so easier to read,
     write, store, or send.

     Note that when the full input size in any dimension is not exactly
     divisible by the given ‘--tilesize’ in that dimension, the edge
     tile(s) will have different sizes (in units of the input's size),
     see ‘--remainderfrac’.  But with this option, all displayed values
     are going to have the (same) size of one data-element.  Hence, in
     such cases, the image proportions are going to be slightly
     different with this option.

     If your input image is not exactly divisible by the tile size and
     you want one value per tile for some higher-level processing, all
     is not lost though.  You can see how many pixels were within each
     tile (for example, to weight the values or discard some for later
     processing) with Gnuastro's Statistics (see *note Statistics::) as
     shown below.  The output FITS file is going to have two extensions,
     one with the median calculated on each tile and one with the number
     of elements that each tile covers.  You can then use the ‘where’
     operator in *note Arithmetic:: to set the values of all tiles that
     do not have the regular area to a blank value.

          $ aststatistics --median --number --ontile input.fits    \
                          --oneelempertile --output=o.fits
          $ REGULAR_AREA=1600    # Check second extension of `o.fits'.
          $ astarithmetic o.fits o.fits $REGULAR_AREA ne nan where \
                          -h1 -h2

     Note that if ‘input.fits’ also has blank values, then the median on
     tiles with blank values will also be ignored with the command above
     (which is desirable).

‘--inteponlyblank’
     When values are to be interpolated, only change the values of the
     blank elements, keep the non-blank elements untouched.

‘--interpmetric=STR’
     The metric to use for finding nearest neighbors.  Currently it only
     accepts the Manhattan (or taxicab) metric with ‘manhattan’, or the
     radial metric with ‘radial’.

     The Manhattan distance between two points is defined with
     $|\Delta{x}|+|\Delta{y}|$.  Thus the Manhattan metric has the
     advantage of being fast, but at the expense of being less accurate.
     The radial distance is the standard definition of distance in a
     Euclidean space: $\sqrt{\Delta{x}^2+\Delta{y}^2}$.  It is accurate,
     but the multiplication and square root can slow down the
     processing.

‘--interpnumngb=INT’
     The number of nearby non-blank neighbors to use for interpolation.


File: gnuastro.info,  Node: Operating mode options,  Prev: Processing options,  Up: Common options

4.1.2.3 Operating mode options
..............................

Another group of options that are common to all the programs in Gnuastro
are those to do with the general operation of the programs.  The
explanation for those that are not only limited to Gnuastro but are
common to all GNU programs start with (GNU option).

‘--’
     (GNU option) Stop parsing the command-line.  This option can be
     useful in scripts or when using the shell history.  Suppose you
     have a long list of options, and want to see if removing some of
     them (to read from configuration files, see *note Configuration
     files::) can give a better result.  If the ones you want to remove
     are the last ones on the command-line, you do not have to delete
     them, you can just add ‘--’ before them and if you do not get what
     you want, you can remove the ‘--’ and get the same initial result.

‘--usage’
     (GNU option) Only print the options and arguments and abort.  This
     is very useful for when you know the what the options do, and have
     just forgot their long/short identifiers, see *note --usage::.

‘-?’
‘--help’
     (GNU option) Print all options with an explanation and abort.
     Adding this option will print all the options in their short and
     long formats, also displaying which ones need a value if they are
     called (with an ‘=’ after the long format followed by a string
     specifying the format, see *note Options::).  A short explanation
     is also given for what the option is for.  The program will quit
     immediately after the message is printed and will not do any form
     of processing, see *note --help::.

‘-V’
‘--version’
     (GNU option) Print a short message, showing the full name, version,
     copyright information and program authors and abort.  On the first
     line, it will print the official name (not executable name) and
     version number of the program.  Following this is a blank line and
     a copyright information.  The program will not run.

‘-q’
‘--quiet’
     Do not report steps.  All the programs in Gnuastro that have
     multiple major steps will report their steps for you to follow
     while they are operating.  If you do not want to see these reports,
     you can call this option and only error/warning messages will be
     printed.  If the steps are done very fast (depending on the
     properties of your input) disabling these reports will also
     decrease running time.

‘--cite’
     Print all necessary information to cite and acknowledge Gnuastro in
     your published papers.  With this option, the programs will print
     the BibTeX entry to include in your paper for Gnuastro in general,
     and the particular program's paper (if that program comes with a
     separate paper).  It will also print the necessary acknowledgment
     statement to add in the respective section of your paper and it
     will abort.  For a more complete explanation, please see *note
     Acknowledgments and short history::.

     Citations and acknowledgments are vital for the continued work on
     Gnuastro.  Gnuastro started, and is continued, based on separate
     research projects.  So if you find any of the tools offered in
     Gnuastro to be useful in your research, please use the output of
     this command to cite and acknowledge the program (and Gnuastro) in
     your research paper.  Thank you.

     Gnuastro is still new, there is no separate paper only devoted to
     Gnuastro yet.  Therefore currently the paper to cite for Gnuastro
     is the paper for NoiseChisel which is the first published paper
     introducing Gnuastro to the astronomical community.  Upon reaching
     a certain point, a paper completely devoted to describing
     Gnuastro's many functionalities will be published, see *note GNU
     Astronomy Utilities 1.0::.

‘-P’
‘--printparams’
     With this option, Gnuastro's programs will read your command-line
     options and all the configuration files.  If there is no problem
     (like a missing parameter or a value in the wrong format or range)
     and immediately before actually running, the programs will print
     the full list of option names, values and descriptions, sorted and
     grouped by context and abort.  They will also report the version
     number, the date they were configured on your system and the time
     they were reported.

     As an example, you can give your full command-line options and even
     the input and output file names and finally just add ‘-P’ to check
     if all the parameters are finely set.  If everything is OK, you can
     just run the same command (easily retrieved from the shell history,
     with the top arrow key) and simply remove the last two characters
     that showed this option.

     No program will actually start its processing when this option is
     called.  The otherwise mandatory arguments for each program (for
     example, input image or catalog files) are no longer required when
     you call this option.

‘--config=STR’
     Parse ‘STR’ as a configuration file name, immediately when this
     option is confronted (see *note Configuration files::).  The
     ‘--config’ option can be called multiple times in one run of any
     Gnuastro program on the command-line or in the configuration files.
     In any case, it will be immediately read (before parsing the rest
     of the options on the command-line, or lines in a configuration
     file).  If the given file does not exist or cannot be read for any
     reason, the program will print a warning and continue its
     processing.  The warning can be suppressed with ‘--quiet’.

     Note that by definition, options on the command-line still take
     precedence over those in any configuration file, including the
     file(s) given to this option if they are called before it.  Also
     see ‘--lastconfig’ and ‘--onlyversion’ on how this option can be
     used for reproducible results.  You can use ‘--checkconfig’ (below)
     to check/confirm the parsing of configuration files.

‘--checkconfig’
     Print options and their values, within the command-line or
     configuration files, as they are parsed (see *note Configuration
     file precedence::).  If an option has already been set, or is
     ignored by the program, this option will also inform you with
     special values like ‘--ALREADY-SET--’.  Only options that are
     parsed after this option are printed, so to see the parsing of all
     input options, it is recommended to put this option immediately
     after the program name before any other options.

     This is a very good option to confirm where the value of each
     option is has been defined in scenarios where there are multiple
     configuration files (for debugging).

‘--config-prefix=STR’
     Accept option names in configuration files that start with the
     given prefix.  Since order matters when reading custom
     configuration files, this option should be called *before* the
     ‘--config’ option(s) that contain options with the given prefix.
     This option does not affect the options within configuration files
     that have the standard name (without a prefix).

     This gives unique features to Gnuastro's configuration files,
     especially in large pipelines.  Let's demonstrate this with the
     simple scenario below.  You have multiple configuration files for
     different instances of one program (let's assume ‘nc-a.conf’ and
     ‘nc-b.conf’).  At the same time, want to load all the option
     names/values into your shell as environment variables (for example
     with ‘source’).  This happens when you want to use the options as
     shell variables in other parts of the your pipeline.

     If the two configuration files have different values for the same
     option (as shown below), and you don't use ‘--config-prefix’, the
     shell will over-write the common option values between the
     configuration files.  But thanks to ‘--config-prefix’, you can give
     a different prefix for the different instances of the same option
     in different configuration files.

          $ cat nc-a.conf
          a_tilesize=20,20

          $ cat nc-b.conf
          b_tilesize=40,40

          ## Load configuration files as shell scripts (to define the
          ## option name and values as shell variables with values).
          ## Just note that 'source' only takes one file at a time.
          $ for c in nc-*.conf; do source $c; done

          $ astnoisechisel img.fits \
                           --config=nc-a.conf --config-prefix=a_
          $ echo "NoiseChisel run with --tilesize=$a_tilesize"

          $ astnoisechisel img.fits \
                           --config=nc-b.conf --config-prefix=b_
          $ echo "NoiseChisel run with --tilesize=$b_tilesize"

‘-S’
‘--setdirconf’
     Update the current directory configuration file for the Gnuastro
     program and quit.  The full set of command-line and configuration
     file options will be parsed and options with a value will be
     written in the current directory configuration file for this
     program (see *note Configuration files::).  If the configuration
     file or its directory does not exist, it will be created.  If a
     configuration file exists it will be replaced (after it, and all
     other configuration files have been read).  In any case, the
     program will not run.

     This is the recommended method(1) to edit/set the configuration
     file for all future calls to Gnuastro's programs.  It will
     internally check if your values are in the correct range and type
     and save them according to the configuration file format, see *note
     Configuration file format::.  So if there are unreasonable values
     to some options, the program will notify you and abort before
     writing the final configuration file.

     When this option is called, the otherwise mandatory arguments, for
     example input image or catalog file(s), are no longer mandatory
     (since the program will not run).

‘-U’
‘--setusrconf’
     Update the user configuration file and quit (see *note
     Configuration files::).  See explanation under ‘--setdirconf’ for
     more details.

‘--lastconfig’
     This is the last configuration file that must be read.  When this
     option is confronted in any stage of reading the options (on the
     command-line or in a configuration file), no other configuration
     file will be parsed, see *note Configuration file precedence:: and
     *note Current directory and User wide::.  Like all on/off options,
     on the command-line, this option does not take any values.  But in
     a configuration file, it takes the values of ‘0’ or ‘1’, see *note
     Configuration file format::.  If it is present in a configuration
     file with a value of ‘0’, then all later occurrences of this option
     will be ignored.

‘--onlyversion=STR’
     Only run the program if Gnuastro's version is exactly equal to
     ‘STR’ (see *note Version numbering::).  Note that it is not
     compared as a number, but as a string of characters, so ‘0’, or
     ‘0.0’ and ‘0.00’ are different.  If the running Gnuastro version is
     different, then this option will report an error and abort as soon
     as it is confronted on the command-line or in a configuration file.
     If the running Gnuastro version is the same as ‘STR’, then the
     program will run as if this option was not called.

     This is useful if you want your results to be exactly reproducible
     and not mistakenly run with an updated/newer or older version of
     the program.  Besides internal algorithmic/behavior changes in
     programs, the existence of options or their names might change
     between versions (especially in these earlier versions of
     Gnuastro).

     Hence, when using this option (probably in a script or in a
     configuration file), be sure to call it before other options.  The
     benefit is that, when the version differs, the other options will
     not be parsed and you, or your collaborators/users, will not get
     errors saying an option in your configuration does not exist in the
     running version of the program.

     Here is one example of how this option can be used in conjunction
     with the ‘--lastconfig’ option.  Let's assume that you were
     satisfied with the results of this command: ‘astnoisechisel
     image.fits --snquant=0.95’ (along with various options set in
     various configuration files).  You can save the state of
     NoiseChisel and reproduce that exact result on ‘image.fits’ later
     by following these steps (the extra spaces, and <\>, are only for
     easy readability, if you want to try it out, only one space between
     each token is enough).

          $ echo "onlyversion X.XX"             > reproducible.conf
          $ echo "lastconfig 1"                >> reproducible.conf
          $ astnoisechisel image.fits --snquant=0.95 -P            \
                                               >> reproducible.conf

     ‘--onlyversion’ was available from Gnuastro 0.0, so putting it
     immediately at the start of a configuration file will ensure that
     later, you (or others using different version) will not get a
     non-recognized option error in case an option was added/removed.
     ‘--lastconfig’ will inform the installed NoiseChisel to not parse
     any other configuration files.  This is done because we do not want
     the user's user-wide or system wide option values affecting our
     results.  Finally, with the third command, which has a ‘-P’ (short
     for ‘--printparams’), NoiseChisel will print all the option values
     visible to it (in all the configuration files) and the shell will
     append them to ‘reproduce.conf’.  Hence, you do not have to worry
     about remembering the (possibly) different options in the different
     configuration files.

     Afterwards, if you run NoiseChisel as shown below (telling it to
     read this configuration file with the ‘--config’ option).  You can
     be sure that there will either be an error (for version mismatch)
     or it will produce exactly the same result that you got before.

          $ astnoisechisel --config=reproducible.conf

‘--log’
     Some programs can generate extra information about their outputs in
     a log file.  When this option is called in those programs, the log
     file will also be printed.  If the program does not generate a log
     file, this option is ignored.

     *‘--log’ is not thread-safe*: The log file usually has a fixed
     name.  Therefore if two simultaneous calls (with ‘--log’) of a
     program are made in the same directory, the program will try to
     write to he same file.  This will cause problems like unreasonable
     log file, undefined behavior, or a crash.

‘-N INT’
‘--numthreads=INT’
     Use ‘INT’ CPU threads when running a Gnuastro program (see *note
     Multi-threaded operations::).  If the value is zero (‘0’), or this
     option is not given on the command-line or any configuration file,
     the value will be determined at run-time: the maximum number of
     threads available to the system when you run a Gnuastro program.

     Note that multi-threaded programming is only relevant to some
     programs.  In others, this option will be ignored.

   ---------- Footnotes ----------

   (1) Alternatively, you can use your favorite text editor.


File: gnuastro.info,  Node: Shell TAB completion,  Next: Standard input,  Prev: Common options,  Up: Command-line

4.1.3 Shell TAB completion (highly customized)
----------------------------------------------

*Under development:* Gnuastro's TAB completion in Bash already greatly
improves usage of Gnuastro on the command-line, but still under
development and not yet complete.  If you are interested to try it out,
please go ahead and activate it (as described below), we encourage this.
But please have in mind that there are known issues(1) and you may find
new issues.  If you do, please get in touch with us as described in
*note Report a bug::.  TAB completion is currently only implemented in
the following programs: Arithmetic, BuildProgram, ConvertType, Convolve,
CosmicCalculator, Crop, Fits and Table.  For progress on this task,
please see Task 15799(2).

   Bash provides a built-in feature called _programmable completion_(3)
to help increase interactive workflow efficiency and minimize the number
of keystrokes _and_ the need to memorize things.  It is also known as
TAB completion, bash completion, auto-completion, or word completion.
Completion is activated by pressing <[TAB]> while you are typing a
command.  For file arguments this is the default behavior already and
you have probably used it a lot with any command-line program.

   Besides this simple/default mode, Bash also enables a high level of
customization features for its completion.  These features have been
extensively used in Gnuastro to improve your work efficiency(4).  For
example, if you are running ‘asttable’ (which only accepts files
containing a table), and you press <[TAB]>, it will only suggest files
containing tables.  As another example, if an option needs image HDUs
within a FITS file, pressing <[TAB]> will only suggest the image HDUs
(and not other possibly existing HDUs that contain tables, or just
metadata).  Just note that the file name has to be already given on the
command-line before reaching such options (that look into the contents
of a file).

   But TAB completion is not limited to file types or contents.
Arguments/Options that take certain fixed string values will directly
suggest those strings with TAB, and completely ignore the file structure
(for example, spectral line names in *note Invoking astcosmiccal::)!  As
another example, the option ‘--numthreads’ option (to specify the number
of threads to use by the program), will find the number of available
threads on the system, and suggest the possible numbers with a TAB!

   To activate Gnuastro's custom TAB completion in Bash, you need to put
the following line in one of your Bash startup files (for example,
‘~/.bashrc’).  If you installed Gnuastro using the steps of *note Quick
start::, you should have already done this (the command just after ‘sudo
make install’).  For a list of (and discussion on) Bash startup files
and installation directories see *note Installation directory::.  Of
course, if Gnuastro was installed in a custom location, replace the
'‘/usr/local’' part of the line below to the value that was given to
‘--prefix’ during Gnuastro's configuration(5).

     # Enable Gnuastro's TAB completion
     source /usr/local/share/gnuastro/completion.bash

   After adding the line above in a Bash startup file, TAB completion
will always be activated in any new terminal.  To see if it has been
activated, try it out with ‘asttable [TAB][TAB]’ and ‘astarithmetic
[TAB][TAB]’ in a directory that contains tables and images.  The first
will only suggest the files with a table, and the second, only those
with an image.

*TAB completion only works with long option names:* As described above,
short options are much more complex to generalize, therefore TAB
completion is only available for long options.  But do not worry!  TAB
completion also involves option names, so if you just type
‘--a[TAB][TAB]’, you will get the list of options that start with an
‘--a’.  Therefore as a side-effect of TAB completion, your commands will
be far more human-readable with minimal key strokes.

   ---------- Footnotes ----------

   (1) 
<http://savannah.gnu.org/bugs/index.php?group=gnuastro&category_id=128>

   (2) <https://savannah.gnu.org/task/?15799>

   (3) 
<https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html>

   (4) To learn how Gnuastro implements TAB completion in Bash, see
*note Bash programmable completion::.

   (5) In case you do not know the installation directory of Gnuastro on
your system, you can find out with this command: ‘which astfits | sed
-e"s|/bin/astfits||"’


File: gnuastro.info,  Node: Standard input,  Next: Shell tips,  Prev: Shell TAB completion,  Up: Command-line

4.1.4 Standard input
--------------------

The most common way to feed the primary/first input dataset into a
program is to give its filename as an argument (discussed in *note
Arguments::).  When you want to run a series of programs in sequence,
this means that each will have to keep the output of each program in a
separate file and re-type that file's name in the next command.  This
can be very slow and frustrating (mis-typing a file's name).

   To solve the problem, the founders of Unix defined pipes to directly
feed the output of one program (its "Standard output" stream) into the
"standard input" of a next program.  This removes the need to make
temporary files between separate processes and became one of the best
demonstrations of the Unix-way, or Unix philosophy.

   Every program has three streams identifying where it reads/writes
non-file inputs/outputs: _Standard input_, _Standard output_, and
_Standard error_.  When a program is called alone, all three are
directed to the terminal that you are using.  If it needs an input, it
will prompt you for one and you can type it in.  Or, it prints its
results in the terminal for you to see.

   For example, say you have a FITS table/catalog containing the B and V
band magnitudes (‘MAG_B’ and ‘MAG_V’ columns) of a selection of galaxies
along with many other columns.  If you want to see only these two
columns in your terminal, can use Gnuastro's *note Table:: program like
below:

     $ asttable cat.fits -cMAG_B,MAG_V

   Through the Unix pipe mechanism, when the shell confronts the pipe
character (<|>), it connects the standard output of the program before
the pipe, to the standard input of the program after it.  So it is
literally a "pipe": everything that you would see printed by the first
program on the command (without any pipe), is now passed to the second
program (and not seen by you).

   To continue the previous example, let's say you want to see the B-V
color.  To do this, you can pipe Table's output to AWK (a wonderful tool
for processing things like plain text tables):

     $ asttable cat.fits -cMAG_B,MAG_V | awk '{print $1-$2}'

   But understanding the distribution by visually seeing all the numbers
under each other is not too useful!  You can therefore feed this single
column information into *note Statistics:: to give you a general feeling
of the distribution with the same command:

     $ asttable cat.fits -cMAG_B,MAG_V | awk '{print $1-$2}' | aststatistics

   Gnuastro's programs that accept input from standard input, only look
into the Standard input stream if there is no first argument.  In other
words, arguments take precedence over Standard input.  When no argument
is provided, the programs check if the standard input stream is already
full or not (output from another program is waiting to be used).  If
data is present in the standard input stream, it is used.

   When the standard input is empty, the program will wait
‘--stdintimeout’ micro-seconds for you to manually enter the first line
(ending with a new-line character, or the <ENTER> key, see *note Input
output options::).  If it detects the first line in this time, there is
no more time limit, and you can manually write/type all the lines for as
long as it takes.  To inform the program that Standard input has
finished, press <CTRL-D> after a new line.  If the program does not
catch the first line before the time-out finishes, it will abort with an
error saying that no input was provided.

*Manual input in Standard input is discarded:* Be careful that when you
manually fill the Standard input, the data will be discarded once the
program finishes and reproducing the result will be impossible.
Therefore this form of providing input is only good for temporary tests.

*Standard input currently only for plain text:* Currently Standard input
only works for plain text inputs like the example above.  We will later
allow FITS files into the programs through standard input also.


File: gnuastro.info,  Node: Shell tips,  Prev: Standard input,  Up: Command-line

4.1.5 Shell tips
----------------

Gnuastro's programs are primarily meant to be run on the command-line
shell environment.  In this section, we will review some useful tips and
tricks that can be helpful in the pipelines that you run.

* Menu:

* Separate shell variables for multiple outputs::  When you get values from one command.
* Truncating start of long string FITS keyword values::  When the end of the string matters.


File: gnuastro.info,  Node: Separate shell variables for multiple outputs,  Next: Truncating start of long string FITS keyword values,  Prev: Shell tips,  Up: Shell tips

4.1.5.1 Separate shell variables for multiple outputs
.....................................................

Sometimes your commands print multiple values and you want to use them
as different shell variables.  Let's describe the problem (shown in the
box below) with an example (that you can reproduce without any external
data).

   With the commands below, we'll first make a noisy ($\sigma=5$) image
($100\times100$ pixels) using *note Arithmetic::.  Then, we'll
measure(1) its mean and standard deviation using *note Statistics::.

     $ astarithmetic 100 100 2 makenew 5 mknoise-sigma -oimg.fits

     $ aststatistics img.fits --mean --std
     -3.10938611484039e-03 4.99607077069093e+00

*THE PROBLEM:* you want the first number printed above to be stored in a
shell variable called ‘my_mean’ and the second number to be stored as
the ‘my_std’ shell variable (you are free to choose any name!).

The first thing that may come to mind is to run Statistics two times,
and write the output into separate variables like below:

     $ my_std=$(aststatistics img.fits --std)           ## NOT SOLUTION! ##
     $ my_mean=$(aststatistics img.fits --mean)         ## NOT SOLUTION! ##

   But this is not a good solution because as ‘img.fits’ becomes larger
(more pixels), the time it takes for Statistics to simply load the data
into memory can be significant.  This will slow down your pipeline and
besides wasting your time, it contributes to global warming (by spending
energy on an un-necessary action; take this seriously because your
pipeline may scale up to involve thousands of large datasets)!
Furthermore, besides loading of the input data, Statistics (and Gnuastro
in general) is designed to do multiple measurements in one pass over the
data as much as possible (to further decrease Gnuastro's carbon
footprint).  So when given ‘--mean --std’, it will measure both in one
pass over the pixels (not two passes!).  In other words, in this case,
you get the two measurements for the cost of one.

   How do you separate the values from the first ‘aststatistics’ command
above?  One ugly way is to write the two-number output string into a
single shell variable and then separate, or tokenize, the string with
two subsequent commands like below:

     $ meanstd=$(aststatistics img.fits --mean --std)   ## NOT SOLUTION! ##
     $ my_mean=$(echo $meanstd | awk '{print $1}')      ## NOT SOLUTION! ##
     $ my_std=$(echo $meanstd | awk '{print $2}')       ## NOT SOLUTION! ##

*SOLUTION:* The solution is to formatted-print (‘printf’) the numbers as
shell variables definitions in a string, and evaluate (‘eval’) that
string as a command:

     $ eval "$(aststatistics img.fits --mean --std \
                             | xargs printf "my_mean=%s; my_std=%s")"

Let's review the solution (in more detail):

  1. We pipe the output into ‘xargs’(2) (extended arguments) which puts
     the two numbers it gets from the pipe, as arguments for ‘printf’
     (formatted print; because ‘printf’ doesn't take input from pipes).
  2. Within the ‘printf’ call, we write the values after putting a
     variable name and equal-sign, and in between them we put a <;> (as
     if it was a shell command).  The ‘%s’ tells ‘printf’ to print each
     input as a string (not to interpret it as a number and loose
     precision).  Here is the output of this phase:

          $ aststatistics img.fits --mean --std \
                                | xargs printf "my_mean=%s; my_std=%s"
          my_mean=-3.10938611484039e-03; my_std=4.99607077069093e+00

  3. But the output above is a string!  To evaluate this string as a
     command, we give it to the eval command like above.

After the solution above, you will have the two ‘my_mean’ and ‘my_std’
variables to use separately in your pipeline:

     $ echo $my_mean
     -3.10938611484039e-03
     $ echo $my_std
     4.99607077069093e+00

   This ‘eval’-based solution has been tested in in GNU Bash, Dash and
Zsh and it works nicely in them (is "portable").  This is because the
constructs used here are pretty low-level (and widely available).

   For examples usages of this technique, see the following sections:
*note Extracting a single spectrum and plotting it:: and *note Synthetic
narrow-band images::.

   ---------- Footnotes ----------

   (1) The actual printed values by ‘aststatistics’ may slightly differ
for you.  This is because of a different random number generator seed
used in ‘astarithmetic’.  To get an exactly reproducible result, see
*note Generating random numbers::

   (2) For more on ‘xargs’, see <https://en.wikipedia.org/wiki/Xargs>.
It will take the standard input (from the pipe in this scenario) and put
it as arguments of the next program (‘printf’ in this scenario).  In
other words, it is good for programs that don't take input from standard
input (‘printf’ in this case; but also includes others like ‘cp’, ‘rm’,
or ‘echo’).

