Documentation overhaul

docs/_config.yml

@@ -1,6 +1,6 @@
 title: " "
 author: The TOAST Development Team
-copyright: 2015-2022
+copyright: 2015-2023
 logo: toast_logo_txt_small.png
 
 # Force re-execution of notebooks on each build.
@@ -21,7 +21,7 @@ latex:
 repository:
   url: https://github.com/hpc4cmb/toast  # Online location of your book
   path_to_book: docs  # Optional path to your book, relative to the repository root
-  branch: main  # Which branch of the repository should be used when creating links (optional)
+  branch: toast3  # Which branch of the repository should be used when creating links (optional)
 
 # Add GitHub buttons to your book
 # See https://jupyterbook.org/customize/config.html#add-a-link-to-your-repository

docs/_toc.yml

@@ -6,21 +6,24 @@ chapters:
   sections:
   - file: install_user
   - file: install_dev
-  - file: nersc
+  - file: install_nersc
+  - file: install_test
+- file: interactive
+# - file: quickstart
 - file: data_model
+# - file: data_io
 - file: processing_model
-- file: simulation_operators
-  sections:
-  - file: simulation_operators_satellite
-  - file: simulation_operators_ground
-  - file: simulation_operators_sky
-  - file: simulation_operators_instrument
-- file: reduction_operators
-- file: utility_operators
-- file: tutorial
-  sections:
-  - file: tutorial_intro
+- file: pointing
+# - file: simulated_observing
+# - file: simulated_signals
+# - file: preprocessing
+# - file: utilities
+# - file: mapmaking
+#   # sections:
+#   # - file: mapmaking_utilities
+#   # - file: mapmaking_templates
+- file: api
 - file: dev
 - file: benchmark
 - file: changes
-- file: api
+

docs/data_model.md

@@ -163,3 +163,17 @@ Here is a picture of what data each process would have.  The global process numb
 shown as well as the rank within the group:
 
 ![image](_static/toast_data_dist.png)
+
+(pixel:)=
+## Distributed Map Domain Objects
+
+
+(pixel:dist)=
+### Pixel Distribution
+
+
+(pixel:data)=
+### Pixel Data
+
+
+

docs/install.md

@@ -1,38 +1,6 @@
 # Installation
 
 TOAST is written in C++ and python3 and depends on several commonly available packages.
-It also has some optional functionality that is only enabled if additional external
+It also has optional functionality that is only enabled if additional external
 packages are available.  The best installation method will depend on your specific
 needs.  We try to clarify the different options in the following sections.
-
-
-(install:test)=
-## Testing the Installation
-
-After installation (regardless of how you did that), you can run both the compiled and
-python unit tests.  These tests will create an output directory named
-`toast_test_output` in your current working directory:
-
-```{code-block} console
-python -c "import toast.tests; toast.tests.run()"
-```
-
-If you have installed the `mpi4py` package, then you can also run the unit tests with
-MPI enabled.  For example:
-
-```{code-block} console
-mpirun -np 4 python -c "import toast.tests; toast.tests.run()"
-```
-
-```{important}
-You should use whatever MPI launcher is appropriate for your system (e.g.  `mpirun`,
-`mpiexec`, `srun`, etc).  In general, be sure to set the `OMP_NUM_THREADS` environment
-variable so that the number of MPI processes times this number of threads is not greater
-than the number of physical CPU cores.
-```
-
-The runtime configuration of toast can also be checked with an included script:
-
-```{code-block} bash
-toast_env
-```

docs/install_dev.md

@@ -1,3 +1,4 @@
+
 (install:dev)=
 # Developer Installation
 
@@ -25,6 +26,49 @@ MADAM mapmaking operator | [libmadam](https://github.com/hpc4cmb/libmadam)
 Conviqt beam convolution operator | [libconviqt](https://github.com/hpc4cmb/libconviqt)
 Totalconvolve beam convolution operator | [ducc0](https://pypi.org/project/ducc0)
 
+
+## Using Conda
+
+One of the easiest ways to setup a development environment is to create a conda environment and install all the toast dependencies.  Begin by installing or setting up your conda base environment as described in the user installation guide [here](install:user:forge) or [here](install:user:anaconda).
+
+Next, decide on the name of your conda env used for development.  For this example, we will use the name `toastdev`.  To set up this environment you can run the included script:
+
+    cd toast
+    ./platforms/conda_dev_setup.sh toastdev
+
+Now, we will use the conda tools to build toast in "development mode".  The compiled extension is built in place and the git source tree will be placed in the python search path:
+
+    cd toast
+    conda activate toastdev
+    ./platforms/conda_dev.sh
+
+Now you can activate the conda env and use the toast git checkout directly.  Any changes you make to the python source will show up immediately.  If you change the C++ source, you will need to re-run the `conda_dev.sh` script to rebuild the compiled extension.
+
+
+    bash Miniforge3-Linux-x86_64.sh -b -f -p $HOME/software/condabase
+
+    source $HOME/software/condabase/etc/profile.d/conda.sh
+    conda activate
+    #conda update --yes -n base conda
+    conda update --yes -n base --all
+
+    conda create -n toastdev conda-build cmake psutil cython pytest fftw libaatm suitesparse tomlkit traitlets h5py astropy ephem healpy pshmem coverage pytest coveralls pytest-cov
+
+    conda activate toastdev
+
+    conda install gcc_linux-64 gxx_linux-64
+    OR
+    conda install compilers
+
+    pip install pixell
+
+    ./dev_build.sh
+
+    NERSC:
+    MPICC="cc -shared" pip install --force-reinstall --no-cache-dir --no-binary=mpi4py mpi4py
+
+
+
 ## Installing Build Dependencies
 
 The compiled extension within TOAST has several libraries that need to be found at build
@@ -148,6 +192,9 @@ to install these packages:
 
 ## Install Optional Dependencies
 
+conda install jupyter-notebook wurlitzer ipywidgets plotly python-kaleido
+
+pip install plotly-resampler
 
 
 ## Installing TOAST with CMake

docs/install_test.md

@@ -0,0 +1,31 @@
+(install:test)=
+# Testing the Installation
+
+After installation (regardless of method), you can run both the compiled and
+python unit tests.  These tests will create an output directory named
+`toast_test_output` in your current working directory:
+
+```{code-block} console
+python -c "import toast.tests; toast.tests.run()"
+```
+
+If you have installed the `mpi4py` package, then you can also run the unit tests with
+MPI enabled.  For example:
+
+```{code-block} console
+export OMP_NUM_THREADS=2
+mpirun -np 2 python -c "import toast.tests; toast.tests.run()"
+```
+
+```{important}
+You should use whatever MPI launcher is appropriate for your system (e.g.  `mpirun`,
+`mpiexec`, `srun`, etc).  In general, be sure to set the `OMP_NUM_THREADS` environment
+variable so that the number of MPI processes times this number of threads is not greater
+than the number of physical CPU cores.
+```
+
+The runtime configuration of toast can also be checked with an included script:
+
+```{code-block} bash
+toast_env
+```

docs/install_user.md

@@ -11,8 +11,8 @@ If you want to use a pre-installed version of TOAST at NERSC,
 
 ## Pip Binary Wheels
 
-If you already have a newer Python3 (\>= 3.7), then you can install pre-built TOAST
-packages from PyPI.  You should always use virtualenv or similar tools to manage your
+If you already have a newer Python3 (\>= 3.8), then you can install pre-built TOAST
+packages from PyPI.  You should **always** use virtualenv or similar tools to manage your
 python environments rather than pip-installing packages as root.
 
 On Debian / Ubuntu Linux, you should install these minimal packages:
@@ -33,7 +33,7 @@ scl enable rh-python38 bash
 ```
 
 On MacOS, you can use homebrew or macports to install a recent python3.
-Now verify that your python is at least 3.7:
+Now verify that your python is at least 3.8:
 
 ```{code-block} bash
 python3 --version
@@ -64,36 +64,13 @@ Next, use pip to install toast and its requirements:
 pip install toast
 ```
 
-(install:user:mpi)=
-### Enabling MPI Support
-
-At this point you have toast installed and you can use it from serial scripts and
-notebooks.  If you want to enable effective parallelism with toast (useful if your
-computer has many cores), then you need to install the `mpi4py` package.  This package
-requires MPI compilers (usually MPICH or OpenMPI).  Your system may already have some
-MPI compilers installed- try this:
-
-```{code-block} bash
-which mpicc
-mpicc -show
-```
-
-If the mpicc command is not found, you should use your OS package manager to install the
-development packages for MPICH or OpenMPI.  Now you can install mpi4py:
-
-```{code-block} bash
-pip install mpi4py
-```
-
-For more details about custom installation options for mpi4py, read the [documentation
-for that package](https://mpi4py.readthedocs.io/en/stable/install.html).  After
-installation, [you should run the unit tests](install:test)
-
 (install:user:conda)=
 ## Conda Packages
 
-If you already use (or would like to use) the conda python stack, then you can install
-TOAST and all of its optional dependencies with the conda package manager.  The
+The conda package manager and conda-forge ecosystem provides thousands of high quality software packages (both python and compiled tools) that are built with a consistent set of compilers and provide a way of managing python environments independent from the operating system environment.
+
+You can install
+TOAST and many of its optional dependencies with the conda package manager.  The
 conda-forge ecosystem allows us to create packages that are built consistently with all
 their dependencies.  When we talk about the `base` conda environment (previously called
 the "root" environment), this is the initial environment loaded when the conda shell
@@ -103,11 +80,12 @@ across all the different projects you might be working on.  It also creates a
 maintenance nightmare when you need to update packages.  In this section we walk through
 creating a conda *environment* to use for TOAST / CMB analysis work.
 
-By keeping a minimal base and using environments for all other work, it is trivial to
+By keeping a minimal base and using other environments for all other work, it is trivial to
 update the conda tool itself and the other essential packages in base.  If one of your
 working environments becomes horribly out of date or broken, just delete it and make a
 new one.
 
+(install:user:anaconda)=
 ### Using Anaconda with conda-forge Packages
 
 If you already have Anaconda python installed, the base conda environment may already be
@@ -135,6 +113,7 @@ conda config --set channel_priority strict
 
 Now skip ahead to the section on [creating an environment](install:user:conda:env).
 
+(install:user:forge)=
 ### Using a Native conda-forge Base
 
 If you are starting from scratch, we recommend using the "miniforge" installer to set up
@@ -189,6 +168,34 @@ conda deactivate
 As always, after installation, [you should run the unit tests](install:test).
 
 
+
+(install:user:mpi)=
+### Enabling MPI Support
+
+At this point you have toast installed and you can use it from serial scripts and
+notebooks.  If you want to enable effective parallelism with toast (useful if your
+computer has many cores), then you need to install the `mpi4py` package.  This package
+requires MPI compilers (usually MPICH or OpenMPI).  Your system may already have some
+MPI compilers installed- try this:
+
+```{code-block} bash
+which mpicc
+mpicc -show
+```
+
+If the mpicc command is not found, you should use your OS package manager to install the
+development packages for MPICH or OpenMPI.  Now you can install mpi4py:
+
+```{code-block} bash
+pip install mpi4py
+```
+
+For more details about custom installation options for mpi4py, read the [documentation
+for that package](https://mpi4py.readthedocs.io/en/stable/install.html).  After
+installation, [you should run the unit tests](install:test)
+
+
+
 ## Something Else
 
 If you have a custom install situation that is not met by the above solutions, then you