Commands¶

The SLM provides some Django management commands out of the box. Management commands are a standard way to implement routine tasks that are installable with your site and may interface with the database using a fully bootstrapped Django stack configured for your site.

The SLM management commands are implemented using Django Typer which extends Django’s base command to work with the Typer CLI library. This also gives us some convenient extra features like shell tab completions.

Most of our SLM management commands manipulate data in ways that would be difficult or too time consuming through the web interface. We also provide some commands that perform backup and restore operations.

We use Django Routines to provide a few named routines to execute batched groups of commands for typical work flows that are required during installation and update deployment.

Management commands are invoked using a Django manage script. If using startproject this script will be called manage.py. If using our Start Project the manage script will be named after your site and indicated in the output report. Manage scripts are typically simple and can be thought of as the main of Django. They bootstrap the Django stack configured for your site and then run the command indicated by the CLI arguments.

Here’s an example of a typical SLM management script:

manage.py¶

import os
import sys

# Use SLM_DEPLOYMENT environment variable to switch between deployments. In
# production you may want to just set this to "production" in your
# administrative user's shell profile
def main(
    default_settings: str = \
    f"sites.<site>.{os.environ.get('SLM_DEPLOYMENT', 'develop')}"
):
    """Run administrative tasks."""

    # SLM configures loggers differently when running management commands
    # so we can distinguish logs on the server that were from requests or
    # admin tasks. This environment variable is used to signal that the SLM
    # runtime is in management mode.
    if len(sys.argv) > 1 and sys.argv[1] != "runserver":
        os.environ['SLM_MANAGEMENT_FLAG'] = 'ON'

    # Django bootstraps off the settings file defined in this import path.
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', default_settings)

    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

Note

In the command helps below, replace <slm> with the name of your manage script.

SLM Management Commands¶
Command	Short Description
`check_upgrade`	Check that the upgrade that is about to be run is safe.
`build_index`	Re-build the head of the file index from the current published database state.
`generate_sinex`	Generate a SINEX file from the published database state.
`head_from_index`	Import data into the database from the latest site log(s) from the file index.
`import_archive`	Add site log files to the file index.
`import_equipment`	Import equipment codes for antennas, receivers and radomes from another SLM.
`set_site`	Set Django Site model fields based on your settings file.
`sitelog`	Generate a serialized site log in a given format from the current database state.
`synchronize`	Synchronize any denormalized state in the database.
`validate_db`	Re-run all site log validation routines on the given (or all) stations.

SLM Management Routines¶
Routine	Short Description
`routine deploy`	Run a collection of commands commonly required when updating SLM code in production.
`routine import`	Run a collection of commands commonly required when importing an external archive of site log data.
`routine install`	Run a collection of commands commonly required when installing to a fresh database. This runs both the `routine deploy` and `routine import` subroutines.

In addition to the SLM specific commands, the common Django commands you will need to use are:

Django Management Commands¶
Command	Short Description
`makemigrations`	Generate migration files from your models that will update the database structure to reflect your code base.
`migrate`	Run any unapplied migration scripts on the database.
`collectstatic`	Gather all static artifacts from Django apps and move them into STATIC_ROOT.
`shell`	Launch an interactive shell with your site bootstrapped and a live connection to the database.

Commands installed by third party apps that may also be useful or necessary:

Third Party Management Commands¶
Command	Short Description
renderstatic	Generate static artifacts from your code/configuration. This mostly includes JavaScript artifacts. See `routine deploy`
shellcompletion	Install/test shell tab completion for all management commands.

check_upgrade¶

Occasionally it is helpful to delete all the migration files and start anew with one simple initial migration. Changes accumulate over time, or old migration files may cause errors on new software stacks. This command facilitates a transition back to one initial migration file on a version upgrade. One side effect of this is that you may need to migrate through multiple versions instead of being able to make one big jump. This command will check for these cases and fail if the migration from the current software version is unsafe on the current database version.

build_index¶

Under normal operations, each time a station’s information is published, a new point in time index is created and logs are serialized in all of the supported formats at that index. If for whatever reason, those logs were not generated or the indexed archived of site logs becomes stale you may run this command to generate indexed site log files of the specified formats.

Warning

This will not regenerate the historical index, but only files from the current database state.

generate_sinex¶

Generates a SINEX file from the current published database state.

The default antex file used is https://files.igs.org/pub/station/general/igs20.atx.gz

SINEX format description:: https://www.iers.org/SharedDocs/Publikationen/EN/IERS/Documents/ac/sinex/sinex_v202_pdf.pdf

Extend this command and (re)implement section producers if you want to customize sinex output.

head_from_index¶

If your site log file index is more current than the database state (i.e. you’ve run import_archive), you may run this command to pull data from the most recent indexed files into the database.

Tip

By default, rich HTML logs of the import process will be written to:

settings.SLM_LOG_DIR / head_from_index.TIMESTAMP

ImportAlerts will also be issued for any errors flagged during import. When, appropriate validation flags will be attached to site log fields. This allows you to manually clean up any import errors through the web interface after the process completes.

import_archive¶

When transitioning site log management into an SLM you will have to import all of the existing data. There are two primary steps involved in doing this:

Import the index of existing serialized point in time site logs.
Populate the site log fields in the database from the most recent site logs for each station.

This command will perform #1 and optionally call head_from_index on the imported stations to also perform #2.

Tip

Rich HTML logs of the import process will be written to:

settings.SLM_LOG_DIR / import_archive.TIMESTAMP

Logs will be parsed and errors reported, but parsing errors will not prevent logs from being indexed.

Warning

If timestamps cannot be determined for files they will not be indexed because each entry in the file index requires a begin and end time.

import_equipment¶

Station equipment including Antennas, Radomes and Receivers are coded by IGS and those standard codes are used to uniquely identify equipment in site log files.

The full IGS change log for equipment codes is recorded in rcvr_ant.tab.

The SLM stores these codes in database tables, so it is possible to instantiate an SLM with a different set of equipment codes. This command can be used to load codes from another SLM or from the IGS by default:

antennas: https://network.igs.org/api/public/antenna/

receivers: https://network.igs.org/api/public/receiver/

radomes: https://network.igs.org/api/public/radome/

manufacturers: https://network.igs.org/api/public/manufacturer/

Tip

import_equipment can be run routinely to synchronize with upstream equipment sources. No equipment will be removed if any site logs reference it in the database.

manufacturers¶

antennas¶

receivers¶

radomes¶

set_site¶

The Django Sites framework is used by the SLM and SLM extensions to store information about the organization name and domain name being served by the current instance. This command populates that information from SLM specific settings:

SLM_SITE_NAME

SLM_ORG_NAME

Note

The sites framework is typically used to support serving multiple domains off the same software stack. In our experience it is usually preferable to serve separate domains off customized software stacks because their requirements may differ substantially. For example network.igs.org and slm.igs.org are both instances of the SLM with different configurations.

sitelog¶

Generate a serialized site log of the given format for a station’s published or unpublished head state. Optionally the file will be pulled from the index of archived site log files.

It is usually more convenient to go through the web interface if you just want to view a sitelog, but this command can be useful for entering into the debugger to troubleshoot parts of this pipeline.

legacy¶

xml¶

synchronize¶

To keep the SLM APIs performant some database state is denormalized. In normal operations this state is synchronized when it needs to be, but if for whatever reason the denormalized state becomes unsynchronized, this command can be manually invoked to force synchronization.

Some examples of denormalized state in the SLM include:

Counts of validation flags

Maximum alert levels for stations

Site log status indicators (PUBLISHED/UNPUBLISHED) for stations.

validate_db¶

The SLM supports pluggable validation logic for all site log sections and fields. Since this logic is configuration specific, it may change, diverging from the validation flags recorded against the previous validation configuration in the database.

Run this command to run all validation routines against all current site log fields in the database.

Routines¶

Routines are groupings of commands that can be configured in settings and run together. They are defined using the app django-routines. The commands that are run as part of a routine can be added to or subtracted from in customized deployments of the SLM. Out of the box the SLM provides the following routines:

deploy¶

The deploy routine includes a collection of commands that should be run each time an updated version of the SLM code base is deployed on a system. These commands will do things like bring the database structure into compliance with the code base, render any static artifacts and copy those artifacts to where the web server expects.

Warning

deploy does not run makemigrations because these files should be checked into version control and we do not want to accidentally generate files and apply them to the database in production. This would likely cause conflicting migration files to be generated in development and production.

migrate will warn if the code is out of sync with the database structure. And you should always run makemigrations before packaging a new version of your software for deployment.

If the SLM validation configuration has changed you will likely want to run deploy with the --re-validate flag to run the updated routines:

<slm> routine deploy --re-validate

You may also wish to not re-validate everything and instead let the new validation routines be run each time a stations data is updated.

import¶

The import routine runs a collection of data import commands you will likely need to run when importing an archive of old site logs.

These commands will import the serialized site logs and attempt to read the data out of those logs and store them in the database.

Note

This import process will import equipment codes from the IGS. You will also be prompted for a path to an archive of historical site logs (either a directory or tar file).

If you need a more bespoke import process you should run each of these import routines separately with customized parameters.

install¶

The install routine runs the initial routine deploy routine as well as the routine import routine. In addition it asks you to create a superuser account.

Run this routine on a fresh database if you have a bunch of site logs to import.