Project — ADMIT project.

This module defines the Admit project class.

class admit.Admit.Admit(baseDir=None, name='none', basefile=None, create=True, dataserver=False, loglevel=20, commit=True)

Container for an ADMIT project. The project is normally based on one single FITS cube (in some cases two, where the ingest stage needs a primary beam to correct the science cube with), although this is not a restriction to ADMIT.

A FITS cube results in an ADMIT directory, within which you will find an admit.xml file describing the project, it’s data products (BDP’s) and the AT’s (tasks) that generated the BDP’s.

If input file/directory are given or admit.xml is located in the current directory then they are loaded into the class, else a new (empty) class is instantiated.

Parameters:

baseDir : str

Base directory for XML files (the “ADMIT directory”).

name : str

Alias name.

basefile : str

Base XML file name (default: admit.xml).

create : bool

Whether to create any needed directories.

dataserver : bool

Whether to start the data browser server.

loglevel : int

The integer log level from the Python logging module. One of:

• logging.CRITICAL = 50
• logging.ERROR = 40
• logging.WARNING = 30
• logging.INFO = 20
• logging.DEBUG = 10

Default is logging.INFO.

commit : bool, optional

Whether to commit XML-backed flows immediately; default is True. Set to True if the flow will not be reconstructed (as in a recipe script) before use; this is usually the case for interactive mode. Set to False in (most) scripts, which reconstruct the flow each time.

Notes

Todo

1. in the current implementation every directory, admit or not-admit, can be made an admit directory (i.e. contain a root admit.xml)

2. we really don’t need a basefile= in the argument list

Attributes

baseDir	(str) Base directory for XML files (the “ADMIT directory”). Guaranteed to end in os.sep.
baseFile	(str) Base XML file name, usually admit.xml.
currDir	(str) Current working directory (at construction).
fm	(FlowManager) Project flow manager instance.
new	(bool) Whether the project is new or constructed from an existing XML file.
pm	(ProjectManager) Project manager instance.
pmode	(int) Plotting mode.
ptype	(int) Plotting type.
count	(int) Flow counter how many times the flow has been run (stored in userData)
project_id	(int) Static project identification number
summaryData	(instance of admit.Summary) AT summary data
userData	(dict) Additional, user-defined data.
_data_browser_port	(int) Port number that the localhost http server for the data browser (aka data GUI) will use. This attribute is set by the operating system.
_data_server	(bool) Whether to start the data browser server.
_server	(HTTP server) Data HTTP server.

Methods

addtask(a[, stuples, dtuples])	Add an AT to the project.
atToHTML()	Write individual AT data to the html form
browse()	Open a web browser tab with the URL of this admit project
check()	Check all project BDPs for name collisions.
clean()	Method to delete orphan bdp’s (files and underlying data)
dir()	See AT.dir() but placed here for convenience as well.
discover([mode, rootdir])	Project data discovery.
dryrun()
exit(exit)	Early cleanup and exit if exit > 0
export([level, casa, fits, out])
find_bdp()	Find all bdp’s in the current admit.
find_files([pattern])	Find files containing a wildcard pattern.
findtask(isMatch)	Finds ATs in the flow matching a criterion.
get(key)	Get a global ADMIT parameter.
getFlow()	Returns flow manager instance.
getManager()	Returns project manager instance.
getlogginglevel()	Method to return the current logging level
has(key)	Query if a global user key exists for this admit project.
logToHTML()	Write the admit.log to an html file
mergeFlow([finalize])	Merges tasks from the XML-derived flow (if any).
mkdir(dirname)	Make a directory in the ADMIT hierarchy, if it doesn’t exist yet.
plotparams([plotmode, plottype])	Determines if plots are saved and in what format.
print_all()	Calls all the methods of this object.
print_attributes()	Print all the attributes of this object and their value(s).
print_methods()	Print all the methods of this object and their doc string(s).
print_summary()	Print out summary data
read()	Reads a project.
run([write, commit])	Runs the project flow.
script(pyfile)	Generates a Python script regenerating the current project.
set(\*\*kwargs)	Sets keys and values in userData.
setAstale(astale[, verbose, dryrun])	Method to toggle the stale flags on all tasks based on a global admit stale for the sole purpose of admit_export to work.
setdir(dirname[, create])	Changes current working directory.
setlogginglevel(level)	Method to set the logging level
show()	Prints project state.
showsetkey([outfile])	Show current keys for tasks
startDataServer()	Starts the data HTTP server.
tesdir()	Revert back from previous setdir (not recursive yet).
updateHTML()	Writes out HTML views of this object.
url()	Print the URL for the data browser
userdata()	Collects current AT userdata.
version()	return version of ADMIT
write()	Writes out the admit.xml file, admit0.py script and project html files.
writeXML([script])	Writes out the admit.xml file and admit0.py script.

addtask(a, stuples=None, dtuples=None)

Add an AT to the project.

Also adjusts the connection mapping between tasks. Usually all but the first task—typically, Ingest_AT—will have ‘stuples’ (a List of Source Tuples (task-id,bdp-id)). A source 2-tuple consists of a task ID (task-id, such as returned by this method) and BDP output slot number (bdp-id, zero-based). If the output slot is zero (the tuple refers to the first BDP output from the task), then the tuple can be replaced by the task ID for convenience—e.g., stuples = [(t1,0), (t2,1)] is equivalent to stuples = [t1, (t2,1)].

Support for re-running scripts: this method will ignore attempts to re-add a task of the same type and ID to the existing flow, if the project has been restored from XML. Between invocations, scripts may be edited to append new tasks to the flow, but not remove or insert them. Keywords for existing ATs may also be changed by the script; if changes are found, the existing task will be marked out-of-date.

Parameters:

a : AT

ADMIT task to append/insert into the flow.

stuples : list of 2-tuples, optional

List of source connection 2-tuples, one per BDP input port.

dtuples : list of 4-tuples, optional

List of destination connection 4-tuples.

Returns:

int

Input task ID on success, else -1 (error detected).

See also

add

atToHTML()

Write individual AT data to the html form

browse()

Open a web browser tab with the URL of this admit project

check()

Check all project BDPs for name collisions. Also identifies orphaned branches of the tree. A topological sort is needed as well, if they are not in the correct execution order.

See also

UNIX

clean()

Method to delete orphan bdp’s (files and underlying data)

Parameters:None Returns:None

dir()

See AT.dir() but placed here for convenience as well.

Parameters:

None

Returns:

str

Base directory.

discover(mode=None, rootdir='.')

Project data discovery.

Parameters:

mode : TBD

Discovery mode.

rootdir : str, optional

Search root directory.

Returns:

list

Search results.

dryrun()

exit(exit)

Early cleanup and exit if exit > 0

Parameters:

exit : int

The exit code to exit with (must be > 0)

Returns:

None

export(level=0, casa=True, fits=False, out=None)

find_bdp()

Find all bdp’s in the current admit.

Parameters:

None

Returns:

list

All *.bdp files within the admit hierarchy.

find_files(pattern='*.fits')

Find files containing a wildcard pattern.

Parameters:

pattern : str, optional

File name wildcard pattern.

Returns:

list

File names matching the pattern.

findtask(isMatch)

Finds ATs in the flow matching a criterion.

Applies the function isMatch to all ATs in the flow, in proper dependency order, accumulating matching ATs in a list (the return value). Downstream ATs are guaranteed to follow their predecessors in this list. Often isMatch may be conveniently expressed as a lambda function.

Parameters:

isMatch : bool functor(AT)

Function object taking one AT as input and returning a Boolean.

Returns:

list of ATs

ATs testing True using isMatch.

Notes

This method is a wrapper for FlowManager.find().

Examples

To find all ATs with ID less than 100 in project p:

>>> p.find(lambda at: at.id() < 100)

get(key)

Get a global ADMIT parameter.

Parameters:

key : str

User-defined data keyword.

Returns:

str

User-defined (userData) keyword value.

Notes

Todo

This method should mirror the way we do this in the AT (setkey/getkey)

getFlow()

Returns flow manager instance.

Parameters:

None

Returns:

FlowManager

Flow manager instance.

getManager()

Returns project manager instance.

Parameters:

None

Returns:

ProjectManager

Project manager instance.

getlogginglevel()

Method to return the current logging level

Parameters:None Returns:An int representing the current logging level

has(key)

Query if a global user key exists for this admit project.

Parameters:

key : str

User-defined data keyword.

Returns:

bool

True if keyword is present in userData, else False.

logToHTML()

Write the admit.log to an html file

loginit = False

mergeFlow(finalize=True)

Merges tasks from the XML-derived flow (if any).

When projects are restored to memory from persistent XML files, that task flow is initially held in stasis while the (possibly modified) flow is being reconstructed, typically by re-running a script. This reconstruction phase lasts from the point where the XML is read up to the first call to this method with finalize set (most typically, the first call to run(), which calls this method internally). Calling this method during reconstruction compares the old flow to the newly constructed flow and tasks present unaltered in the new flow (i.e., same BDP inputs and keyword values as before) are marked up to date, if they were up to date in the original flow. Other relevant attributes are transferred as appropriate.

Parameters:

finalize : bool, optional

Whether to discard the XML-derived flow after merge analysis, preventing future merge attempts.

Returns:

None

Notes

It is permissible to add or remove arbitrary tasks from the flow, in an arbitrary order, while reconstructing it. Tasks unaffected by the changes (if any) will not be re-executed gratuitously.

After finalization, the old flow is forgotten and subsequent calls will have no effect (and likewise for fresh projects not backed by XML).

mkdir(dirname)

Make a directory in the ADMIT hierarchy, if it doesn’t exist yet. It also allows an absolute path, in the classical unix sense, but this is normally not needed.

Parameters:

dirname : str

Directory name.

Returns:

None

plotparams(plotmode=32, plottype=5)

Determines if plots are saved and in what format. These are based on simple matplotlib diagrams. Common output formats are png and pdf. Note: this only applies to new AT’s started in a flow, to change existing parameters in a re-run for example, you will need to manually change the AT._plot_mode and AT._plot_type

Parameters:

plotmode : int

Plotting mode. Default: PlotControl.BATCH

plottype : int

Plot format type. Default: PlotControl.PNG.

Returns:

None

See also

util.PlotControl

print_all()

Calls all the methods of this object.

Parameters:None Returns:None

print_attributes()

Print all the attributes of this object and their value(s).

Parameters:None Returns:None

print_methods()

Print all the methods of this object and their doc string(s).

Parameters:None Returns:None

print_summary()

Print out summary data

Parameters:None Returns:None

project_id = 0

read()

Reads a project.

Notes

Not implemented.

run(write=True, commit=True)

Runs the project flow.

Run those pieces of the pipeline flow deemed out of date. After the run, the flow tasks gather their summary into ADMIT’s summaryData, ensuring that summaryData always is consistent with the current flow, and does not contain remnants from orphans.

Parameters:

write : bool, optional

Whether to write the project XML files after running the flow; default is True.

commit: bool, optional

Whether to commit the current flow after merging flow tasks with the XML-derived flow (if present). Set to False during incremental run()/addtask() flow reconstruction. Once a flow is committed, all requests to add or remove flow tasks will vest immediately. Default is True.

Returns:

None

See also

mergeFlow

Notes

This method supports intelligent re-running of projects read from XML. Task flows may be reconstructed (as in a script) in any order, from the point where the XML is read up to the first call to run() (with commit=True). Tasks present (unaltered) in the new flow and marked up to date in the XML will not be re-executed.

script(pyfile)

Generates a Python script regenerating the current project.

The resulting script is intended to recreate the project results from scratch and to be run from the parent of the project directory. Running the script over existing project results is unpredictable and not supported.

Parameters:

pyfile : str

Output Python script file name.

Returns:

None

set(**kwargs)

Sets keys and values in userData.

Parameters:

kwargs : dictionary like

Command line arguments for the function, can be a=x,b=y or **{a:x, b:y} format

Returns:

None

setAstale(astale, verbose=False, dryrun=False)

Method to toggle the stale flags on all tasks based on a global admit stale for the sole purpose of admit_export to work. It is dangerous to call this routine when not all tasks are either stale or not stale.

This function needs to be called with True first, so it makes a stale backup, then during the 2nd False call, the stale backup is pushed back.

@todo This is a patch solution for admit 1.1 - general solution needed

setdir(dirname, create=True)

Changes current working directory. The directory is assumed to contain parameter file.

Note

Deprecated. See pushd()/popd() for a better version.

Parameters:

dirname : str

Directory to work in.

create : bool, optional

Whether to create the directory if it doesn’t exist.

Notes

Todo

the new mkdir() and self.baseDir are the way to work in ADMIT

setlogginglevel(level)

Method to set the logging level

Parameters:

level : int

The logging level to use

Returns:

None

show()

Prints project state.

Parameters:None Returns:None

Notes

Currently only display FlowManager contents.

showsetkey(outfile=None)

Show current keys for tasks For now on screen, but meant to aid writing a template file for rerun

Parameters:

outfile : str

The name of the output file

Returns:

None

startDataServer()

Starts the data HTTP server. On a separate thread, start the http server on localhost:_data_browser_port that will allow web browsing of data products. Also attempt to open a browser window at that URL. When this method returns, the variable self._data_browser_port will have the value of the port returned by the OS. See util.AdmitHTTP.AdmitHTTPServer

Parameters:None Returns:None

tesdir()

Revert back from previous setdir (not recursive yet).

Note

Deprecated. See pushd()/popd() for a better version.

updateHTML()

Writes out HTML views of this object. It is expected that summary() has been called first.

Parameters:None Returns:None

url()

Print the URL for the data browser

Parameters:None Returns:String representing localhost url on which data can be viewed.

userdata()

Collects current AT userdata. warning: No check is done for duplicate keys!

Parameters:None Returns:None

version()

return version of ADMIT

write()

Writes out the admit.xml file, admit0.py script and project html files.

Parameters:None Returns:None

writeXML(script=True)

Writes out the admit.xml file and admit0.py script.

Reading the XML file occurs in the constructor.

Parameters:None Returns:None