Task — ADMIT task (AT) base.

This module defines the ADMIT task (AT) base class.

class admit.AT.AT(keys={}, keyvals={})

Base class for Admit Task (AT) objects.

The ADMIT task base class implements generic interface features common to concrete ATs. Tasks are the basic execution unit in ADMIT and are responsible for operating on BDPs to perform calculations or otherwise transform them into new data products. Each task includes a unique set of keywords controlling its behavior.

Parameters:

keys : dictionary of str keywords with default values, optional.

Task keyword settings; default keyword values vary by task.

Currently the following keywords, defined by the base class, are activated for each task:

alias

: str

By default ATs will use an alias by appending the file basename they normally use with “-alias”. For example, if an AT would normally transform “foo.im” to “foo.pv”, with “alias=test” you would get “foo-test.pv”. The notable exception is Ingest_AT, where the alias replaces the (potentially long) cube file basename, i.e., “verylongsourcename.fits” becomes “alias.im”. Must be unique for each concrete type of AT (will be appended with a unique suffix otherwise). Alias is a system keyword, inaccessible (via getkey()) after the task is instantiated.

taskid

: int

Forces a pre-determined task ID number when instantiating the task. Normally left defaulted (-1, meaning auto-select), but useful in special circumstances. This is a system keyword, inaccessible (via getkey()) after the task is instantiated.

variflow

: bool

Whether the flow emanating from an AT with variadic outputs should be dynamically adjusted according to the actual number of outputs. Normally left defaulted (True), but may be set to False if this will be managed separately or if the number of outputs is pre-determined for a particular instance. This is a system keyword, inaccessible (via getkey()) after the task is instantiated.

keyvals : dictionary of new values for the keywords, optional.

These keyword value pairs can be fed in form the command line instantiation of the class.

See also

isAutoAlias

Notes

As a general rule the attributes of an AT should not be directly manipulated by the code in an AT—all are taken care of internally or accessed via methods (e.g., setkey).

Attributes

_alias	(str) Task alias name. Appears in various filenames, HTML output, flow diagrams, etc.
_taskid	(int) Task ID number (defaults to None, normally set by FlowManager).
_baseDir	(str) Project base directory (None if unknown).
_bdp_in	(list of BDPs) BDP input cache.
_bdp_in_map	(list of ints) BDP IDs corresponding to those listed in _bdp_in.
_bdp_out	(list of BDPs) BDP output cache.
_bdp_out_map	(list of ints) BDP IDs corresponding to those listed in _bdp_out.
_keys	(dict) Task keyword dictionary.
_link	(int) Link count (for multiflows).
_type	(str) Concrete task type.
_stale	(boolean) Whether the AT needs to be run (e.g., a keyword changed).
_enabled	(boolean) Whether the AT is enabled; if set to False, the flow manager will not execute the local run method.
_running	(boolean) Whether the AT is currently running (within the execute() method). Useful for crash recovery diagnostics.
_plot_mode	(int) Plot mode, one of util.PlotControl plot mode (e.g., PlotControl.INTERACTIVE). Default: PlotControl.NOPLOT
_plot_type	(int) Plotting format, one of util.PlotControl plot type (e.g., PlotControl.PNG). Default: PlotControl.NONE
_valid_bdp_in	(List) List of tuples indicating what types and how many BDPs are expected as input.
_valid_bdp_out	(List) List of tuples indicating what types and how many BDPs are expected as output.
_needToSave	(boolean) Whether this AT and underlying BDPs need to be saved to disk.
_variflow	(bool) Whether sub-flows attached to the instance should be automatically cloned in the case of variadic output.

Methods

addinput(item[, slot])	Add a BDP input to an AT.
addinputbdp(item[, slot, insert])	Add a BDP to the _bdp_in list.
addoutput(item[, slot])	Add a BDP output product to an AT.
addoutputbdp(item[, slot, insert])	Add a BDP to the _bdp_out list.
baseDir([path])	Get/set project base directory.
bestMatch(at1, at2)	Determines the better match of two tasks to the current one.
checkfiles()	Check if the files from all the BDP_out’s in an AT exist.
checktype(item)	Check the type of an object to see if it is a BDP.
clearinput()	Clear the input BDP list.
clearoutput([delete])	Clear the output BDP list.
copy()	Creates an independent duplicate of the task.
delete()	Method to delete the AT and underlying BDPs.
delinput(slot)	Delete a specific BDP in the _bdp_in list.
deloutput(slot)	Delete a specific BDP in the _bdp_out list.
dir([filename])	Absolute directory reference of the ADMIT project.
dryrun()	Method to do a dry run of the AT, generally just checks input values for errors.
enabled([state])	Returns current task enabled setting, with optional reset.
execute([args])	Executes the task.
freeAlias(aliases[, alias])	Deletes alias reservation, if present.
get(attrib)	Method to get the given attributes value
getProject()	Retrieves project ID associated with the task.
getVersion()	Return the version string.
getdtd(fl)	Method to write out the dtd data.
geteffectivelevel()	Method to get the effective logging level of the logging subsystem
getkey(key)	Retrieval value for a key.
getloggername()	Method to get the name of the logger for this AT instance
getlogginglevel()	Method to get the current logging level of the AT
haskey(key)	Query if a key exists for an AT.
html(inheader)	Method to represent the current AT in HTML format.
id([strip])	Returns task ID number.
isAutoAlias([withEmpty, compat])	Whether the task alias appears to be auto-generated.
isequal(at)	Method to determine if two ATs are the same.
isstale()	Returns whether the AT is out of date.
len2()	Returns the length of _bdp_in and _bdp_out in a tuple.
link()	Increments the task link count.
markChanged()	Mark an AT that it’s state was changed, so it would need to be rerun.
markUpToDate()	Resets _stale to indicate that the AT does not need to be run.
merge(at[, aliases])	Merges attributes from another task.
mkdir(dirname)	Make a directory in the ADMIT hierarchy.
mkext(filename, ext[, alias])	Return a new filename with a new extension with optional ADMIT alias.
newId(tid)	Assigns the task a new ID number.
reset(a)	Performs an in-place shallow copy.
run()	Runs the task.
running([state])	Returns current task execution flag, with optional reset.
save()	Save (write) any BDPs connected to this AT.
set(item, val)	Method to set protected attributes, rather than direct access
setAlias(aliases[, alias, auto])	Sets and registers the task alias, guaranteed unique among registered aliases.
setProject(pid)	Adds a project ID to task ID.
set_bdp_in([bdpin])	Validate the _valid_bdp_in list and digest it into the appropriate attributes.
set_bdp_out([bout])	Validate the _valid_bdp_out list and digest it into the appropriate attributes.
seteffectivelevel(level)	Method to set the effective logging level of the logging subsystem
setkey([name, value, isinit])	Set keyword value.
setloggername(name)	Method to set the name of the logger for this AT instance
setlogginglevel(level)	Method to set the logging level
show()	Return the AT type.
statusicons()	return some html icons representing the enabled/stale status of this task
summary()	Returns the summary dictionary from the AT, for merging into the ADMIT Summary object.
unlink()	Decrements the task link count.
userdata()	Returns the user dictionary from the AT, for merging into the ADMIT userdata object.
validateinput([describe])	Method to validate the _bdp_in’s against a dictionary of expected types.
validatekeys()	Method to error check all input keys.
write(node)	Method to write the AT to disk.

addinput(item, slot=-1)

Add a BDP input to an AT. The BDP type will be validated before it is added to the input list.

Parameters:

item : BDP

The BDP to be added to the input list.

slot : int, optional

The slot in the BDP to add it to. The BDP type must match the expected type for the slot. Default: -1 (i.e. insert into the first available slot)

Returns:

None

addinputbdp(item, slot=-1, insert=False)

Add a BDP to the _bdp_in list.

Parameters:

item : BDP

The BDP to add.

slot : int, optional

Where the BDP should be added to the list; default : -1

insert : Boolean, optional

Whether to insert into the middle of the list (not replace); defaults to False.

Returns:

None

Notes

This method should not be called directly, use addinput instead.

addoutput(item, slot=-1)

Add a BDP output product to an AT. The BDP type will be validated before it is added to the output list.

Parameters:

item : BDP

The BDP to be added to the output list.

slot : int, optional

The slot in the BDP to add it to. The BDP type must match the expected type for the slot. Default: -1 (i.e. insert into the first available slot)

Returns:

None

addoutputbdp(item, slot=-1, insert=False)

Add a BDP to the _bdp_out list.

Parameters:

item : BDP

The BDP to add.

slot : int, optional

Where the BDP should be added to the list; default : -1

insert : Boolean, optional

Whether to insert into the middle of the list (not replace); defaults to False.

Returns:

None

Notes

This method should not be called directly, use addoutput instead.

baseDir(path=None)

Get/set project base directory.

Unless empty, the base directory is guaranteed to end in os.sep.

Parameters:

path : str, optional

New project base directory (ignored if None).

Returns:

Updated project base directory.

bestMatch(at1, at2)

Determines the better match of two tasks to the current one.

Type is most important, then alias, then keywords (most matching keywords, followed by most matching values).

Parameters:

at1 : AT

First task.

at2 : AT

Second task.

Returns:

int

ID number of best matching task (in the case of a tie, the first task ID is returned).

checkfiles()

Check if the files from all the BDP_out’s in an AT exist. Return the list of files not found.

Parameters:None Returns:None

Notes

In multiflows, BDPs for ATs linked from other projects (distinguished by a non-zero value of getProject()) are not (re)checked for existence; this occurs when the parent project is read.

checktype(item)

Check the type of an object to see if it is a BDP.

Parameters:

item : Any

The item to check.

Returns:

Boolean

True if the item is a BDP, False otherwise.

clearinput()

Clear the input BDP list.

Parameters:None Returns:None

clearoutput(delete=True)

Clear the output BDP list.

Parameters:

delete : bool

If True then delete the output bdps. Default: True

Returns:

None

copy()

Creates an independent duplicate of the task.

The task ID of the duplicate will be unset and the alias marked to allow auto-updating.

Parameters:

aliases : dict of set of str

Reserved alias registry.

Returns:

Reference to the new (cloned) task.

Notes

The current implementation performs a deepcopy() and updates a few members.

delete()

Method to delete the AT and underlying BDPs.

It is recommended that any AT that stores images inside of lists, dictionaries, tuples, etc. override this method with a customized version.

Parameters:None Returns:None

delinput(slot)

Delete a specific BDP in the _bdp_in list.

Parameters:

slot : int

The slot to remove the BDP from

Returns:

None

deloutput(slot)

Delete a specific BDP in the _bdp_out list.

Parameters:

slot : int

The slot to remove the BDP from

Returns:

None

dir(filename=None)

Absolute directory reference of the ADMIT project.

Returns the absolute directory name of this ADMIT project or a derefenced filename address. Normally you will need such an absolute file or directory name when non-ADMIT tools are needed for I/O. Also note that this project directory is guarenteed to end with the operating dependent directory separator (usually ‘/’). This way filenames can be constructed as follows: Admit.dir() + ‘foobar.dat’ or more conveniently Admit.dir(‘foobar.dat’)

Parameters:

filename : string, optional

Filename to be appended to the absolute directory name. This can include additional subdirectories.

Returns:

string

The directory name within which all ADMIT files reside.

dryrun()

Method to do a dry run of the AT, generally just checks input values for errors.

Parameters:None Returns:bool, True if all is ok, False otherwise

enabled(state=None)

Returns current task enabled setting, with optional reset.

Parameters:

state : bool, optional

New value of enabled flag (default None keeps current value).

Returns:

bool

Whether task is enabled (prior to applying state, if provided).

execute(args=None)

Executes the task.

Execution performs the following actions: 0. Mark task running. 1. Update BDP inputs (if args is given). 2. Validate task keywords. 3. Validate task BDP inputs. 4. Call task run(). 5. Mark task up to date. 6. Mark persistent BDP outputs out of date. 7. Disable running attribute.

Parameters:

args : list of BDPs, optional

List of BDP inputs used by run() method; default is to use the most recently supplied inputs.

Returns:

None

See also

run

Notes

If args is not given, the previously cached BDP inputs are reused; hence execution (input validation) will fail if required arguments are absent.

freeAlias(aliases, alias=None)

Deletes alias reservation, if present.

Parameters:

aliases : 2-tuple of dict of str to set of str

Reserved alias registry, keyed by alias base name (index 0) or AT type (index 1). For index 0, the value is the set of aliases created with that base name; for index 1, it is the set of default aliases created for ATs of that type. In both cases, the duplicate aliases are appended with ‘@N’ for some integer N.

alias : str, optional

Alias name to release; default None means use the current AT’s alias.

Returns:

None

get(attrib)

Method to get the given attributes value

Parameters:

attrib : str

The name of the attribute to get

Returns:

varies, depends on the type of the attribute

The value of the attribute, or None is it does not exist

getProject()

Retrieves project ID associated with the task.

Multiflows involve adding tasks belonging to other projects into a single flow, the multiflow. To prevent task ID collisions, tasks in multiflows embed a project number into the task ID.

This method returns the project ID attached to the task. This will be zero unless setProject() has been called—normally only by the ProjectManager when a project is added for use in a multiflow.

Returns:

int

Task project ID.

See also

setProject

Notes

Project IDs are local to each multiflow. Hence the same task, linked in two independent multiflows, may have a different project ID in each. This has no effect on the task in its parent flow (for which this method will return zero).

getVersion()

Return the version string.

Parameters:

None

Returns:

String

The version of the AT

getdtd(fl)

Method to write out the dtd data.

Parameters:

fl : file handle

Open output file handle.

Returns:

None

geteffectivelevel()

Method to get the effective logging level of the logging subsystem

Parameters:None Returns:int, the effective logging level

getkey(key)

Retrieval value for a key. If the key had not been set yet, a default value can be returned, else None is returned. Notice no type checking has been done here, the caller is responsible for this.

Parameters:

key : string

The key whose value is to be returned.

Returns:

The value of the key.

See also

haskey

getloggername()

Method to get the name of the logger for this AT instance

Parameters:None Returns:String containing the name of the logger.

getlogginglevel()

Method to get the current logging level of the AT

Parameters:None Returns:int, the current logging level

haskey(key)

Query if a key exists for an AT.

Parameters:

key : string

The key to check for existence.

Returns:

Boolean

True if the key exists, False otherwise.

html(inheader)

Method to represent the current AT in HTML format.

Parameters:

inheader : str

Base header info for the html output

Returns:

An html representation of the AT

id(strip=False)

Returns task ID number.

Parameters:

strip : bool, optional

Whether to strip project ID (if any) from returned value (default: False).

Returns:

int

Task ID number.

Notes

Raises an exception if no ID assigned (freshly constructed tasks do not have an ID, it is set when the task is added to a project).

isAutoAlias(withEmpty=True, compat=None)

Whether the task alias appears to be auto-generated.

The form of auto-generated aliases is a string ending in ‘@’ followed by a decimal integer. Users should avoid defining aliases which look like ADMIT-generated aliases as they may be reset automatically, but are not actively prevented from doing so.

Parameters:

withEmpty : bool, optional

Whether an empty alias is considered auto-generated.

compat : str, optional

An alias name to test for compatibility. If not None, the alias stem (with its ‘@N’ removed) must match the stem for compat for this method to return True.

Returns:

bool

Whether the task alias was auto-generated (or, optionally, empty) and compatible with compat (if specified).

Notes

It does not affect the result whether compat itself is an auto-generated alias; e.g., ‘foo’ and ‘foo@1’ are equivalent arguments.

isequal(at)

Method to determine if two ATs are the same. Useful for testing.

Parameters:

at : AT

The AT to compare against

Returns:

Boolean

Whether or not the ATs are equal

isstale()

Returns whether the AT is out of date.

Parameters:

None

Returns:

boolean

True if the AT is out of date, else False.

len2()

Returns the length of _bdp_in and _bdp_out in a tuple.

If you just want the _bdp_out, the intrinsic len(my_at) Thus my_at.len2()[1] is the same as len(my_at).

Returns:

Tuple

Tuple contains 2 values, the length of _bdp_in and _bdp_out

link()

Increments the task link count.

Returns:None

markChanged()

Mark an AT that it’s state was changed, so it would need to be rerun. The FlowManager will take care of any dependants that will need to be marked as well. Use “markUpToDate” if you want the opposite function.

Parameters:None Returns:None

markUpToDate()

Resets _stale to indicate that the AT does not need to be run.

Parameters:None Returns:None

merge(at, aliases=None)

Merges attributes from another task.

The purpose of this method is to facilitate re-running of flows by transferring select attributes from one task to a compatible replacement task. To maintain transparency, attributes are overwritten only where necessary to preserve flow integrity; in particular, items such as keyword values and the task ID are never modified.

Parameters:

at : ADMIT task

Task to merge with self.

aliases : 2-tuple of dict of str to set of str

Reserved alias registry, keyed by alias base name (index 0) or AT type (index 1). For index 0, the value is the set of aliases created with that base name; for index 1, it is the set of default aliases created for ATs of that type. In both cases, the duplicate aliases are appended with ‘@N’ for some integer N.

Returns:

None

Notes

It is the caller’s responsibility to set the alias as desired. Doing so locally (here) may result in unnecessary reassignments, causing tasks which are otherwise unaltered to be marked stale.

The task enabled/disabled status is always copied from the input task. This is necessary for variflow maintenance but applied uniformly for simplicity.

mkdir(dirname)

Make a directory in the ADMIT hierarchy. Checks whether the directory already exists. It also allows an absolute path, in the classical UNIX sense, but this is normally not needed.

Parameters:

dirname : str

Directory name.

Notes

Todo

this routine is identical to the one in Admit()

mkext(filename, ext, alias='')

Return a new filename with a new extension with optional ADMIT alias.

This will either append an extension to a (file)name without an extension, or replace one. If an alias was set by this task, the alias will replace the alias found in the basename dash.

You can also provide a local alias, instead of using the task alias.

Examples:

(“x” , “z” ) -> “x.z” (“x.y”, “z” ) -> “x.z” (“x.y”, “z”, “a”) -> “x-a.z”

newId(tid)

Assigns the task a new ID number.

Related attributes are updated accordingly.

Parameters:

tid : int

New task ID.

Returns:

int

Assigned ID number.

reset(a)

Performs an in-place shallow copy.

Parameters:

a : ADMIT Task

Task reference to copy from.

Returns:

None

run()

Runs the task.

Running a task transforms its input BDPs into output BDPs according to its function. Following this call the output BDPs accurately reflect the current state of the task (keywords and BDP inputs). Each concrete AT must implement its own run() method.

Returns:None

See also

execute

running(state=None)

Returns current task execution flag, with optional reset.

Parameters:

state : bool, optional

New value of enabled flag (default None keeps current value).

Returns:

bool

Whether task is currently being executed (prior to applying state, if provided).

save()

Save (write) any BDPs connected to this AT.

Parameters:None Returns:None

set(item, val)

Method to set protected attributes, rather than direct access

Parameters:

item : str

The name of the variable to set

val : varies

The value to set the attribute to

Returns:

None

setAlias(aliases, alias=None, auto=False)

Sets and registers the task alias, guaranteed unique among registered aliases.

Checks whether the input alias is already in use. If so, raises an exception unless the name appears auto-generated (or is empty), in which case the aliases is updated to ensure uniqueness. The latter are names appended with a string of the form ‘@N’, where N is some decimal integer value. Resets the task alias name to the final result.

Normally users should not call this function directly; user-defined aliases should be set via the AT constructor argument instead.

Parameters:

aliases : 2-tuple of dict of str to set of str

Reserved alias registry, keyed by alias base name (index 0) or AT type (index 1). For index 0, the value is the set of aliases created with that base name; for index 1, it is the set of default aliases created for ATs of that type. In both cases, the duplicate aliases are appended with ‘@N’ for some integer N.

alias : str, optional

The (suggested) task alias name; default None uses the current _alias value.

auto : bool, optional

Whether to automatically append a unique ‘@N’ suffix if the alias is already in use; otherwise, an exception will be raised (unless the alias already ends in ‘@N’, which implies auto = True).

Returns:

str

The assigned alias name.

Notes

Performs an iterative search for the first available name following the (estimated) final assigned alias for the AT of the given type.

Users may specify an alias of the form ‘foo@0’ to indicate that a sequence of aliases ‘foo@0’, ‘foo@1’, ... be auto-generated as necessary. This can be useful in scripts where tasks are generated in loops and a predictable series of aliases is desired.

setProject(pid)

Adds a project ID to task ID.

Multiflows involve adding tasks belonging to other projects into a single flow, the multiflow. To prevent task ID collisions, tasks in multiflows embed a project number into the task ID.

This method performs the actual embedding operation. The project ID is inserted beginning at bit 32 and replaces any previous project ID. Since Python integers are automatically multi-precision there is never danger of truncation of the project ID.

Parameters:

pid : int

Project ID number.

Returns:

None

See also

getProject

set_bdp_in(bdpin=[])

Validate the _valid_bdp_in list and digest it into the appropriate attributes.

Parameters:

bdpin : list

List containing tuples of the valid BDP inputs and their counts; defaults to empty list.

Returns:

None

set_bdp_out(bout=[])

Validate the _valid_bdp_out list and digest it into the appropriate attributes.

Parameters:

bout : list

List containing tuples of the valid BDP outputs and their counts; defaults to empty list.

Returns:

None

seteffectivelevel(level)

Method to set the effective logging level of the logging subsystem

Parameters:

level : int

The logging level to use

Returns:

None

setkey(name='', value='', isinit=False)

Set keyword value.

Two styles are possible:

1. name = {key:val} e.g. setkey({“a”:1})
2. name = “key”, value = val e.g. setkey(“a”, 1)

This method checks the type of the keyword value, as it must remain the same. Also new keywords cannot be added.

Parameters:

name : dictionary or string, optional

Dictionary of keyword value pairs, or a string containing a single key name; defaults to empty string.

value : any, optional

The (string) keyword value; defaults to empty string.

isinit : bool

Whether keyword is being set for the first time.

Returns:

None

setloggername(name)

Method to set the name of the logger for this AT instance

Parameters:

name : str

The name of the logger for this AT

Returns:

None

setlogginglevel(level)

Method to set the logging level

Parameters:

level : int

The logging level to use

Returns:

None

show()

Return the AT type.

Parameters:

None

Returns:

string

The type of the AT.

statusicons()

return some html icons representing the enabled/stale status of this task

summary()

Returns the summary dictionary from the AT, for merging into the ADMIT Summary object. Derived classes should override this if they provide summary data.

unlink()

Decrements the task link count.

Returns:None

userdata()

Returns the user dictionary from the AT, for merging into the ADMIT userdata object. Derived classes should override this if they provide user data.

validateinput(describe=False)

Method to validate the _bdp_in’s against a dictionary of expected types.

Parameters:

describe : bool, optional

Whether to include a list of what was found (or not) in the return value (as the second item in a tuple).

Returns:

bool

False if anything is amiss, else True.

validatekeys()

Method to error check all input keys. If an error is found an error message should be issued, not an exception. This allows for all values to be error checked before an ecpetion is raised. This method should return a bool where True means all key values are ok, and False otherwise. An exception will be raised by validatekeyvalues, which calls this method, if False is returned. The generic form of this method does not check anything, but should be overridden in individual AT’s that need to check key values.

Parameters:None Returns:bool, True if all key values are ok, False otherwise.

write(node)

Method to write the AT to disk.

Parameters:

node : elementtree node

The node to attach to (usually supplied by flowmanager)

Returns:

node : elementtree node

The modified node

dtd : Text string of the dtd