Visualizing Data Gaps

To look at the temporal distribution of the gaps, we can use this function.

network load_file("./data/ohio.network")
network csv_count_na("./data/ts/natural.csv", outattr = "nat_na")
network csv_data_blocks_svg(
	csvfile="./data/ts/natural.csv",
	outfile="./output/natural-blocks.svg",
	label="{_NAME} ({=(/ (st+num 'nat_na) 365.0):f(1)} yr)"
)
network clip()
network echo("../output/natural-blocks.svg")

Results:

network load_file("./data/ohio.network")
network csv_count_na("./data/ts/observed.csv", outattr = "obs_na")
network csv_data_blocks_svg(
	csvfile="./data/ts/observed.csv",
	outfile="./output/observed-blocks.svg",
	label="{_NAME} ({obs_na})"
)
network clip()
network echo("../output/observed-blocks.svg")

Results:

Internal Plugins

There are some plugins that are provided with the nadi_core library. They are part of the library, so users can directly use them.

For example in the following tasks file, the functions that are highlighted are functions available from the core plugins. Other functions need to be loaded from plugins.

# sample .tasks file which is like a script with functions
node print_attrs("uniqueID")
node show_node()
network save_graphviz("/tmp/test.gv")

node[WV04113,WV04112,WV04112] print_attr_toml("testattr2")
node render("{NAME} {uniqueID} {_Dam_Height_(Ft)?}")
node list_attr("; ")
# some functions can take variable number of inputs
network calc_attr_errors(
    "Dam_Height_(Ft)",
    "Hydraulic_Height_(Ft)",
    "rmse", "nse", "abserr"
)
node sum_safe("Latitude")
node.inputsfirst render("Hi {SUM_ATTR}")
# multiple line for function arguments
network save_table(
	"test.table",
	"/tmp/test.tex",
	true,
	radius=0.2,
	start = 2012-19-20,
	end = 2012-19-23 12:04
	)
node set_attrs(testattr = 2)
node set_attrs_render(testattr2 = "{testattr:calc(+2)}")
node[WV04112] render("{testattr} {testattr2}")

# here we use a complicated template that can do basic logic handling
node set_attrs_render(
    testattr2 = "=(if (and (st+has 'Latitude) (> (st+num 'Latitude) 39)) 'true 'false)"
)
# same thing can be done if you need more flexibility in variable names
node load_toml_string(
    "testattr2 = =(if (and (st+has 'Latitude) (> (st+num 'Latitude) 39)) 'true 'false)"
)
# selecting a list of nodes to run a function
node[
	# comment here?
    WV04113,
    WV04112
] print_attr_toml("testattr2")
# selecting a path
node[WV04112 -> WV04113] render("=(> 2 3)")

Env Functions

strmap

env attrs.strmap(
    attr: '& str',
    attrmap: '& AttrMap',
    default: 'Option < Attribute >'
)

Arguments

• attr: '& str' => Value to transform the attribute
• attrmap: '& AttrMap' => Dictionary of key=value to map the data to
• default: 'Option < Attribute >' => Default value if key not found in attrmap

map values from the attribute based on the given table

float_transform

env attrs.float_transform(value: 'f64', transformation: '& str')

Arguments

• value: 'f64' => value to transform
• transformation: '& str' => transformation function, can be one of log/log10/sqrt

map values from the attribute based on the given table

Node Functions

load_attrs

node attrs.load_attrs(filename: 'PathBuf')

Arguments

• filename: 'PathBuf' => Template for the filename to load node attributes from

Loads attrs from file for all nodes based on the given template

Arguments

• filename: Template for the filename to load node attributes from
• verbose: print verbose message

The template will be rendered for each node, and that filename from the rendered template will be used to load the attributes.

Errors

The function will error out in following conditions:

• Template for filename is not given,
• The template couldn’t be rendered,
• There was error loading attributes from the file.

print_all_attrs

node attrs.print_all_attrs()

Arguments

Print all attrs in a node

No arguments and no errors, it’ll just print all the attributes in a node with node::attr=val format, where,

• node is node name
• attr is attribute name
• val is attribute value (string representation)

print_attrs

node attrs.print_attrs(*attrs, name: 'bool')

Arguments

• *attrs =>
• name: 'bool' =>

Print the given node attributes if present

Arguments

• attrs,… : list of attributes to print
• name: Bool for whether to show the node name or not

Error

The function will error if

• list of arguments are not String
• the name argument is not Boolean

The attributes will be printed in key=val format.

set_attrs

node attrs.set_attrs(**attrs)

Arguments

• **attrs => Key value pairs of the attributes to set

Set node attributes

Use this function to set the node attributes of all nodes, or a select few nodes using the node selection methods (path or list of nodes)

Error

The function should not error.

Example

Following will set the attribute a2d to true for all nodes from A to D

node[A -> D] set_attrs(a2d = true)

get_attr

node attrs.get_attr(attr: '& str', default: 'Option < Attribute >')

Arguments

• attr: '& str' => Name of the attribute to get
• default: 'Option < Attribute >' => Default value if the attribute is not found

Retrive attribute

has_attr

node attrs.has_attr(attr: '& str')

Arguments

• attr: '& str' => Name of the attribute to check

Check if the attribute is present

first_attr

node attrs.first_attr(attrs: '& [String]', default: 'Option < Attribute >')

Arguments

• attrs: '& [String]' => attribute names
• default: 'Option < Attribute >' => Default value if not found

Return the first Attribute that exists

set_attrs_ifelse

node attrs.set_attrs_ifelse(cond: 'bool', **values)

Arguments

• cond: 'bool' => Condition to check
• **values => key = [val1, val2] where key is set as first if cond is true else second

if else condition with multiple attributes

set_attrs_render

node attrs.set_attrs_render(**kwargs)

Arguments

• **kwargs => key value pair of attribute to set and the Template to render

Set node attributes based on string templates

load_toml_render

node attrs.load_toml_render(toml: '& Template', echo: 'bool' = false)

Arguments

• toml: '& Template' => String template to render and load as TOML string
• echo: 'bool' = false => Print the rendered toml or not

Set node attributes based on string templates

Network Functions

set_attrs

network attrs.set_attrs(**attrs)

Arguments

• **attrs => key value pair of attributes to set

Set network attributes

Arguments

• key=value - Kwargs of attr = value

set_attrs_render

network attrs.set_attrs_render(**kwargs)

Arguments

• **kwargs => Kwargs of attr = String template to render

Set network attributes based on string templates

Env Functions

exists

env command.exists(path: 'PathBuf', min_lines: 'Option < usize >')

Arguments

• path: 'PathBuf' => Path to check
• min_lines: 'Option < usize >' => Minimum number of lines the file should have

Checks if the given path exists

Node Functions

exists

node command.exists(path: 'Template', min_lines: 'Option < usize >')

Arguments

• path: 'Template' => Path to check
• min_lines: 'Option < usize >' => Minimum number of lines the file should have

Checks if the given path exists when rendering the template

command

node command.command(
    cmd: '& Template',
    verbose: 'bool' = true,
    echo: 'bool' = false
)

Arguments

• cmd: '& Template' => String Command template to run
• verbose: 'bool' = true => Show the rendered version of command, and other messages
• echo: 'bool' = false => Echo the stdout from the command

Run the given template as a shell command.

Run any command in the shell. The standard output of the command will be consumed and if there are lines starting with nadi:var: and followed by key=val pairs, it’ll be read as new attributes to that node.

For example if a command writes nadi:var:name="Joe" to stdout, then the for the current node the command is being run for, name attribute will be set to Joe. This way, you can write your scripts in any language and pass the values back to the NADI system.

It will also print out the new values or changes from old values, if verbose is true.

Errors

The function will error if,

• The command template cannot be rendered,
• The command cannot be executed,
• The attributes from command’s stdout cannot be parsed properly

run

node command.run(
    command: '& str',
    inputs: '& str',
    outputs: '& str',
    verbose: 'bool' = true,
    echo: 'bool' = false
)

Arguments

• command: '& str' => Node Attribute with the command to run
• inputs: '& str' => Node attribute with list of input files
• outputs: '& str' => Node attribute with list of output files
• verbose: 'bool' = true => Print the command being run
• echo: 'bool' = false => Show the output of the command

Run the node as if it’s a command if inputs are changed

This function will not run a command node if all outputs are older than all inputs. This is useful to networks where each nodes are tasks with input files and output files.

Network Functions

parallel

network command.parallel(
    cmd: '& Template',
    _workers: 'i64' = 4,
    verbose: 'bool' = true,
    echo: 'bool' = false
)

Arguments

• cmd: '& Template' => String Command template to run
• _workers: 'i64' = 4 => Number of workers to run in parallel
• verbose: 'bool' = true => Print the command being run
• echo: 'bool' = false => Show the output of the command

Run the given template as a shell command for each nodes in the network in parallel.

Warning

Currently there is no way to limit the number of parallel processes, so please be careful with this command if you have very large number of nodes.

command

network command.command(
    cmd: 'Template',
    verbose: 'bool' = true,
    echo: 'bool' = false
)

Arguments

• cmd: 'Template' => String Command template to run
• verbose: 'bool' = true => Print the command being run
• echo: 'bool' = false => Show the output of the command

Run the given template as a shell command.

Run any command in the shell. The standard output of the command will be consumed and if there are lines starting with nadi:var: and followed by key=val pairs, it’ll be read as new attributes to that node.

See node command.command for more details as they have the same implementation

Network Functions

load_file

network connections.load_file(file: 'PathBuf', append: 'bool' = false)

Arguments

• file: 'PathBuf' => File to load the network connections from
• append: 'bool' = false => Append the connections in the current network

Load the given file into the network

This replaces the current network with the one loaded from the file.

subset

network connections.subset(keep: 'bool' = true)

Arguments

• keep: 'bool' = true => Keep the selected nodes (false = removes the selected)

Take a subset of network by only including the selected nodes

save_file

network connections.save_file(
    file: 'PathBuf',
    quote_all: 'bool' = true,
    graphviz: 'bool' = false
)

Arguments

• file: 'PathBuf' => Path to the output file
• quote_all: 'bool' = true => quote all node names; if false, doesn’t quote valid identifier names
• graphviz: 'bool' = false => wrap the network into a valid graphviz file

Save the network into the given file

For more control on graphviz file writing use save_graphviz from graphviz plugin instead.

Env Functions

type_name

env core.type_name(value: 'Attribute', recursive: 'bool' = false)

Arguments

• value: 'Attribute' => Argument to get type
• recursive: 'bool' = false => Recursively check types for array and table

Type name of the arguments

float

env core.float(value: 'Attribute', parse: 'bool' = true)

Arguments

• value: 'Attribute' => Argument to convert to float
• parse: 'bool' = true => parse string to float

make a float from value

str

env core.str(value: 'Attribute', quote: 'bool' = false)

Arguments

• value: 'Attribute' => Argument to convert to float
• quote: 'bool' = false => quote it if it’s literal string

make a string from value

int

env core.int(
    value: 'Attribute',
    parse: 'bool' = true,
    round: 'bool' = true,
    strfloat: 'bool' = false
)

Arguments

• value: 'Attribute' => Argument to convert to int
• parse: 'bool' = true => parse string to int
• round: 'bool' = true => round float into integer
• strfloat: 'bool' = false => parse string first as float before converting to int

make an int from the value

array

env core.array(*attributes)

Arguments

• *attributes => List of attributes

make an array from the arguments

attrmap

env core.attrmap(**attributes)

Arguments

• **attributes => name and values of attributes

make an array from the arguments

Network Functions

count

network core.count()

Arguments

Count the number of nodes in the network

If propagation is present, only counts those nodes

Network Functions

debug

network debug.debug(*args, **kwargs)

Arguments

• *args => Function arguments
• **kwargs => Function Keyword arguments

Print the args and kwargs on this function

This function will just print out the args and kwargs the function is called with. This is for debugging purposes to see if the args/kwargs are identified properly. And can also be used to see how the nadi system takes the input from the function call.

echo

network debug.echo(
    line: 'String',
    error: 'bool' = false,
    newline: 'bool' = true
)

Arguments

• line: 'String' => line to print
• error: 'bool' = false => print to stderr instead of stdout
• newline: 'bool' = true => print newline at the end

Echo the string to stdout or stderr

This simply echoes anything given to it. This can be used in combination with nadi tasks that create files (image, text, etc). The echo function can be called to get the link to those files back to the stdout.

Also useful for nadi preprocessor.

clip

network debug.clip(error: 'bool' = false)

Arguments

• error: 'bool' = false => print in stderr instead of in stdout

Echo the ––8<–– line for clipping syntax

This function is a utility function for the generation of nadi book. This prints out the ----8<---- line when called, so that mdbook preprocessor for nadi knows where to clip the output for displaying it in the book.

This makes it easier to only show the relevant parts of the output in the documentation instead of having the user see output of other unrelated parts which are necessary for generating the results.

Example

Given the following tasks file:

net load_file("...")
net load_attrs("...")
net clip()
net render("{_NAME} {attr1}")

The clip function’s output will let the preprocessor know that only the parts after that are relevant to the user. Hence, it’ll discard outputs before that during documentation generation.

Env Functions

ifelse

env logic.ifelse(
    cond: 'bool',
    iftrue: 'Attribute',
    iffalse: 'Attribute'
)

Arguments

• cond: 'bool' => Attribute that can be cast to bool value
• iftrue: 'Attribute' => Output if cond is true
• iffalse: 'Attribute' => Output if cond is false

Simple if else condition

gt

env logic.gt(a: '& Attribute', b: '& Attribute')

Arguments

• a: '& Attribute' => first attribute
• b: '& Attribute' => second attribute

Greater than check

lt

env logic.lt(a: '& Attribute', b: '& Attribute')

Arguments

• a: '& Attribute' => first attribute
• b: '& Attribute' => second attribute

Greater than check

eq

env logic.eq(a: '& Attribute', b: '& Attribute')

Arguments

• a: '& Attribute' => first attribute
• b: '& Attribute' => second attribute

Greater than check

and

env logic.and(*conds)

Arguments

• *conds => List of attributes that can be cast to bool

Boolean and

or

env logic.or(*conds)

Arguments

• *conds => List of attributes that can be cast to bool

boolean or

not

env logic.not(cond: 'bool')

Arguments

• cond: 'bool' => attribute that can be cast to bool

boolean not

Env Functions

str_match

env regex.str_match(pattern: 'Regex', attr: '& str')

Arguments

• pattern: 'Regex' => Regex pattern to match
• attr: '& str' => attribute to check for pattern

Check if the given pattern matches the value or not

str_replace

env regex.str_replace(
    pattern: 'Regex',
    attr: '& str',
    rep: '& str'
)

Arguments

• pattern: 'Regex' => Regex pattern to match
• attr: '& str' => attribute to replace
• rep: '& str' => replacement string

Replace the occurances of the given match

str_find

env regex.str_find(pattern: 'Regex', attr: '& str')

Arguments

• pattern: 'Regex' => Regex pattern to match
• attr: '& str' => attribute to check for pattern

Find the given pattern in the value

str_find_all

env regex.str_find_all(pattern: 'Regex', attr: '& str')

Arguments

• pattern: 'Regex' => Regex pattern to match
• attr: '& str' => attribute to check for pattern

Find all the matches of the given pattern in the value

str_count

env regex.str_count(pattern: 'Regex', attr: '& str')

Arguments

• pattern: 'Regex' => Regex pattern to match
• attr: '& str' => attribute to check for pattern

Count the number of matches of given pattern in the value

Node Functions

render

node render.render(template: '& Template', safe: 'bool' = false)

Arguments

• template: '& Template' => String template to render
• safe: 'bool' = false => if render fails keep it as it is instead of exiting

Render the template based on the node attributes

For more details on the template system. Refer to the String Template section of the NADI book.

Network Functions

render

network render.render(template: 'PathBuf', outfile: 'Option < PathBuf >')

Arguments

• template: 'PathBuf' => Path to the template file
• outfile: 'Option < PathBuf >' => output file

Render a File template for the nodes in the whole network

Write the file with templates for input variables in the same way you write string templates. It’s useful for markdown files, as the curly braces syntax won’t be used for anything else that way. Do be careful about that. And the program will replace those templates with their values when you run it with inputs.

It’ll repeat the same template for each node and render them. If you want only a portion of the file repeated for nodes inclose them with lines with ---8<--- on both start and the end. The lines containing the clip syntax will be ignored, ideally you can put them in comments.

You can also use ---include:<filename>[::line_range] syntax to include a file, the line_range syntax, if present, should be in the form of start[:increment]:end, you can exclude start or end to denote the line 1 or last line (e.g. :5 is 1:5, and 3: is from line 3 to the end)

Arguments

• template: Path to the template file
• outfile [Optional]: Path to save the template file, if none it’ll be printed in stdout

Node Functions

sr_count

node series.sr_count()

Arguments

Number of series in the node

sr_list

node series.sr_list()

Arguments

List all series in the node

sr_dtype

node series.sr_dtype(name: '& str', safe: 'bool' = false)

Arguments

• name: '& str' => Name of the series
• safe: 'bool' = false => Do not error if series does’t exist

Type name of the series

sr_len

node series.sr_len(name: '& str', safe: 'bool' = false)

Arguments

• name: '& str' => Name of the series
• safe: 'bool' = false => Do not error if series does’t exist

Length of the series

sr_mean

node series.sr_mean(name: '& str')

Arguments

• name: '& str' => Name of the series

Type name of the series

sr_sum

node series.sr_sum(name: '& str')

Arguments

• name: '& str' => Name of the series

Sum of the series

set_series

node series.set_series(
    name: '& str',
    value: 'Attribute',
    dtype: '& str'
)

Arguments

• name: '& str' => Name of the series to save as
• value: 'Attribute' => Argument to convert to series
• dtype: '& str' => type

set the following series to the node

sr_to_array

node series.sr_to_array(name: '& str', safe: 'bool' = false)

Arguments

• name: '& str' => Name of the series
• safe: 'bool' = false => Do not error if series does’t exist

Make an array from the series

Network Functions

save_csv

network table.save_csv(path: '& Path', *fields)

Arguments

• path: '& Path' =>
• *fields =>

Save CSV

table_to_markdown

network table.table_to_markdown(
    table: 'Option < PathBuf >',
    template: 'Option < String >',
    outfile: 'Option < PathBuf >',
    connections: 'Option < String >'
)

Arguments

• table: 'Option < PathBuf >' => Path to the table file
• template: 'Option < String >' => String template for table
• outfile: 'Option < PathBuf >' => Path to the output file
• connections: 'Option < String >' => Show connections column or not

Render the Table as a rendered markdown

Error

The function will error out if,

• error reading the table file,
• error parsing table template,
• neither one of table file or table template is provided,
• error while rendering markdown (caused by error on rendering cell values from templates)
• error while writing to the output file

Node Functions

ts_count

node timeseries.ts_count()

Arguments

Number of timeseries in the node

ts_list

node timeseries.ts_list()

Arguments

List all timeseries in the node

ts_dtype

node timeseries.ts_dtype(name: '& str', safe: 'bool' = false)

Arguments

• name: '& str' => Name of the timeseries
• safe: 'bool' = false => Do not error if timeseries does’t exist

Type name of the timeseries

ts_len

node timeseries.ts_len(name: '& str', safe: 'bool' = false)

Arguments

• name: '& str' => Name of the timeseries
• safe: 'bool' = false => Do not error if timeseries does’t exist

Length of the timeseries

ts_print

node timeseries.ts_print(
    name: '& String',
    header: 'bool' = true,
    head: 'Option < i64 >'
)

Arguments

• name: '& String' => name of the timeseries
• header: 'bool' = true => show header
• head: 'Option < i64 >' => number of head rows to show (all by default)

Print the given timeseries values in csv format

TODO

• save to file instead of showing with outfile: Option<PathBuf>

Network Functions

ts_print_csv

network timeseries.ts_print_csv(
    name: 'String',
    head: 'Option < usize >',
    nodes: 'Option < HashSet < String > >'
)

Arguments

• name: 'String' => Name of the timeseries to save
• head: 'Option < usize >' => number of head rows to show (all by default)
• nodes: 'Option < HashSet < String > >' => Include only these nodes (all by default)

Save timeseries from all nodes into a single csv file

TODO: error/not on unqual length TODO: error/not on no timeseries, etc… TODO: output to file: PathBuf

Network Functions

set_nodesize_attrs

network visuals.set_nodesize_attrs(
    attr: 'String',
    default: 'Option < f64 >',
    minsize: 'f64' = 4.0,
    maxsize: 'f64' = 12.0
)

Arguments

• attr: 'String' => Attribute to use for size scaling
• default: 'Option < f64 >' => default value of the attribute if not found
• minsize: 'f64' = 4.0 => minimum size of the node
• maxsize: 'f64' = 12.0 => maximum size of the node

Set the node size of the nodes based on the attribute value

External Plugins

This section showcases the functions from external plugins developed along side the NADI project due to various reasons.

The plugins listed here can be installed with following steps:

• clone the repository of external plugins,
• compile it locally with cargo,
• move all generated dynamic libraries to the nadi plugin directory.

Node Functions

count_node_if

node dams.count_node_if(count_attr: '& str', cond: 'bool')

Arguments

• count_attr: '& str' =>
• cond: 'bool' =>

Count the number of nodes upstream at each point that satisfies a certain condition

min_year

node dams.min_year(yearattr: '& str', write_var: '& str' = "MIN_YEAR")

Arguments

• yearattr: '& str' =>
• write_var: '& str' = "MIN_YEAR" =>

Propagate the minimum year downstream

Node Functions

load_csv_fill

node datafill.load_csv_fill(
    name: 'String',
    file: 'Template',
    columns: '(String, String)',
    timefmt: 'String',
    method: 'DataFillMethod' = Linear,
    dtype: 'String' = "Floats"
)

Arguments

• name: 'String' => Name of the timeseries
• file: 'Template' => Template of the CSV file for the nodes
• columns: '(String, String)' => Names of date column and value column
• timefmt: 'String' => date time format, if you only have date, but have time on format string, it will panic
• method: 'DataFillMethod' = Linear => Method to use for data filling: forward/backward/linear
• dtype: 'String' = "Floats" => DataType to load into timeseries

Node Functions

calc_ts_error

node errors.calc_ts_error(
    ts1: '& str',
    ts2: '& str',
    error: '& str' = "rmse"
)

Arguments

• ts1: '& str' => Timeseries value to use as actual value
• ts2: '& str' => Timeseries value to be used to calculate the error
• error: '& str' = "rmse" => Error type, one of rmse/nrmse/abserr/nse

Calculate Error from two timeseries values in the node

It calculates the error between two timeseries values from the node

calc_ts_errors

node errors.calc_ts_errors(
    ts1: '& String',
    ts2: '& String',
    errors: '& [String]'
)

Arguments

• ts1: '& String' => Timeseries value to use as actual value
• ts2: '& String' => Timeseries value to be used to calculate the error
• errors: '& [String]' => Error types to calculate, one of rmse/nrmse/abserr/nse

Calculate Error from two timeseries values in the node

It calculates the error between two timeseries values from the node.

Network Functions

calc_attr_error

network errors.calc_attr_error(
    attr1: 'String',
    attr2: 'String',
    error: 'String' = "rmse"
)

Arguments

• attr1: 'String' => Attribute value to use as actual value
• attr2: 'String' => Attribute value to be used to calculate the error
• error: 'String' = "rmse" => Error type, one of rmse/nrmse/abserr/nse

Calculate Error from two attribute values in the network

It calculates the error using two attribute values from all the nodes.

Network Functions

fancy_print

network fancy_print.fancy_print()

Arguments

Fancy print a network

Network Functions

plot_timeseries

network gnuplot.plot_timeseries(
    csvfile: 'Template',
    datecol: '& str',
    datacol: '& str',
    outfile: '& Path',
    timefmt: '& str' = "%Y-%m-%d",
    config: '& GnuplotConfig' = GnuplotConfig { outfile: None, terminal: None, csv: false, preamble: "" },
    skip_missing: 'bool' = false
)

Arguments

• csvfile: 'Template' =>
• datecol: '& str' =>
• datacol: '& str' =>
• outfile: '& Path' =>
• timefmt: '& str' = "%Y-%m-%d" =>
• config: '& GnuplotConfig' = GnuplotConfig { outfile: None, terminal: None, csv: false, preamble: "" } =>
• skip_missing: 'bool' = false =>

Generate a gnuplot file that plots the timeseries data in the network

Node Functions

attr_fraction_svg

node graphics.attr_fraction_svg(
    attr: '& str',
    outfile: '& Template',
    color: '& AttrColor',
    height: 'f64' = 80.0,
    width: 'f64' = 80.0,
    margin: 'f64' = 10.0
)

Arguments

• attr: '& str' =>
• outfile: '& Template' =>
• color: '& AttrColor' =>
• height: 'f64' = 80.0 =>
• width: 'f64' = 80.0 =>
• margin: 'f64' = 10.0 =>

Create a SVG file with the given network structure

Network Functions

csv_load_ts

network graphics.csv_load_ts(
    file: 'PathBuf',
    name: 'String',
    date_col: 'String' = "date",
    timefmt: 'String' = "%Y-%m-%d",
    data_type: 'String' = "Floats"
)

Arguments

• file: 'PathBuf' =>
• name: 'String' =>
• date_col: 'String' = "date" =>
• timefmt: 'String' = "%Y-%m-%d" =>
• data_type: 'String' = "Floats" =>

Count the number of na values in CSV file for each nodes in a network

Arguments

• file: Input CSV file path to read (should have column with node names for all nodes)
• name: Name of the timeseries
• date_col: Date Column name
• timefmt: date time format, if you only have date, but have time on format string, it will panic
• data_type: Type of the data to cast into

csv_count_na

network graphics.csv_count_na(
    file: 'PathBuf',
    outattr: 'Option < String >',
    sort: 'bool' = false,
    skip_zero: 'bool' = false,
    head: 'Option < i64 >'
)

Arguments

• file: 'PathBuf' =>
• outattr: 'Option < String >' =>
• sort: 'bool' = false =>
• skip_zero: 'bool' = false =>
• head: 'Option < i64 >' =>

Count the number of na values in CSV file for each nodes in a network

Arguments

• file: Input CSV file path to read (should have column with node names for all nodes)
• outattr: Output attribute to save the count of NA to. If empty print to stdout
• sort: show the nodes with larger gaps on top, only applicable while printing
• head: at max show only this number of nodes
• skip_zero: skip nodes with zero missing numbers

csv_data_blocks_svg

network graphics.csv_data_blocks_svg(
    csvfile: 'PathBuf',
    outfile: 'PathBuf',
    label: 'Template',
    date_col: 'String' = "date",
    config: 'NetworkPlotConfig' = NetworkPlotConfig { width: 250.0, height: 300.0, delta_x: 20.0, delta_y: 20.0, offset: 30.0, radius: 3.0, fontsize: 16.0, fontface: FontFace { inner: Shared { inner: 0x5747b6ea6a20 } } },
    blocks_width: 'f64' = 500.0,
    fit: 'bool' = false
)

Arguments

• csvfile: 'PathBuf' =>
• outfile: 'PathBuf' =>
• label: 'Template' =>
• date_col: 'String' = "date" =>
• config: 'NetworkPlotConfig' = NetworkPlotConfig { width: 250.0, height: 300.0, delta_x: 20.0, delta_y: 20.0, offset: 30.0, radius: 3.0, fontsize: 16.0, fontface: FontFace { inner: Shared { inner: 0x5747b6ea6a20 } } } =>
• blocks_width: 'f64' = 500.0 =>
• fit: 'bool' = false =>

Draw the data blocks with arrows in timeline

export_svg

network graphics.export_svg(
    outfile: 'PathBuf',
    config: 'NetworkPlotConfig' = NetworkPlotConfig { width: 250.0, height: 300.0, delta_x: 20.0, delta_y: 20.0, offset: 30.0, radius: 3.0, fontsize: 16.0, fontface: FontFace { inner: Shared { inner: 0x5747b6ea6a20 } } },
    fit: 'bool' = false,
    label: 'Option < Template >',
    highlight: '& [usize]' = []
)

Arguments

• outfile: 'PathBuf' =>
• config: 'NetworkPlotConfig' = NetworkPlotConfig { width: 250.0, height: 300.0, delta_x: 20.0, delta_y: 20.0, offset: 30.0, radius: 3.0, fontsize: 16.0, fontface: FontFace { inner: Shared { inner: 0x5747b6ea6a20 } } } =>
• fit: 'bool' = false =>
• label: 'Option < Template >' =>
• highlight: '& [usize]' = [] =>

Create a SVG file with the given network structure

table_to_svg

network graphics.table_to_svg(
    outfile: 'PathBuf',
    table: 'Option < PathBuf >',
    template: 'Option < String >',
    config: 'NetworkPlotConfig' = NetworkPlotConfig { width: 250.0, height: 300.0, delta_x: 20.0, delta_y: 20.0, offset: 30.0, radius: 3.0, fontsize: 16.0, fontface: FontFace { inner: Shared { inner: 0x5747b6ea6a20 } } },
    fit: 'bool' = false,
    highlight: '& [String]' = []
)

Arguments

• outfile: 'PathBuf' =>
• table: 'Option < PathBuf >' =>
• template: 'Option < String >' =>
• config: 'NetworkPlotConfig' = NetworkPlotConfig { width: 250.0, height: 300.0, delta_x: 20.0, delta_y: 20.0, offset: 30.0, radius: 3.0, fontsize: 16.0, fontface: FontFace { inner: Shared { inner: 0x5747b6ea6a20 } } } =>
• fit: 'bool' = false =>
• highlight: '& [String]' = [] =>

Create a SVG file with the given network structure

Network Functions

save_graphviz

network graphviz.save_graphviz(
    outfile: '& Path',
    name: '& str' = "network",
    global_attrs: '& str' = "",
    node_attr: 'Option < & Template >',
    edge_attr: 'Option < & Template >'
)

Arguments

• outfile: '& Path' =>
• name: '& str' = "network" =>
• global_attrs: '& str' = "" =>
• node_attr: 'Option < & Template >' =>
• edge_attr: 'Option < & Template >' =>

Save the network as a graphviz file

Arguments:

• outfile - Path to the output file
• name - Name of the graph

Network Functions

export_map

network html.export_map(
    outfile: '& Path',
    template: 'Template',
    pagetitle: '& str' = "NADI Network",
    nodetitle: 'Template' = Template { original: "{_NAME}", parts: [Var("_NAME", "")] },
    connections: 'bool' = true
)

Arguments

• outfile: '& Path' =>
• template: 'Template' =>
• pagetitle: '& str' = "NADI Network" =>
• nodetitle: 'Template' = Template { original: "{_NAME}", parts: [Var("_NAME", "")] } =>
• connections: 'bool' = true =>

Exports the network as a HTML map

Network Functions

gis_load_attrs

network nadi_gis.gis_load_attrs(
    file: 'PathBuf',
    node: 'String',
    layer: 'Option < String >',
    geometry: 'String' = "GEOM",
    ignore: 'String' = "",
    sanitize: 'bool' = true,
    err_no_node: 'bool' = false
)

Arguments

• file: 'PathBuf' => GIS file to load (can be any format GDAL can understand)
• node: 'String' => Field in the GIS file corresponding to node name
• layer: 'Option < String >' => layer of the GIS file, first one picked by default
• geometry: 'String' = "GEOM" => Attribute to save the GIS geometry in
• ignore: 'String' = "" => Field names separated by comma, to ignore
• sanitize: 'bool' = true => sanitize the name of the fields
• err_no_node: 'bool' = false => Error if all nodes are not found in the GIS file

Load node attributes from a GIS file

The function reads a GIS file in any format (CSV, GPKG, SHP, JSON, etc) and loads their fields as attributes to the nodes.

gis_save_connections

network nadi_gis.gis_save_connections(
    file: 'PathBuf',
    geometry: 'String',
    driver: 'Option < String >',
    layer: 'String' = "network"
)

Arguments

• file: 'PathBuf' =>
• geometry: 'String' =>
• driver: 'Option < String >' =>
• layer: 'String' = "network" =>

Save GIS file of the connections

gis_save_nodes

network nadi_gis.gis_save_nodes(
    file: 'PathBuf',
    geometry: 'String',
    attrs: 'HashMap < String, String >' = {},
    driver: 'Option < String >',
    layer: 'String' = "nodes"
)

Arguments

• file: 'PathBuf' =>
• geometry: 'String' =>
• attrs: 'HashMap < String, String >' = {} =>
• driver: 'Option < String >' =>
• layer: 'String' = "nodes" =>

Save GIS file of the nodes

Node Functions

print_node

node print_node.print_node()

Arguments

Print the node with its inputs and outputs

Network Functions

print_attr_csv

network print_node.print_attr_csv(*args)

Arguments

• *args =>

Print the given attributes in csv format with first column with node name

Node Functions

check_negative

node streamflow.check_negative(ts_name: '& str')

Arguments

• ts_name: '& str' =>

Check the given streamflow timeseries for negative values

Data Structure

This section will describe the data structures associated with NADI system in brief.

For more accurate and upto date details on the data structures and their available methods. Look at the API reference of nadi_core on docs.rs.

Node

Points with attributes and timeseries. These can be any point as long as they’ll be on the network and connection to each other.

The attributes can be any format. There is a special type of attribute timeseries to deal with timeseries data that has been provided by the system. But users are free to make their own attributes and plugins + functions that can work with those attributes.

Since attributes are loaded using TOML file, simple attributes can be stored and parsed from strings, moderately complex ones can be saved as a combination of array and tables, and more complex ones can be saved in different files and their path can be stored as node attributes.

Here is an example node attribute file. Here we have string, float, int and boolean values, as well as a example csv timeseries

stn="smithland"
nat_7q10=12335.94850131619
orsanco_7q10=16900
lock=true

[ts.csv]
streamflow = {path="data/smithland.csv", datetime="date", data="flow"}

Network

Collection of Nodes, with Connection information. The connection information is saved in the nodes itself (=inputs= and =output= variables), but they are assigned from the network.

The nadi system (lit, river system), is designed for the connections between points along a river. Out of different types of river networks possible, it can only handle non-branching tributaries system, where each point can have zero to multiple inputs, but can only have one output. Overall the system should have a single output point. There can be branches in the river itself in the physical sense as long as they converse before the next point of interests. There cannot be node points that have more than one path to reach another node in the representative system.

Network file are simple text files with each edge on one line. Node names can be words with alphanumeric characters with the additional character _, similar to how rust identifiers work. The Node names can also be quoted strings, in those cases any characters are supported inside the quotes.

Here is an example network file,

cannelton -> newburgh
newburgh -> evansville
evansville -> "jt-myers"
# comments are supported
"jt-myers" -> "old-shawneetown"
"old-shawneetown" -> golconda
markland -> mcalpine
golconda -> smithland

Drawing it out:

network load_file("./data/mississippi.net")
network export_svg(
   "./output/mississippi.svg",
	label="[{INDEX}] {_NAME:repl(-, ):case(title)}"
)
network clip()
# the link path needs to be relative to this file
network echo("../output/mississippi.svg")

Results:

The program also plans to support the connection import from the DOT format (graphviz package).

Network file without any connection format can be written as a node per line, but those network can only call sequential functions, and not input dependent ones.

Depending on the use cases, it can probably be applied to other systems that are similar to a river system. Or even without the connection information, the functions that are independent to each other can be run in sequential order.

Timeseries

Timeseries of values, at regular interval. Can support integers, floats, booleans, strings, Arrays and Tables.

For timeseries that are not in a format that NADI can understand. The path to the timeseries can be provided as a node attribute and plugin functions can be written to use that path to load the timeseries for the node.

String Templates

The templating system will be used by an external library developed by me. The library can be modified if there are specific needs for this project.

The template system is feature rich, allowing for formatting, simple string transformations, and airthmatic calculations based on the variables (node attributes in this case). This can be used to generate file paths, and similar strings based on node attributes, as well as to format the cell values for exported table, figures, etc.

The template library is also available for Rust, C and C++, but all the interactions with the templates will be done through the nadi interface, so that is not required.

Documentations on the template system, can be redirected to the string_template_plus library page.

Brief explanation on the template system is given below.

Template Parts

Templates have variables, time formats, expressions, and commands (disabled by default);

Hi, my name is {name}, my address is {address?"N/A"}.
Current time is {%H} hour  {%M} minutes.

Results (with: name=John; address=123 Road, USA):

Hi, my name is John, my address is 123 Road, USA.
Current time is 21 hour  56 minutes.

Optional Variables

Variables can be chained in an optional way, so the first one that’s found will be used (e.g. {nickname?name} will render nickname if it’s present, else name);

Hi, I am {nickname?name}, my address is {address?"N/A"}.

Results (with: name=John; nickname=J; address=123 Road, USA):

Hi, I am J, my address is 123 Road, USA.

String Literal

Variables when replaced with literal strings (quoted strings), they will be used directly {address?"N/A"} will render N/A is address is not present;

Hi, I am {nickname?name}, my address is {address?"N/A"}.

Results (with: name=John):

Hi, I am John, my address is N/A.

Transformers

Variables can have optional transformers which transform the string based on their rules, (e.g. float transformer will truncate the float, upcase will make the string UPPERCASE, etc.);

Hi, I am {nickname?name:case(up)}, my address is {address?"N/A"}.

Results (with: name=Joe):

Hi, I am JOE, my address is N/A.

Time formats

time formats are formatted current time (e.g. {%Y} will become 2024 as of now);

Today is {%B %d} of the year {%Y}.

Results (with: name=John):

Today is January 27 of the year 2025.

Lisp Expressions

expressions are lisp expressions that will be evaluated and the results will be used. The lisp expression can also access any variables and do any supported programming. (e.g. (+ 1 1) in lisp will become 2);

guess my age(x) if: (x + 21) * 4 = =(* (+ (st+num 'age) 21) 4).

Results (with: age=20):

guess my age(x) if: (x + 21) * 4 = 164.

NADI Specific options

Besides the above points, specific to nadi system, any node template will have all the variables from node attributes available as strings for template. For string variables, their name can be used to access quoted string format, while their name with underscore prefix will be unquoted raw string. (e.g. if we have attribute name="smithland", then {name} will render to "smithland", while {_name} will render to smithland).

Nadi system uses templates in a variety of place, and plugin functions also sometimes take templates for file path, or strings, and such things. Look at the help string of the function to see if it takes String or Template type.

For example render is a function that takes a template and prints it after rendering it for each node.

network load_file("./data/mississippi.net")
node[ohio] set_attrs(river="the Ohio River", streamflow=45334.12424343)
node[ohio,red] render(
	"(=(+ 1 (st+num 'INDEX))th node) {_NAME:case(title)}
	River Flow = {streamflow:calc(/10000):f(3)?\"NA\"} x 10^4"
)

Results:

{
  ohio = "(6th node) Ohio\n\tRiver Flow = 4.533 x 10^4",
  red = "(5th node) Red\n\tRiver Flow = NA x 10^4"
}

As seen in above example, you can render variables, transform them, use basic calculations.

Or you can use lisp syntax to do more complex calculations. Refer to Nadi Extension Capabilities section for more info on how to use lisp on string template.

network load_file("./data/mississippi.net")
node[ohio] set_attrs(river="the Ohio River", streamflow=45334.12424343)
node[ohio] render(
	"{_river:case(title)} Streamflow
	from lisp = {=(/ (st+num 'streamflow) 1000):f(2)} x 10^3 cfs"
)

Results:

{
  ohio = "The Ohio River Streamflow\n\tfrom lisp = 45.33 x 10^3 cfs"
}

Some Complex Examples

Optional variables and a command; note that commands can have variables inside them:

hi there, this {is?a?"test"} for $(echo a simple case {that?} {might} be "possible")

Results (with: might=may):

hi there, this test for $(echo a simple case  may be possible)

Optional variables with transformers inside command.

Hi {like?a?"test"} for $(this does {work:case(up)} now) (yay)

Results (with: work=Fantastic Job):

Hi test for $(this does FANTASTIC JOB now) (yay)

If you need to use { and } in a template, you can escape them. Following template shows how LaTeX commands can be generated from templates.

more {formatting?} options on {%F} and
\\latex\{command\}\{with {variable}\}, should work.

Results (with: command=Error;variable=Var):

more  options on 2025-01-27 and
\latex{command}{with Var}, should work.

This just combined a lot of different things from above:

let's try {every:f(2)?and?"everything"}
for $(a complex case {that?%F?} {might?be?not?found} be "possible")

see $(some-command --flag "and the value" {problem})
=(+ 1 2 (st+num 'hithere) (st+num "otherhi"))
{otherhi?=(1+ pi):f(4)}

*Error*:

None of the variables ["might", "be", "not", "found"] found

This shows the error for the first template part that errors out, even if {problem} will also error later, so while solving for problems in string templates, you might have to give it multiple tries.

Advanced String Template with LISP

Nadi Template string is useful when you want to represent node specific string, or file path in a network. This is not as advanced as the formatted strings in python. But it can be used for complex situations based on the current functionality.

The most important extension capability of the string template is the embedded lisp system.

As we know, templates can render variables, and have some capacity of transforming them:

{name:case(title):repl(-, )} River Streamflow = {streamflow} cfs

Results (with: name=Ohio; streamflow=12000):

Ohio River Streamflow = 12000 cfs

But for numerical operation, the transformers capabilities are limited as they are made for strings.

With lisp, we can add more logic to our templates.

{name:case(title):repl(-, )} River Streamflow is =(
	if (> (st+num 'streamflow) 10000)
	'Higher 'Lower
) than the threshold of 10^5 cfs.

Results (with: name=Ohio; streamflow=12000):

Ohio River Streamflow is Higher than the threshold of 10^5 cfs.

The available lisp functions are also limited, but the syntax itself gives us better airthmetic and logical calculations.

Note

As the template string can get complicated, and the parsing is done through Regex, it is not perfect. If you come across any parsing problems, please raise an issue at string template plus github repo.

Commands

Note that running commands within the templates is disabled for now.

echo today=$(date +%Y-%m-%d) {%Y-%m-%d}

Results (with: ):

echo today=$(date +%Y-%m-%d) 2025-01-27

But if you are writing a command template to run in bash, then it’ll be executed as the syntax is similar.

network command("echo today=$(date +%Y-%m-%d) {%Y-%m-%d}")

Results:

$ echo today=$(date +%Y-%m-%d) 2025-01-27

Here although the $(date +%Y-%m-%d) portion was not rendered on template rendering process, the command was still valid, and was executed.

Tables

Tables are data types with headers and the value template. Tables can be rendered/exported into CSV, JSON, and LaTeX format. Other formats can be added later. Although tables are not exposed to the plugin system, functions to export different table formats can be written as a network function.

A sample Table file showing two columns, left aligned name for station in title case, and right aligned columns for latitude and longitude with float value of 4 digits after decimal:

network load_file("./data/mississippi.net")
<Name => {_NAME:repl(-, ):case(title)}
^Ind => =(+ (st+num 'INDEX) 1)
>Order => {ORDER}
^Level => {LEVEL}
# something is wrong with the set_level algorithm
# Ohio - tenessee should be level 1, and missouri/yellowstone should be 0

Results:

Here the part before => is the column header and the part after is the template. Presence of < or > in the beginning of the line makes the column left or right aligned, with center aligned (^) by default.

Exporting the table in svg instead of markdown allows us better network diagram.

network load_file("./data/mississippi.net")
network echo("../output/example-table2.svg")
<Name => {_NAME:repl(-, ):case(title)}
^Ind => =(+ (st+num 'INDEX) 1)
>Order => {ORDER}
^Level => {LEVEL}

Results:

A SVG Table can also be generated using the table file, using the task system like this:

network load_file("./data/mississippi.net")
network table_to_svg(
	table = "./data/sample.table",
	# either table = "path/to/table", or template = "table template"
	outfile = "./output/example-table.svg",
	config = {fontsize = 16, delta_y = 20, fontface="Noto Serif"}
)
network clip()
# the link path needs to be relative to this file
network echo("../output/example-table.svg")

Results:

File Templates

File templates are templates that use string templates, but they are a whole file that can be used to generate rendered text files.

File templates also have sections which can be repeated for different nodes, with corresponding syntax.

Following template will render a markdown table with headers and all the name and index of the nodes.

| Node | Index |
|------|-------|
<!-- ---8<--- -->
| {_NAME} | {INDEX} |
<!-- ---8<--- -->

Tasks

Task is a function call that the system performs. The function call can be a node function or a network function. The function can have arguments and keyword arguments that can determine its functionality. Node functions will be called on a node at a time, while the network function will be called with the whole network at once.

Currently tasks are performed one after another. The functions that any task can use can be internal functions provided by the library or the external functions provided by the plugins.

A sample tasks file is shown below:

node print_attrs()
network save_graphviz("/tmp/test.gv", offset=1.3, url="{_NAME}")
node savedss(
	"natural",
	"test.dss",
	"/OHIO-RIVER/{_NAME}/01Jan1994/01Jan2012/1Day/NATURAL/"
	)
node check_sf("sf")
node.inputsfirst route_sf("observed")
node render("Node {NAME} at index {INDEX}")

Here each line corresponds to one task. And if it’s a node task, then it’ll be called for each node (in sequential order by default). The last line node.inputsfirst will call that function in input node before the current node. Those functions can only be called for network with an output node.

Node Functions

Node functions are functions that take a node, and the function context to do some operations on it. They take mutable reference to the node, hence can read all node attributes, inputs, outputs, their attributes and timeseries.

Node functions can be run from the system for all the nodes in the network in different orders.

Currently the task system only supports running node functions for all nodes in the following 6 ways,

• Sequential order,
• Reverse order,
• Run input nodes before the current node (recursively),
• Run output node before the current node (recursively),
• Run a list of nodes, and
• Run on a path between two nodes (inclusive).

Depending on the way the function works, it might be required to be run in a particular order. For example, a function that counts the number of dams upstream of each point, might have to be run inputs first, so that you can cumulate the number as you move downstream.

Network Functions

Network functions are functions that take the network as a mutable reference and run on it.

Some examples of network functions:

• List all the networks with their inputs/outputs,
• Checks if any nodes have some attribute larger than their output,
• Export the node attributes as a single CSV file,
• Export the nodes in LaTeX file using Tikz to draw the network,
• Calculate rmse,mse,etc errors between two attribute values for all nodes,
• Generate an interactive HTML/PDF with network information and some other template, etc.

Developer Notes

This section contains my notes as I develop the NADI system. Kind of like a dev blog.

The software package will consists of multiple components. It is planned to be designed in such a way that users can add their functionality and extend it with ease.

Along with the Free and Open Source Software (FOSS) principles, the plugin system will make extension of the software functionality and sharing between users. As well as a way to develop in-house functionality for niche use cases.

Motivation

As Hydrologist, we often deal with the data related to the points in the river. Since most of the analysis requires doing the same things in multiple points, the initial phase of data cleaning process can be automated.

We spend a beginning phases of all projects preparing the data for analysis. And combining the time spent on visualizing the data, it’s a significant chunk of our time.

Data visualization influences the decision making from the stakeholders. And can save time by making any problems obvious from the very beginning. For examples, things like showing the quality of data (continuity for time series), interactive plots to compare data in different locations/formats, etc can help people understand their data better.

Besides plot, the example below shows how simply adding a column with connection visual can immediately make it easier to understand the relationship between the data points in a river. Without it people need to be familiar with the names of the data points and their location, or consult a different image/map to understand the relationship.

The inspiration on making this software package comes from many years of struggle with doing the same thing again and again in different projects like these. And the motivation to make something generic that can be used for plethora of projects in the future.

Why Rust?

Rust¹ is an open source programming language that claims to be fast and memory efficient to power performance critical services. Rust is also able to integrate with other programming languages.

Rust provides a memory safe way to do modern programming. The White House has a recent press release² about the need to have memory safe language in future softwares. The report³ has following sentense about the Rust language.

The results of the survey from stackoverflow⁴ shows Rust has been a top choice for developers who want to use a new technology for the past 8 years, and the analysis also shows Rust is a language that generates for desire to use it once you get to know.

Writing this Book

I’m used to emacs’s org-mode, where you can evaluate code and show output and all those things. Like markdown in steroids.

mdbook seems to have some of those functionality in it as well. Though I think emacs’s extension through elisp is lot more flexible and easier to extend. mdbook supporting custom preprocessors and renderer means we can extend it as well.

In the process of writing this book. I made the following things.

Syntax Highlight for NADI specific syntax

mdbook uses highlight.js to syntax highlight the code blocks in it. And since nadi system has a lot of its own syntax for string templates, task system, table system, network system etc. I wanted syntax highlight for those things. Although the attribute files are subset of TOML format, so we have syntax highlight for it. Everything else needed a custom code.

Following the comments in this github issue led me to find a workaround for the custom syntax hightlight. I don’t know for how long it will work, but this works well for now.

Basically I am using the custom JS feature of mdbook like:

[output.html]
additional-js = ["theme/syntax-highlight.js"]

To insert custom highlight syntax. For example adding the syntax highlight for network text is:

// network connections comments and node -> node syntax
hljs.registerLanguage("network", (hljs) => ({
    name: "Network",
    aliases: [ 'net' ],
    contains: [
	hljs.QUOTE_STRING_MODE,
	hljs.HASH_COMMENT_MODE,
	{
	    scope: "meta",
	    begin: '->',
	    className:"built_in",
	},
    ]
}));

The syntax for network is really simple, for others (task, table, string-template, etc) refer to the theme/syntax-highlight.js file in the repository for this book.

After registering all the languages, you re-initialize the highlight.js:

hljs.initHighlightingOnLoad();

mdbook-nadi preprocessor

Instead of just showing the syntax of how to use the task system, I wanted to also show the output of the examples for readers. So I started this with writing some elisp code to run the text in selection and then copying the output to clipboard that I could paste in output block. It was really easy in emacs.

Following code takes the selection, saves them in temporary tasks file, runs them and then puts the output in the clipboard that I can paste manually.

(defun nadi-run-tasks (BEG END)
  (interactive "r")
  (let ((tasks-file (make-temp-file "tasks-")))
    (write-region BEG END tasks-file)
    (let ((output '(shell-command-to-string (format "nadi %s" tasks-file))))
	  (message output)
	  (kill-new output)
	  (delete-file tasks-file))))

But this is manual process with a bit of automation. So I wanted a better solution, and that’s where the mdbook preprocessor comes in.

With the mdbook-nadi preprocessor, I can extract the code blocks, run it, and insert the contents just below the code block as output.

Once I had a working prototype for this, I also started adding support for rendering string templates, and generating tables along with the task system.

String templates

For string templates, write the templates in stp blocks like below that will have the syntax hightlight.

Hi my name is {name}.

If you add run into it, it’ll run the template with any key=val pairs provided after run.

Basically writing the following in the mdbook markdown:

```stp run name=John
Hi my name is {name}.
```

Will become:

Hi my name is {name}.

Results (with: name=John):

Hi my name is John.

Tasks

For tasks, similary write a block with task as language. You can use ! character at the start of the line to hide it in the view. Use them for essential code that are needed for results but are not the current focus. And when you add run it’ll run and show the output.

```task run
!network load_file("data/mississippi.net")
node render("Node {NAME}")
```

network load_file("data/mississippi.net")
node render("Node {NAME}")

Results:

{
  lower-mississippi = "Node \"lower-mississippi\"",
  upper-mississippi = "Node \"upper-mississippi\"",
  missouri = "Node \"missouri\"",
  arkansas = "Node \"arkansas\"",
  red = "Node \"red\"",
  ohio = "Node \"ohio\"",
  tenessee = "Node \"tenessee\""
}

Tables

The implementation for tables are little weird right now, but it works. Since we need to be able to load network, and perform actions before showing a table.

So the current implementation takes the hidden lines using !and runs them as task system, with additional task of rendering the table at the end.

Example:

```table run markdown
!network load_file("./data/mississippi.net")
<Name => {_NAME:repl(-, ):case(title)}
^Ind => =(+ (st+num 'INDEX) 1)
>Order => {ORDER}
```

Becomes:

network load_file("./data/mississippi.net")
<Name => {_NAME:repl(-, ):case(title)}
^Ind => =(+ (st+num 'INDEX) 1)
>Order => {ORDER}

Results:

I’d like to refine this further.

Task can be used to generate markdown in the same way as the tables can:

For example task run of this:

network load_file("./data/mississippi.net")
network table_to_markdown(template="
<Name => {_NAME:repl(-, ):case(title)}
^Ind => =(+ (st+num 'INDEX) 1)
>Order => {ORDER}
")

Results:

| Name              | Ind | Order |
|:------------------|:---:|------:|
| Lower Mississippi |  1  |     7 |
| Upper Mississippi |  2  |     1 |
| Missouri          |  3  |     1 |
| Arkansas          |  4  |     1 |
| Red               |  5  |     1 |
| Ohio              |  6  |     2 |
| Tenessee          |  7  |     1 |

If you do task run markdown then:

network load_file("./data/mississippi.net")
network table_to_markdown(template="
<Name => {_NAME:repl(-, ):case(title)}
^Ind => =(+ (st+num 'INDEX) 1)
>Order => {ORDER}
")

Results:

Which means it can be used for other things:

network load_file("./data/mississippi.net")
network echo("**Details about the Nodes:**")
node render("
=(+ (st+num 'INDEX) 1). {_NAME:repl(-, ):case(title)} River
")

Results:
Details about the Nodes: { lower-mississippi = “\n1. Lower Mississippi River\n”, upper-mississippi = “\n2. Upper Mississippi River\n”, missouri = “\n3. Missouri River\n”, arkansas = “\n4. Arkansas River\n”, red = “\n5. Red River\n”, ohio = “\n6. Ohio River\n”, tenessee = “\n7. Tenessee River\n” }

You can also use the same method to insert images like this, at the end of your tasks, so that the image generated by the tasks can be inserted here.

# do some tasks
network echo("Some other output form your tasks")
network clip()
network echo("../images/ohio-low.svg")

Results:

Optimization Algorithms

We can have input variables to change, and output variables to optimize, but how do we take what function to run to calculate the output variable…

One simple idea can be to take a command template to run. So we will change the input variables, run the command for each node or network, and then that command will update the output variable that we can optimize for.

We might require an option to call other functions in this case. Then maybe we can just pass the name of the function.

Complex idea could be to add the support for loop syntax in task system.

Interactive Plots

An experiment using the cairo graphics library shows that a PDF can be directly produced without using LaTeX as intermediate using the network information. This functionality — although not as complete as the one in the example — has been exposed as an internal network function for now. Further functionality related to this idea can be embedding network information in simple plots, or generate the whole plot along side the network information.

It might be a good idea to make several functions that can export the interactive plots in LaTeX, PDF, PNG, SVG, HTML, etc. separately instead of single format.

LaTeX and HTML will be easier due to text nature, for others I might have to spend time with some more experimentation on cairo.