Mike Gerwitz

Activist for User Freedom

aboutsummaryrefslogtreecommitdiffstats
path: root/doc
diff options
context:
space:
mode:
authorMike Gerwitz <gerwitzm@lovullo.com>2017-06-29 10:50:02 -0400
committerMike Gerwitz <gerwitzm@lovullo.com>2017-06-29 10:50:02 -0400
commit35af4bdf191cec00d947acf0fdced62834109e3e (patch)
treedf1e9404787e07f64b9690dd79e60933fe35cb32 /doc
parent3be28a7858b1fc62152050560fa6cf01d8dc991d (diff)
parent0c24e3d2807a3d5aa5f9c768f449c6ba2b7fc6dd (diff)
downloadliza-35af4bdf191cec00d947acf0fdced62834109e3e.tar.gz
liza-35af4bdf191cec00d947acf0fdced62834109e3e.tar.bz2
liza-35af4bdf191cec00d947acf0fdced62834109e3e.zip
Server-side Data API support
The Data API stuff is pretty well coupled to the concept of a UI, and the code is a mess that was just extracted from Program.js, so this isn't the prettiest implementation. I hope that we'll have more time for it in the somewhat-near future, since this is going to be core to all server-side network requests for data.
Diffstat (limited to 'doc')
-rw-r--r--doc/Makefile.am5
-rw-r--r--doc/assert.texi29
-rw-r--r--doc/bucket.texi36
-rw-r--r--doc/client.texi98
-rw-r--r--doc/dapi.texi149
-rw-r--r--doc/design.texi450
-rw-r--r--doc/liza.texi16
-rw-r--r--doc/macros.texi52
-rw-r--r--doc/pred.texi74
-rw-r--r--doc/program.texi282
-rw-r--r--doc/server.texi268
-rw-r--r--doc/validation.texi25
12 files changed, 1038 insertions, 446 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am
index ccbd36a..508ed8b 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -21,7 +21,10 @@
MAKEINFOHTML = $(MAKEINFO) --html --css-include liza.css
info_TEXINFOS = liza.texi
-liza_TEXINFOS = design.texi macros.texi config.texi liza.css \
+liza_TEXINFOS = macros.texi config.texi liza.css \
+ design.texi assert.texi bucket.texi client.texi \
+ dapi.texi pred.texi program.texi server.texi \
+ validation.texi \
diagram/event-graph.svg
diagram/%.svg: diagram/%.dot
diff --git a/doc/assert.texi b/doc/assert.texi
new file mode 100644
index 0000000..ce16ce9
--- /dev/null
+++ b/doc/assert.texi
@@ -0,0 +1,29 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Assertions
+@chapter Assertions
+@maintstart
+@footnote{
+ Assertions are compiled from the Program XML (@pxref{Program XML}).
+ Rather than using a library,
+ it compiles a mess of largely duplicate code inline.
+ This system needs to be @emph{replaced},
+ not modified.
+
+ A replacement can either be in the form of a library (removing most
+ if not all code generation from the Program XML compiler),
+ or possibly compile into classifications and use the classification
+ system.
+ @emph{The latter option is preferred,
+ and would be more powerful with less maintenance.}}
+@maintend
+
+@helpwanted
diff --git a/doc/bucket.texi b/doc/bucket.texi
new file mode 100644
index 0000000..0b4a810
--- /dev/null
+++ b/doc/bucket.texi
@@ -0,0 +1,36 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Bucket
+@chapter Bucket
+@helpwanted
+
+@menu
+* Value Assignment:Bucket Assignment. Writing data to the Bucket.
+* Bucket Diff:: Representing bucket changes.
+* Calculated Values:: Dynamic data derived from other values.
+@end menu
+
+
+@c TODO
+@node Bucket Assignment
+@section Bucket Value Assignment
+@helpwanted
+
+
+@node Bucket Diff
+@section Bucket Diff
+@cindex Bucket diff
+@helpwanted
+
+
+@node Calculated Values
+@section Calculated Values
+@helpwanted
diff --git a/doc/client.texi b/doc/client.texi
new file mode 100644
index 0000000..8955264
--- /dev/null
+++ b/doc/client.texi
@@ -0,0 +1,98 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Client
+@chapter Client
+@maintstart
+@footnote{
+ The client is largely managed by a single class,
+ @srcref{src/client/Client.js, Client},
+ which has grown out of control.
+ @code{Client} mediates essentially the entire system.
+ Code is to be extracted out of this class as it is touched.
+
+ The other system mammoth is @code{Ui} (@pxref{Program UI}).}
+@maintend
+
+@helpwanted
+
+@menu
+* Error Handling::
+@end menu
+
+
+@node Error Handling
+@section Error Handling
+
+@maintenance{
+ The complexity of this system and integration into legacy
+ layers has caused maintenance trouble in the past.
+ Each of the error checking layers need to be integrated
+ to reduce complexity.}
+
+@cindex Error
+@cindex Failure
+There are three layers of error checking:@footnote{
+ Primarily for legacy reasons.
+ They are being consolodated as the system is touched.}
+
+@enumerate
+ @item Required field checking@mdash{
+ }whether all required questions have been answered.
+ @item Type Validation@mdash{
+ }verify that questions contain valid data according to their
+ declared type. @ref{Validation}.
+ @item Assertions@mdash{
+ }arbitrary checks on data.
+ @ref{Assertions}.
+@end enumerate
+
+@cindex Required Field
+@cindex Field, Required
+@cindex Field, Fixed
+@cindex Error, Fixed
+@cindex Error, Required
+@cindex Fixed, Error
+Required fields fail serially@mdash{
+ }the system will notify the user of the required field,
+ and direct him/her to it (usually through scrolling).
+A field is marked as @dfn{fixed} according to the rules
+ in @ref{Managing Error State}.
+
+
+@menu
+* Managing Error State:: Determining when failures should be marked as ``fixed''
+@end menu
+
+
+@node Managing Error State
+@subsection Managing Error State
+
+@cindex Failure Stack
+@cindex Error Stack
+Each failure caused by assertions is associated with a
+ @dfn{failure stack}.
+The stack represents the trail of assertions that have run,
+ containing the ids of all values asserted against.
+When any field or classification changes that is represented on the
+ failure stack,
+ the failure for the failed field associated with that failure stack
+ is cleared.
+
+@exnotice{
+ If an assertion for some question @var{foo} first checked the value
+ of bucket field @var{bar},
+ and within its failure checked the value @var{c:predicate},
+ the failure stack would contain both of those ids.
+ If either @var{bar} or the @var{predicate} classification changed,
+ the question @var{foo} would have its error cleared.}
+
+Error state is managed by
+ @srcref{src/validate/ValidStateMonitor.js, ValidStateMonitor}.
diff --git a/doc/dapi.texi b/doc/dapi.texi
new file mode 100644
index 0000000..ca61c92
--- /dev/null
+++ b/doc/dapi.texi
@@ -0,0 +1,149 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Data API
+@chapter Data API
+@maintenance{This is a complex system with too much logic lying in
+ @srcrefjs{dapi,DataApiManager} (having been extracted
+ from its old home in @srcrefjs{program,Program} ).}
+
+@helpwanted{}
+
+The @dfn{Data API} is a declarative abstraction for accessing and
+ processing remote data (e.g. a RESTful service).
+The name stems from how it is used@mdash{
+ }to declare an remote API's inputs and outputs.
+
+This system is generally used indirectly through the @progxmlref{}.@footnote{
+ @proguicxref{Data API}.}
+
+@tip{All interaction with this system should be had through the
+ @srcrefjs{dapi,DataApiManager}.}
+
+The @srcrefjs{dapi,DataApiManager} manages the entire operation@mdash{
+ }from triggering the initial request,
+ to performing mapping,
+ to populating bucket data.
+It takes only a @srcrefjs{dapi,DataApiFactory} and @dapi{} definitions.
+
+Definitions have the following schema:@footnote{
+ There are poor design decisions that will likely persist
+ indefinitely because of integration with other systems,
+ so future extensions may be messy (especially in the case of
+ @samp{retvals}).
+}
+
+@verbatim
+{
+ "type": "string",
+ "source": "string",
+ "method": "string",
+ "params": {
+ ["string(name)"]: {
+ "name": "string(name)",
+ "default": {
+ "type": "string",
+ "value": "string"
+ },
+ ...
+ },
+ },
+ "retvals": [ "string", ... ],
+ "static": [
+ {
+ ["string(param)"]: "string",
+ ...
+ },
+ ...
+ ],
+ "static_nonempty": boolean,
+ "static_multiple": boolean
+}
+@end verbatim
+
+Each of the above fields are defined by:
+
+@table @code
+ @item type
+ Any type supported by @srcrefjs{dapi,DataApiFactory} (e.g. @samp{rest}).
+
+ @item source
+ Type-specific source of data.
+ For e.g. @samp{rest}, this is a URI.
+
+ @item method
+ Type-specific method for interacting with the API.
+ For e.g. @samp{rest}, this is an HTTP@tie{}method.
+
+ @item params
+ Key-value mapping of input parameter names (as received by @samp{source})
+ to their default values.
+ These inputs must be populated by the caller at the time of the request.
+
+ @item retvals
+ Array of fields returned by the data source.
+
+ @item static
+ Static values to prepend to the returned data.
+ This is often used for adding ``please select'' text, for example.
+
+ @item static_nonempty
+ Whether statics should be added when there is return data;
+ Otherwise,
+ they will be added only if the response yields no results.
+
+ @item static_multiple
+ Whether statics should be added only if multiple data are returned.
+ For example,
+ a ``please select'' is only useful if there is more than one
+ option for the user to select from.
+ When @samp{true},
+ this has the convenient side-effect of auto-selecting the only
+ result.
+@end table
+
+An example definition appears in @ref{f:dapi-ex}
+
+@float Figure, f:dapi-ex
+@example
+ @{
+ "type": "rest",
+ "source": "/foo/city",
+ "method": "post",
+ "params": @{
+ "getVal": @{
+ "name": "getVal",
+ "default": @{
+ "type": "string",
+ "value": "getCityOptions"
+ @}
+ @},
+ "zipcode": @{
+ "name": "zipcode",
+ "default": @{
+ "type": "ref",
+ "value": ""
+ @}
+ @}
+ @},
+ "retvals": [ "city", "id", "state", "county", "country" ],
+ "static": [ @{
+ "city": "(Please Select)",
+ "id": "",
+ "state": "",
+ "county": "",
+ "country": ""
+ @} ],
+ "static_nonempty": false,
+ "static_multiple": true
+ @},
+@end example
+@caption{Example @dapi{} definition}
+@end float
diff --git a/doc/design.texi b/doc/design.texi
index 7bde80d..d605fde 100644
--- a/doc/design.texi
+++ b/doc/design.texi
@@ -34,10 +34,6 @@ The main components of the system are:
and provides hooks that drive the rest of the system.
@xref{Bucket}.
- @cindex Calculated Value
- @item Calculated Values
- A small sub-system for calculating bucket values from other values.
-
@cindex Client
@item Client
Basic logic for navigating between steps,
@@ -47,6 +43,12 @@ The main components of the system are:
etc.
@xref{Client}.
+ @cindex Data API
+ @item Data API
+ Declarative abstraction for accessing and processing remote data
+ (e.g. a RESTful service).
+ @xref{Data API}.
+
@cindex Developer Dialog
@item Developer Dialog
Renders information about the system for debugging the client.
@@ -91,6 +93,7 @@ The main components of the system are:
and other types of processing.
Code is shared with the client,
ensuring identical behavior for appropriate behaviors.
+ @xref{Server}.
@cindex Type Validation
@cindex Validation, Type
@@ -102,441 +105,4 @@ The main components of the system are:
@xref{Validation}.
@end table
-More information about each can be found in their respective section.
-
-
-@menu
-* Assertions::
-* Bucket::
-* Client::
-* Predicate System::
-* Program::
-* Program UI::
-* Program XML::
-* Validation::
-@end menu
-
-
-
-@node Assertions
-@section Assertions
-@maintstart
-@footnote{
- Assertions are compiled from the Program XML (@pxref{Program XML}).
- Rather than using a library,
- it compiles a mess of largely duplicate code inline.
- This system needs to be @emph{replaced},
- not modified.
-
- A replacement can either be in the form of a library (removing most
- if not all code generation from the Program XML compiler),
- or possibly compile into classifications and use the classification
- system.
- @emph{The latter option is preferred,
- and would be more powerful with less maintenance.}}
-@maintend
-
-@helpwanted
-
-
-
-@node Bucket
-@section Bucket
-@helpwanted
-
-@menu
-* Value Assignment:Bucket Assignment. Writing data to the Bucket.
-@end menu
-
-
-@c TODO
-@node Bucket Assignment
-@subsection Bucket Value Assignment
-@helpwanted
-
-
-
-@node Client
-@section Client
-@maintstart
-@footnote{
- The client is largely managed by a single class,
- @srcref{src/client/Client.js, Client},
- which has grown out of control.
- @code{Client} mediates essentially the entire system.
- Code is to be extracted out of this class as it is touched.
-
- The other system mammoth is @code{Ui} (@pxref{Program UI}).}
-@maintend
-
-@helpwanted
-
-@menu
-* Error Handling::
-@end menu
-
-
-@node Error Handling
-@subsection Error Handling
-
-@maintenance{
- The complexity of this system and integration into legacy
- layers has caused maintenance trouble in the past.
- Each of the error checking layers need to be integrated
- to reduce complexity.}
-
-@cindex Error
-@cindex Failure
-There are three layers of error checking:@footnote{
- Primarily for legacy reasons.
- They are being consolodated as the system is touched.}
-
-@enumerate
- @item Required field checking@mdash{
- }whether all required questions have been answered.
- @item Type Validation@mdash{
- }verify that questions contain valid data according to their
- declared type. @ref{Validation}.
- @item Assertions@mdash{
- }arbitrary checks on data.
- @ref{Assertions}.
-@end enumerate
-
-@cindex Required Field
-@cindex Field, Required
-@cindex Field, Fixed
-@cindex Error, Fixed
-@cindex Error, Required
-@cindex Fixed, Error
-Required fields fail serially@mdash{
- }the system will notify the user of the required field,
- and direct him/her to it (usually through scrolling).
-A field is marked as @dfn{fixed} according to the rules
- in @ref{Managing Error State}.
-
-
-@menu
-* Managing Error State:: Determining when failures should be marked as ``fixed''
-@end menu
-
-
-@node Managing Error State
-@subsubsection Managing Error State
-
-@cindex Failure Stack
-@cindex Error Stack
-Each failure caused by assertions is associated with a
- @dfn{failure stack}.
-The stack represents the trail of assertions that have run,
- containing the ids of all values asserted against.
-When any field or classification changes that is represented on the
- failure stack,
- the failure for the failed field associated with that failure stack
- is cleared.
-
-@exnotice{
- If an assertion for some question @var{foo} first checked the value
- of bucket field @var{bar},
- and within its failure checked the value @var{c:predicate},
- the failure stack would contain both of those ids.
- If either @var{bar} or the @var{predicate} classification changed,
- the question @var{foo} would have its error cleared.}
-
-Error state is managed by
- @srcref{src/validate/ValidStateMonitor.js, ValidStateMonitor}.
-
-
-
-@node Predicate System
-@section Predicate System
-@maintstart
-@footnote{
- New programs (using the old incarnation of TAME) use the classifier
- embedded into the rater by TAME.
- Old ones, however, still use the @dfn{Global Classifier}.
- This system isn't as well tested as TAME's@mdash{
- }which needs to work properly for the sake of calculating premium@mdash{
- }and has suffered from a number of bugs in the past.
-
- The solution is to migrate all programs to TAME and remove that old
- code.}
-@maintend
-
-@tip{
- For a practical application of these concepts, see its use in the
- Program@tie{}XML (@pxref{Specifying Predicates}).}
-
-@cindex Predicate
-@cindex Classifier
-@cindex Applicability
-@cindex Domain of discourse, Predicate
-The @dfn{predicate system} determines the @dfn{applicability} of
- certain objects (like questions and assertions) by associating them
- with predicates.
-The domain of discourse (variables which may be quantified) is listed
- in @ref{t:predicate-dod}.
-
-What it means for some object to be applicable depends on the context.
-
-@float Table, t:predicate-dod
-@multitable @columnfractions 0.25 0.10 0.65
- @headitem Type @tab Prefix @tab Description
-
- @item Classifications
- @tab @emph{None}
- @tab Results of applying an external @dfn{classifier} to the bucket
- (@pxref{Bucket}).
-
- @item Bucket Truth Predicate
- @tab @code{q:}
- @tab
- Whether the given name in the bucket (@pxref{Bucket}) is
- non-empty and non-zero.
- The prefix @samp{q:} refers to its most common use
- case---questions (@pxref{Program UI,,Program@tie{}UI}).
-@end multitable
-@caption{Predicate system domain of discourse}
-@end float
-
-This system is limited to universal quantification over the domain of
- discourse.
-For other quantifiers and higher-order logic,
- defer to one of the systems that contributes to the domain of
- discourse,
- like the classifier.@footnote{
- This is usually TAME.
- The Program XML also supports inline classifications
- with TAME's syntax (@pxref{Specifying Predicates}).}
-
-Predicates are usually specified in the Program XML
- (@pxref{Specifying Predicates}) and compiled into the program
- (@pxref{Program}).
-
-
-
-@node Program
-@section Program
-@maintstart
-@footnote{
- The @code{Program} class was one of the first prototypes created,
- and has evolved poorly with the rest of the system.
- It is the base class for all compiled programs,
- and it glues together too many other systems with a terrible
- API and little to no encapsulation.
-
- With that said, it is one of the least touched classes (thus its
- state); developers rarely have the need to touch @code{Program}.}
-@maintend
-
-
-@cindex Program
-The @dfn{Program} is a declarative representation of an entire system.
-It is the highest level of abstraction from a data perspective.
-The user observes and interacts with a Program using the
- @ref{Program UI,,Program@tie{}UI}.
-
-@cindex Program, XML
-Programs contain a lot of metadata that is not in a convenience
- human-readable (or modifiable) format,
- some of which are redundant.
-Programs are ideally compiled from a @ref{Program XML,,Program@tie{}XML}
- document.
-
-
-
-@node Program UI
-@section Program UI
-@maintenance{
- The @code{Ui} class,
- in addition to @srcref{src/client/Client,Client} (@pxref{Client}),
- represent the two monoliths of the system.
- This mediates all UI-related tasks,
- and still has far too many concerns with far too many
- dependencies.
- Code is to be extracted out of this class as it is touched.
-}
-
-
-@cindex Program, User Interface
-@cindex User Interface, Program
-The @dfn{Program UI} renders a @ref{Program} as a form.
-
-@cindex Step
-@cindex Group
-At the highest level,
- steps are rendered in a tab-like manner,
- above the main form content.
-A step contains groups,
- which in turn contain elements such as questions.
-Groups are delimited in some manner defined by their style
- (@pxref{Group Styles}).
-
-@cindex Question
-@cindex Question, Value Formatting
-@cindex Bucket, Updating
-Questions are rendered as form fields.
-Any time the respective @ref{Bucket} field is changed,
- the form field is updated to reflect those changes,
- after having first been formatted with the appropriate validator
- (@pxref{Formatting Values}).
-When a question is changed by the user,
- the value is expected to be propagated to the Bucket
- (@pxref{Bucket Assignment}).
-
-@cindex Navigation Bar
-@cindex User Interface, Navigation Bar
-@cindex User Interface, Button Navigation
-Navigation between steps can be done via the
- @dfn{Navigation Bar} above the step@tie{}content,
- or using ``Go@tie{}Back'' and ``Continue'' buttons at the foot of the
- step@tie{}content.
-
-@cindex Sidebar
-A @dfn{Sidebar} is rendered adjacent to the step content.
-It displays the name of the Program,
- as well as configurable metadata (usually through the @samp{sidebar}
- node of the @ref{Program XML,,Program@tie{}XML}).
-It also displays question help text (also configured through the XML)
- and any error messages (@pxref{Error Handling}).
-
-@menu
-* Group Styles:: Different ways of displaying groups of questions to
- the user.
-@end menu
-
-
-@node Group Styles
-@subsection Group Styles
-@refactor{
- Some group styles still use jQuery;
- they should be modified to use modern formatters and Liza DOM
- abstractions (see @srcrefraw{src/ui/field}
- and @srcrefraw{src/ui/styler}).}
-
-@cindex Group, Styling
-Groups support a number of @dfn{group styles} that determine how
- they are delimited from other groups;
- how the elements they contain are rendered and laid out;
- and how multiple indexes are displayed, added, and removed.
-A list of available styles is detailed in @ref{t:group-styles}.
-
-@float Table, t:group-styles
-@multitable @columnfractions 0.15 0.65 0.10 0.10
- @headitem Name @tab Description @tab Multi-Index? @tab Add/Remove Index?
-
- @item @samp{default}
- @tab
- Groups are unstyled by default@mdash{
- }they render elements as flat fields like a traditional form.
- Only the first index of elements is rendered.
- @tab@center N
- @tab@center N
-
- @item @samp{collapsetable}
- @tab
- Renders element label in the leftmost column like @samp{sidetable}.
- Indexes are groups of rows delimited by headings,
- which collapse the respective group of rows when clicked.
- @tab@center Y
- @tab@center Add
-
- @item @samp{sidetable}
- @tab
- Renders elements as rows with label in the leftmost column rather
- than the top row.
- Each index is rendered as a column.
- @tab@center Y
- @tab@center Add
-
- @item @samp{tabbedblock}
- @tab
- Each group is rendered as a block,
- with each index rendered as a tab to the right of it.
- Clicking a tab toggles the body content to the associated index.
- Elements are rendered within the box.
- @tab@center Y
- @tab@center N
-
- @item @samp{tabbed}
- @tab
- Like @samp{default},
- but each index has a tab at the top of the group.
- Clicking a tab toggles the body content to the associated index.
- @tab@center Y
- @tab@center Y
-
- @item @samp{table}
- @tab
- A vanilla table with elements as columns,
- their labels across the top row.
- Each index is rendered in its own row.
- @tab@center Y
- @tab@center Y
-@end multitable
-@caption{Group styles and index support}
-@end float
-
-
-
-@node Program XML
-@section Program XML
-@helpwanted
-
-@menu
-* Specifying Predicates::
-@end menu
-
-
-@node Specifying Predicates
-@subsection Specifying Predicates
-
-Object predicates (@pxref{Predicate System}) are specified using the
- @xmlattr{when} attribute of certain nodes.
-It must contain a string of references understood by the system
- (see domain of discourse, @ref{Predicate System}),
- all of which much match for the predicate to be true.
-
-@float Figure, f:pred-when
-@example
- <question id="describe" type="noyes"
- label="Any special notes for this location?" />
-
- <question id="vacant_desc" type="textarea"
- when="q:describe vacant property"
- label="Show only when a vacant property with the
- question 'describe' non-empty and non-zero" />
-@end example
-@caption{Using the @xmlattr{when} attribute}
-@end float
-
-In @ref{f:pred-when} above,
- question @samp{vacant_desc} will be applicable when @emph{all} of
- the values of @samp{vacant}, @samp{property},
- and@tie{}@samp{q:describe} are true.@footnote{
- @xref{Predicate System} for what ``true'' means for a particular
- variable in the domain of discourse.}
-Within the context of the @progxml,
- this concretely means that the classifications
- @samp{vacant} and@tie{}@samp{property} are true,
- and that the question @samp{describe} is answered ``yes''.
-It reads as a sentence:
- ``@samp{vacant_desc}'' is applicable when we should @tie{}``describe
- a vacant property''.
-
-
-
-@node Validation
-@section Validation
-@helpwanted
-
-@menu
-* Formatting Values::
-@end menu
-
-
-@node Formatting Values
-@subsection Formatting Values
-
-@cindex Question
-@cindex Question, Value Formatting
-@helpwanted
+More information about each can be found in their respective chapter/section.
diff --git a/doc/liza.texi b/doc/liza.texi
index d091cd9..c8bd87e 100644
--- a/doc/liza.texi
+++ b/doc/liza.texi
@@ -76,6 +76,14 @@ Free Documentation License''.
@menu
* Design and Architecture:Design. Design and structure of framework
+* Assertions::
+* Bucket::
+* Client::
+* Data API::
+* Predicate System::
+* Program::
+* Server::
+* Validation::
* License:: Document License
* Concept Index::
@ifset DEVNOTES
@@ -89,6 +97,14 @@ Free Documentation License''.
@end ifnottex
@include design.texi
+@include assert.texi
+@include bucket.texi
+@include client.texi
+@include dapi.texi
+@include pred.texi
+@include program.texi
+@include server.texi
+@include validation.texi
@include license.texi
@node Concept Index
diff --git a/doc/macros.texi b/doc/macros.texi
index db725ba..e5a283c 100644
--- a/doc/macros.texi
+++ b/doc/macros.texi
@@ -66,12 +66,17 @@
@end ifset
@end macro
+@c documentation TODO
+@macro todo{text}
+@devnotice{TODO: \text\}
+@end macro
+
@c indicate that help is needed to produce docs
@macro helpwanted{}
@cindex TODO, Missing Docs
@dnindex Missing Docs
-@notice{There's nothing here yet. Maybe you can help?}
+@notice{There isn't much here yet. Maybe you can help?}
@end macro
@@ -140,11 +145,15 @@ This system has maintenance concerns.
@c the path to the file
@ifset SRCURI
@macro srcref{path, display}
-@url{@value{SRCURI}/\path\, @code{\display\}}
+@url{@value{SRCURI}/\path\, @file{\display\}}
@end macro
@macro srcrefraw{path}
-@url{@value{SRCURI}/\path\, @code{\path\}}
+@url{@value{SRCURI}/\path\, @file{\path\}}
+@end macro
+
+@macro srcrefjs{base,module}
+@srcref{src/\base\/\module\.js, \module\}
@end macro
@end ifset
@@ -156,6 +165,12 @@ This system has maintenance concerns.
@macro srcrefraw{path}
@file{\path\}
@end macro
+
+@c intended to display a name without JS,
+@c so just do that rather than the actual path
+@macro srcrefjs{base,path}
+@srcrefraw{\base\/\path\}
+@end macro
@end ifclear
@@ -169,7 +184,38 @@ This system has maintenance concerns.
@end macro
+@c JS formatting
+@macro jsmethod{name}
+@code{#\name\}
+@end macro
+
+
@c text to avoid repeated e.g. ties and other formatting
@macro progxml
Program@tie{}XML
@end macro
+
+@macro progxmlref
+@ref{Program XML,,Program@tie{}XML}
+@end macro
+
+@macro dapi
+Data@tie{}API
+@end macro
+
+@macro dapiref
+@dapi (@pxref{Data API,,Data@tie{}API})
+@end macro
+
+@c todo: link to reference directly
+@macro proguicref{ref}
+`\ref\' @proguicrefsuffix
+@end macro
+
+@macro proguicxref{ref}
+See `\ref\' @proguicrefsuffix
+@end macro
+
+@macro proguicrefsuffix{}
+in the Liza Program@tie{}UI Compiler manual
+@end macro
diff --git a/doc/pred.texi b/doc/pred.texi
new file mode 100644
index 0000000..8ee01eb
--- /dev/null
+++ b/doc/pred.texi
@@ -0,0 +1,74 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Predicate System
+@chapter Predicate System
+@maintstart
+@footnote{
+ New programs (using the old incarnation of TAME) use the classifier
+ embedded into the rater by TAME.
+ Old ones, however, still use the @dfn{Global Classifier}.
+ This system isn't as well tested as TAME's@mdash{
+ }which needs to work properly for the sake of calculating premium@mdash{
+ }and has suffered from a number of bugs in the past.
+
+ The solution is to migrate all programs to TAME and remove that old
+ code.}
+@maintend
+
+@tip{
+ For a practical application of these concepts, see its use in the
+ Program@tie{}XML (@pxref{Specifying Predicates}).}
+
+@cindex Predicate
+@cindex Classifier
+@cindex Applicability
+@cindex Domain of discourse, Predicate
+The @dfn{predicate system} determines the @dfn{applicability} of
+ certain objects (like questions and assertions) by associating them
+ with predicates.
+The domain of discourse (variables which may be quantified) is listed
+ in @ref{t:predicate-dod}.
+
+What it means for some object to be applicable depends on the context.
+
+@float Table, t:predicate-dod
+@multitable @columnfractions 0.25 0.10 0.65
+ @headitem Type @tab Prefix @tab Description
+
+ @item Classifications
+ @tab @emph{None}
+ @tab Results of applying an external @dfn{classifier} to the bucket
+ (@pxref{Bucket}).
+
+ @item Bucket Truth Predicate
+ @tab @code{q:}
+ @tab
+ Whether the given name in the bucket (@pxref{Bucket}) is
+ non-empty and non-zero.
+ The prefix @samp{q:} refers to its most common use
+ case---questions (@pxref{Program UI,,Program@tie{}UI}).
+@end multitable
+@caption{Predicate system domain of discourse}
+@end float
+
+This system is limited to universal quantification over the domain of
+ discourse.
+For other quantifiers and higher-order logic,
+ defer to one of the systems that contributes to the domain of
+ discourse,
+ like the classifier.@footnote{
+ This is usually TAME.
+ The Program XML also supports inline classifications
+ with TAME's syntax (@pxref{Specifying Predicates}).}
+
+Predicates are usually specified in the Program XML
+ (@pxref{Specifying Predicates}) and compiled into the program
+ (@pxref{Program}).
diff --git a/doc/program.texi b/doc/program.texi
new file mode 100644
index 0000000..94a1e12
--- /dev/null
+++ b/doc/program.texi
@@ -0,0 +1,282 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+
+@node Program
+@chapter Program
+@maintstart
+@footnote{
+ The @code{Program} class was one of the first prototypes created,
+ and has evolved poorly with the rest of the system.
+ It is the base class for all compiled programs,
+ and it glues together too many other systems with a terrible
+ API and little to no encapsulation.
+
+ With that said, it is one of the least touched classes (thus its
+ state); developers rarely have the need to touch @code{Program}.}
+@maintend
+
+
+@cindex Program
+The @dfn{Program} is a declarative representation of an entire system.
+It is the highest level of abstraction from a data perspective.
+The user observes and interacts with a Program using the
+ @ref{Program UI,,Program@tie{}UI}.
+
+@cindex Program, XML
+Programs contain a lot of metadata that is not in a convenience
+ human-readable (or modifiable) format,
+ some of which are redundant.
+Programs are ideally compiled from a @ref{Program XML,,Program@tie{}XML}
+ document.
+
+@menu
+* Program UI::
+* Program XML::
+* Document Metadata:: Document-level data that cannot be modified by
+ the client.
+@end menu
+
+
+@node Program UI
+@section Program UI
+@maintenance{
+ The @code{Ui} class,
+ in addition to @srcref{src/client/Client,Client} (@pxref{Client}),
+ represent the two monoliths of the system.
+ This mediates all UI-related tasks,
+ and still has far too many concerns with far too many
+ dependencies.
+ Code is to be extracted out of this class as it is touched.
+}
+
+
+@cindex Program, User Interface
+@cindex User Interface, Program
+The @dfn{Program UI} renders a @ref{Program} as a form.
+
+@cindex Step
+@cindex Group
+At the highest level,
+ steps are rendered in a tab-like manner,
+ above the main form content.
+A step contains groups,
+ which in turn contain elements such as questions.
+Groups are delimited in some manner defined by their style
+ (@pxref{Group Styles}).
+
+@cindex Question
+@cindex Question, Value Formatting
+@cindex Bucket, Updating
+Questions are rendered as form fields.
+Any time the respective @ref{Bucket} field is changed,
+ the form field is updated to reflect those changes,
+ after having first been formatted with the appropriate validator
+ (@pxref{Formatting Values}).
+When a question is changed by the user,
+ the value is expected to be propagated to the Bucket
+ (@pxref{Bucket Assignment}).
+
+@cindex Navigation Bar
+@cindex User Interface, Navigation Bar
+@cindex User Interface, Button Navigation
+Navigation between steps can be done via the
+ @dfn{Navigation Bar} above the step@tie{}content,
+ or using ``Go@tie{}Back'' and ``Continue'' buttons at the foot of the
+ step@tie{}content.
+
+@cindex Sidebar
+A @dfn{Sidebar} is rendered adjacent to the step content.
+It displays the name of the Program,
+ as well as configurable metadata (usually through the @samp{sidebar}
+ node of the @ref{Program XML,,Program@tie{}XML}).
+It also displays question help text (also configured through the XML)
+ and any error messages (@pxref{Error Handling}).
+
+@menu
+* Group Styles:: Different ways of displaying groups of questions to
+ the user.
+@end menu
+
+
+@node Group Styles
+@subsection Group Styles
+@refactor{
+ Some group styles still use jQuery;
+ they should be modified to use modern formatters and Liza DOM
+ abstractions (see @srcrefraw{src/ui/field}
+ and @srcrefraw{src/ui/styler}).}
+
+@cindex Group, Styling
+Groups support a number of @dfn{group styles} that determine how
+ they are delimited from other groups;
+ how the elements they contain are rendered and laid out;
+ and how multiple indexes are displayed, added, and removed.
+A list of available styles is detailed in @ref{t:group-styles}.
+
+@float Table, t:group-styles
+@multitable @columnfractions 0.15 0.65 0.10 0.10
+ @headitem Name @tab Description @tab Multi-Index? @tab Add/Remove Index?
+
+ @item @samp{default}
+ @tab
+ Groups are unstyled by default@mdash{
+ }they render elements as flat fields like a traditional form.
+ Only the first index of elements is rendered.
+ @tab@center N
+ @tab@center N
+
+ @item @samp{collapsetable}
+ @tab
+ Renders element label in the leftmost column like @samp{sidetable}.
+ Indexes are groups of rows delimited by headings,
+ which collapse the respective group of rows when clicked.
+ @tab@center Y
+ @tab@center Add
+
+ @item @samp{sidetable}
+ @tab
+ Renders elements as rows with label in the leftmost column rather
+ than the top row.
+ Each index is rendered as a column.
+ @tab@center Y
+ @tab@center Add
+
+ @item @samp{tabbedblock}
+ @tab
+ Each group is rendered as a block,
+ with each index rendered as a tab to the right of it.
+ Clicking a tab toggles the body content to the associated index.
+ Elements are rendered within the box.
+ @tab@center Y
+ @tab@center N
+
+ @item @samp{tabbed}
+ @tab
+ Like @samp{default},
+ but each index has a tab at the top of the group.
+ Clicking a tab toggles the body content to the associated index.
+ @tab@center Y
+ @tab@center Y
+
+ @item @samp{table}
+ @tab
+ A vanilla table with elements as columns,
+ their labels across the top row.
+ Each index is rendered in its own row.
+ @tab@center Y
+ @tab@center Y
+@end multitable
+@caption{Group styles and index support}
+@end float
+
+
+
+@node Program XML
+@section Program XML
+@helpwanted
+
+@menu
+* Specifying Predicates::
+@end menu
+
+
+@node Specifying Predicates
+@subsection Specifying Predicates
+
+Object predicates (@pxref{Predicate System}) are specified using the
+ @xmlattr{when} attribute of certain nodes.
+It must contain a string of references understood by the system
+ (see domain of discourse, @ref{Predicate System}),
+ all of which much match for the predicate to be true.
+
+@float Figure, f:pred-when
+@example
+ <question id="describe" type="noyes"
+ label="Any special notes for this location?" />
+
+ <question id="vacant_desc" type="textarea"
+ when="q:describe vacant property"
+ label="Show only when a vacant property with the
+ question 'describe' non-empty and non-zero" />
+@end example
+@caption{Using the @xmlattr{when} attribute}
+@end float
+
+In @ref{f:pred-when} above,
+ question @samp{vacant_desc} will be applicable when @emph{all} of
+ the values of @samp{vacant}, @samp{property},
+ and@tie{}@samp{q:describe} are true.@footnote{
+ @xref{Predicate System} for what ``true'' means for a particular
+ variable in the domain of discourse.}
+Within the context of the @progxml,
+ this concretely means that the classifications
+ @samp{vacant} and@tie{}@samp{property} are true,
+ and that the question @samp{describe} is answered ``yes''.
+It reads as a sentence:
+ ``@samp{vacant_desc}'' is applicable when we should @tie{}``describe
+ a vacant property''.
+
+
+
+@node Document Metadata
+@section Document Metadata
+@dfn{Document metadata} are metadata that describe certain aspects of the document;
+ they are stored adjacent to the bucket in @samp{meta}@tie{}on the
+ document root.@footnote{
+ Terminology note: ``document'' and ``quote'' are the same thing;
+ the latter is transitioning to the former for generality.}
+They should be used in place of a bucket field any time
+ the client has no business knowing about the data.
+The @samp{meta} record is called the @dfn{Metabucket}.
+
+@c don't use a dapi xref here; don't want to confuse the reader by
+@c directing them away from this section before they continue reading
+@tip{Metadata in the Metabucket should@tie{}@emph{not} be
+ directly populated by external systems@mdash{
+ }@dapi integration should be used instead (see below).}
+
+Metadata can be populated using any@tie{}@dapiref@mdash{
+ }return data populate the Metabucket in the same way that they
+ populate the Bucket.
+Definitions are stored in @code{meta.fields},
+ as shown in @ref{f:meta-fields}.
+
+@float Figure, f:meta-fields
+@example
+"fields":@{
+ ["string(name)": @{
+ "desc": "string",
+ "dapi": @{
+ "name": "string",
+ "map": @{
+ "string(dest field)": "string(source field)"
+ @}
+ @}
+ @}
+@}
+@end example
+@caption{Format of @code{meta.fields}.}
+@end float
+
+Further, a key-value mapping of all bucket fields that@mdash{
+ }when modified,
+ need to result in a metadata API@tie{}call@mdash{
+ }are stored in the @code{mapis}@tie{}object;
+ this is shown in @ref{f:mapis}.
+
+@float Figure, f:mapis
+@example
+"mapis":@{
+ ["string(field name)"]: [ "string(dapi name)", ... ]
+@}
+@end example
+@caption{Format of @code{mapis}.}
+@end float
diff --git a/doc/server.texi b/doc/server.texi
new file mode 100644
index 0000000..ca9d777
--- /dev/null
+++ b/doc/server.texi
@@ -0,0 +1,268 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Server
+@chapter Liza Server
+@maintenance{The @srcrefjs{server/daemon,Daemon} monolith and
+ @srcrefjs{server,Server},
+ among other things,
+ need refactoring.}
+
+@helpwanted{}
+
+@cindex Server
+The @dfn{server}@footnote{
+ @cindex Quote Server
+ Which may also be referenced as ``quote server'' in certain legacy
+ contexts,
+ referring to Liza's origin as an insurance rating system.}
+ is a RESTful service that serves as the HTTP server.
+It is designed to run under Node.js,
+ motivated by the benefits of sharing code with the@tie{}Client
+ (@pxref{Client}).
+The daemon is handled by the abstract @srcrefjs{server/daemon,Daemon}
+ monolith,
+ which requires that a concrete @code{#getEncryptionService}
+ method be defined by a subtype or trait.
+An example script to start the server is shown in @ref{f:server-start}.
+
+@cindex Encryption Service
+@tip{For local development,
+ or to avoid use of any encryption service,
+ use @srcrefjs{server/daemon,DevDaemon},
+ which uses a dummy encryption service.}
+
+@float Figure, f:server-start
+@example
+ const @{ Daemon @} = require( 'liza' ).server.daemon;
+ const port = 8080;
+ const log_priority = 10;
+
+ Daemon( port, log_priority ).start();
+@end example
+@caption{Starting the server daemon}
+@end float
+
+@cindex HTTP Server
+The HTTP server is managed by
+ @srcrefjs{server/daemon,http_server}.
+
+
+@menu
+* Requests:: Handling HTTP requests.
+* Posting Data:: Handling step saves and other posts.
+* Server-Side Data API Calls:: Accessing external resources on the server.
+* Encryption Service:: Managing sensitive data.
+@end menu
+
+
+
+@node Requests
+@section HTTP Requests
+@helpwanted{}
+
+@cindex Session
+@cindex PHPSESSID
+@cindex Memcache
+Each HTTP request produces a @srcrefjs{server/request,UserRequest}
+ associated with a @srcrefjs{server/request,UserSession}.
+Sessions are tightly coupled with PHP@footnote{
+ They don't have to be@mdash{}refactoring is needed.};
+ an existing PHP session is expected,
+ as identified by the @samp{PHPSESSID} cookie.
+Sessions are shared via Memcache
+ (see @srcrefjs{server/cache,ResilientMemcache}).@footnote{
+ Via a @url{https://secure.php.net/manual/en/memcached.sessions.php,memcache session handler}.}
+ If a session is not found (or is invalid),
+ an HTTP@tie{}@code{500} status code is returned and the
+ HTTP@tie{}request is aborted.
+
+@cindex Timeout
+@cindex Request timeout
+Requests are subject to a 120@tie{}second timeout,
+ after which the request will be served an HTTP@tie{}@code{408}
+ status code.
+Note that this @emph{does not stop background processing}@mdash{
+ }this timeout exists to prevent the user from hanging indefinitely.
+
+@cindex Long-running requests
+@tip{If a process intends to perform background processing for any length
+ of time (longer than a few seconds),
+ it should complete the request as quickly as possible and
+ use some other mechanism to report back progress
+ (e.g. polling).}
+
+The @srcrefjs{server/request,UserRequest} exposes raw request data with
+ minor processing.
+
+@table @strong
+ @item Path (@jsmethod{getUri})
+ The path component of the URI. The method name is unfortunate.
+
+ @item Query data (@jsmethod{getGetData})
+ Query string processed into a key/value object.
+ Despite the name,
+ this is also populated if non-GET requests contain query strings.
+
+ @item POST data (@jsmethod{getPostData})
+ POST data processed into an object as if it were a query string
+ (just as @jsmethod{getGetData}).
+ Since this requires data that is streamed asynchronously,
+ this method takes a callback that waits for all data to become
+ available;
+ if the data are already available,
+ it is immediately invoked with the processed POST data.
+
+ @item Cookies (@jsmethod{getCookies})
+ Cookies parsed into a key/value object.
+
+ @item Remote address (@jsmethod{getRemoteAddr})
+ IP address of the origin of the request.
+ If the server is behind a proxy that sets the
+ @samp{X-Forwarded-For} header,
+ it is used instead.
+
+ @item Host address (@jsmethod{getHostAddr})
+ Hostname of the server.
+ If the server is behind a proxy that sets the
+ @samp{X-Forwarded-Host} header,
+ it is used instead.
+
+ @item Origin (@jsmethod{getOrigin})
+ Origin of request.
+ Only available if at lease one of the @samp{Origin} or
+ @samp{Referer} headers are set.
+ This is useful mainly for determining the protocol and host while
+ behind a proxy.
+
+ @item User agent (@jsmethod{getUserAgent})
+ The user agent string of the request.
+
+ @item Session ID (@jsmethod{getSessionId})
+ The user's unique session id (@samp{PHPSESSID}).
+
+ @item Session ID name (@jsmethod{getSessionIdName})
+ The name of the cookie from which the session ID originated
+ (hard-coded to @samp{PHPSESSID}).
+@end table
+
+@todo{Document return format and writing response data.}
+
+
+
+@node Posting Data
+@section Posting Data
+@cindex Post
+@cindex Bucket diff
+@cindex Step save
+A diff of the bucket data (@pxref{Bucket Diff}) is posted to the
+ server on step@tie{}save.
+This operation is performed asynchronously@mdash{
+ }the client need not wait for the step to save before the next can
+ be requested.
+
+Since validations are shared between the server and the client
+ (@pxref{Validation}),
+ saving should only fail in exception situations.
+Should a failure occur,
+ the server will instruct the client to kick the user back to the
+ previous step (@dfn{kickback}).
+
+A step cannot be saved if it is locked;
+ such attempts will result in an error.
+
+To prevent a user from skipping steps,
+ the client may post only one step past the last step that has
+ successfully saved;
+ otherwise, the user is kicked back to the last step that was saved.
+
+Once those basic checks have passed,
+ the document is updated:
+
+@enumerate
+ @item
+ @cindex Data sanitization
+ The diff is first @dfn{sanitized} to strip out unknown fields,
+ internal fields posted by non-internal users,
+ and to filter fields on permitted characters;
+
+ @item
+ The sanitized diff is then applied to the existing bucket on the
+ document;
+
+ @item
+ @cindex Calculated values, server-side
+ Calculated values marked for storage (@pxref{Calculated Values}) are
+ re-calculated on the server (the values posted by the client have
+ already been discarded by the first step in this list);
+
+ @item
+ Server-side @dapi{} calls (@pxref{Data API}) are triggered using the
+ diff as input data and an empty bucket for response storage
+ (@pxref{Server-Side Data API Calls});
+
+ @item
+ @cindex Premium calculation date
+ The last premium calculation date is cleared (indicating that
+ premiums are no longer valid);@footnote{
+ This concept is tightly coupled with insurance;
+ it should be factored out at some point.}
+
+ @item
+ @cindex Encryption
+ Data marked as sensitive is encrypted and the ciphertext written to
+ the bucket in place of the plaintext (@pxref{Encryption Service});
+
+ @item
+ @cindex Top visited step
+ The current step is incremented and the @dfn{top visited
+ step}@tie{} is set to the larger of the incremented step or the
+ existing top visited step id; and then
+
+ @item
+ The new document state and bucket data are written to the database.
+@end enumerate
+
+
+
+@node Server-Side Data API Calls
+@section Server-Side Data API Calls
+@maintenance{This makes use of @srcrefjs{server/meta,DapiMetaSource}
+ to encapsulate the horrible API of @srcrefjs{dapi,DataApiManager};
+ the latter needs cleanup to remove the former.}
+
+@cindex Data API
+@cindex Document metadata
+Server-side @dapi{} calls (@pxref{Data API}) are triggered on
+ step save (@pxref{Posting Data}) and are handled much like they are
+ on the client.
+Such calls are made automatically only for document metadata.
+Results of sever-side calls are @emph{not} written to the bucket
+ and are therefore useful for data that the client should not be
+ permitted to modify;
+ it also allows data to be kept secret from the client.@footnote{
+ All bucket data is served to the client,
+ with the exception of internal fields if the user is non-internal.}
+
+@dapi{} results on the client can be mapped back to multiple bucket values;
+ the server, however, has serious concerns with how data are
+ propagated for data integrity and security reasons.
+Further,
+ document metadata can be structured,
+ unlike the Bucket which has a rigid matrix format (@pxref{Bucket}).
+Therefore,
+ the entire response is mapped into the parent field;
+ defined return values are used only for filtering.
+
+
+
+@node Encryption Service
+@section Encryption Service
+@helpwanted
diff --git a/doc/validation.texi b/doc/validation.texi
new file mode 100644
index 0000000..8ad0e29
--- /dev/null
+++ b/doc/validation.texi
@@ -0,0 +1,25 @@
+@c This document is part of the Liza Data Collection Framework manual.
+@c Copyright (C) 2017 R-T Specialty, LLC.
+@c
+@c Permission is granted to copy, distribute and/or modify this document
+@c under the terms of the GNU Free Documentation License, Version 1.3
+@c or any later version published by the Free Software Foundation;
+@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
+@c Texts. A copy of the license is included in the section entitled ``GNU
+@c Free Documentation License''.
+
+@node Validation
+@chapter Validation
+@helpwanted
+
+@menu
+* Formatting Values::
+@end menu
+
+
+@node Formatting Values
+@section Formatting Values
+
+@cindex Question
+@cindex Question, Value Formatting
+@helpwanted