diff options
author | Mike Gerwitz <gerwitzm@lovullo.com> | 2017-06-29 10:50:02 -0400 |
---|---|---|
committer | Mike Gerwitz <gerwitzm@lovullo.com> | 2017-06-29 10:50:02 -0400 |
commit | 35af4bdf191cec00d947acf0fdced62834109e3e (patch) | |
tree | df1e9404787e07f64b9690dd79e60933fe35cb32 /doc | |
parent | 3be28a7858b1fc62152050560fa6cf01d8dc991d (diff) | |
parent | 0c24e3d2807a3d5aa5f9c768f449c6ba2b7fc6dd (diff) | |
download | liza-35af4bdf191cec00d947acf0fdced62834109e3e.tar.gz liza-35af4bdf191cec00d947acf0fdced62834109e3e.tar.bz2 liza-35af4bdf191cec00d947acf0fdced62834109e3e.zip |
Server-side Data API support
The Data API stuff is pretty well coupled to the concept of a UI, and
the code is a mess that was just extracted from Program.js, so this
isn't the prettiest implementation. I hope that we'll have more time
for it in the somewhat-near future, since this is going to be core to
all server-side network requests for data.
Diffstat (limited to 'doc')
-rw-r--r-- | doc/Makefile.am | 5 | ||||
-rw-r--r-- | doc/assert.texi | 29 | ||||
-rw-r--r-- | doc/bucket.texi | 36 | ||||
-rw-r--r-- | doc/client.texi | 98 | ||||
-rw-r--r-- | doc/dapi.texi | 149 | ||||
-rw-r--r-- | doc/design.texi | 450 | ||||
-rw-r--r-- | doc/liza.texi | 16 | ||||
-rw-r--r-- | doc/macros.texi | 52 | ||||
-rw-r--r-- | doc/pred.texi | 74 | ||||
-rw-r--r-- | doc/program.texi | 282 | ||||
-rw-r--r-- | doc/server.texi | 268 | ||||
-rw-r--r-- | doc/validation.texi | 25 |
12 files changed, 1038 insertions, 446 deletions
diff --git a/doc/Makefile.am b/doc/Makefile.am index ccbd36a..508ed8b 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -21,7 +21,10 @@ MAKEINFOHTML = $(MAKEINFO) --html --css-include liza.css info_TEXINFOS = liza.texi -liza_TEXINFOS = design.texi macros.texi config.texi liza.css \ +liza_TEXINFOS = macros.texi config.texi liza.css \ + design.texi assert.texi bucket.texi client.texi \ + dapi.texi pred.texi program.texi server.texi \ + validation.texi \ diagram/event-graph.svg diagram/%.svg: diagram/%.dot diff --git a/doc/assert.texi b/doc/assert.texi new file mode 100644 index 0000000..ce16ce9 --- /dev/null +++ b/doc/assert.texi @@ -0,0 +1,29 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Assertions +@chapter Assertions +@maintstart +@footnote{ + Assertions are compiled from the Program XML (@pxref{Program XML}). + Rather than using a library, + it compiles a mess of largely duplicate code inline. + This system needs to be @emph{replaced}, + not modified. + + A replacement can either be in the form of a library (removing most + if not all code generation from the Program XML compiler), + or possibly compile into classifications and use the classification + system. + @emph{The latter option is preferred, + and would be more powerful with less maintenance.}} +@maintend + +@helpwanted diff --git a/doc/bucket.texi b/doc/bucket.texi new file mode 100644 index 0000000..0b4a810 --- /dev/null +++ b/doc/bucket.texi @@ -0,0 +1,36 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Bucket +@chapter Bucket +@helpwanted + +@menu +* Value Assignment:Bucket Assignment. Writing data to the Bucket. +* Bucket Diff:: Representing bucket changes. +* Calculated Values:: Dynamic data derived from other values. +@end menu + + +@c TODO +@node Bucket Assignment +@section Bucket Value Assignment +@helpwanted + + +@node Bucket Diff +@section Bucket Diff +@cindex Bucket diff +@helpwanted + + +@node Calculated Values +@section Calculated Values +@helpwanted diff --git a/doc/client.texi b/doc/client.texi new file mode 100644 index 0000000..8955264 --- /dev/null +++ b/doc/client.texi @@ -0,0 +1,98 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Client +@chapter Client +@maintstart +@footnote{ + The client is largely managed by a single class, + @srcref{src/client/Client.js, Client}, + which has grown out of control. + @code{Client} mediates essentially the entire system. + Code is to be extracted out of this class as it is touched. + + The other system mammoth is @code{Ui} (@pxref{Program UI}).} +@maintend + +@helpwanted + +@menu +* Error Handling:: +@end menu + + +@node Error Handling +@section Error Handling + +@maintenance{ + The complexity of this system and integration into legacy + layers has caused maintenance trouble in the past. + Each of the error checking layers need to be integrated + to reduce complexity.} + +@cindex Error +@cindex Failure +There are three layers of error checking:@footnote{ + Primarily for legacy reasons. + They are being consolodated as the system is touched.} + +@enumerate + @item Required field checking@mdash{ + }whether all required questions have been answered. + @item Type Validation@mdash{ + }verify that questions contain valid data according to their + declared type. @ref{Validation}. + @item Assertions@mdash{ + }arbitrary checks on data. + @ref{Assertions}. +@end enumerate + +@cindex Required Field +@cindex Field, Required +@cindex Field, Fixed +@cindex Error, Fixed +@cindex Error, Required +@cindex Fixed, Error +Required fields fail serially@mdash{ + }the system will notify the user of the required field, + and direct him/her to it (usually through scrolling). +A field is marked as @dfn{fixed} according to the rules + in @ref{Managing Error State}. + + +@menu +* Managing Error State:: Determining when failures should be marked as ``fixed'' +@end menu + + +@node Managing Error State +@subsection Managing Error State + +@cindex Failure Stack +@cindex Error Stack +Each failure caused by assertions is associated with a + @dfn{failure stack}. +The stack represents the trail of assertions that have run, + containing the ids of all values asserted against. +When any field or classification changes that is represented on the + failure stack, + the failure for the failed field associated with that failure stack + is cleared. + +@exnotice{ + If an assertion for some question @var{foo} first checked the value + of bucket field @var{bar}, + and within its failure checked the value @var{c:predicate}, + the failure stack would contain both of those ids. + If either @var{bar} or the @var{predicate} classification changed, + the question @var{foo} would have its error cleared.} + +Error state is managed by + @srcref{src/validate/ValidStateMonitor.js, ValidStateMonitor}. diff --git a/doc/dapi.texi b/doc/dapi.texi new file mode 100644 index 0000000..ca61c92 --- /dev/null +++ b/doc/dapi.texi @@ -0,0 +1,149 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Data API +@chapter Data API +@maintenance{This is a complex system with too much logic lying in + @srcrefjs{dapi,DataApiManager} (having been extracted + from its old home in @srcrefjs{program,Program} ).} + +@helpwanted{} + +The @dfn{Data API} is a declarative abstraction for accessing and + processing remote data (e.g. a RESTful service). +The name stems from how it is used@mdash{ + }to declare an remote API's inputs and outputs. + +This system is generally used indirectly through the @progxmlref{}.@footnote{ + @proguicxref{Data API}.} + +@tip{All interaction with this system should be had through the + @srcrefjs{dapi,DataApiManager}.} + +The @srcrefjs{dapi,DataApiManager} manages the entire operation@mdash{ + }from triggering the initial request, + to performing mapping, + to populating bucket data. +It takes only a @srcrefjs{dapi,DataApiFactory} and @dapi{} definitions. + +Definitions have the following schema:@footnote{ + There are poor design decisions that will likely persist + indefinitely because of integration with other systems, + so future extensions may be messy (especially in the case of + @samp{retvals}). +} + +@verbatim +{ + "type": "string", + "source": "string", + "method": "string", + "params": { + ["string(name)"]: { + "name": "string(name)", + "default": { + "type": "string", + "value": "string" + }, + ... + }, + }, + "retvals": [ "string", ... ], + "static": [ + { + ["string(param)"]: "string", + ... + }, + ... + ], + "static_nonempty": boolean, + "static_multiple": boolean +} +@end verbatim + +Each of the above fields are defined by: + +@table @code + @item type + Any type supported by @srcrefjs{dapi,DataApiFactory} (e.g. @samp{rest}). + + @item source + Type-specific source of data. + For e.g. @samp{rest}, this is a URI. + + @item method + Type-specific method for interacting with the API. + For e.g. @samp{rest}, this is an HTTP@tie{}method. + + @item params + Key-value mapping of input parameter names (as received by @samp{source}) + to their default values. + These inputs must be populated by the caller at the time of the request. + + @item retvals + Array of fields returned by the data source. + + @item static + Static values to prepend to the returned data. + This is often used for adding ``please select'' text, for example. + + @item static_nonempty + Whether statics should be added when there is return data; + Otherwise, + they will be added only if the response yields no results. + + @item static_multiple + Whether statics should be added only if multiple data are returned. + For example, + a ``please select'' is only useful if there is more than one + option for the user to select from. + When @samp{true}, + this has the convenient side-effect of auto-selecting the only + result. +@end table + +An example definition appears in @ref{f:dapi-ex} + +@float Figure, f:dapi-ex +@example + @{ + "type": "rest", + "source": "/foo/city", + "method": "post", + "params": @{ + "getVal": @{ + "name": "getVal", + "default": @{ + "type": "string", + "value": "getCityOptions" + @} + @}, + "zipcode": @{ + "name": "zipcode", + "default": @{ + "type": "ref", + "value": "" + @} + @} + @}, + "retvals": [ "city", "id", "state", "county", "country" ], + "static": [ @{ + "city": "(Please Select)", + "id": "", + "state": "", + "county": "", + "country": "" + @} ], + "static_nonempty": false, + "static_multiple": true + @}, +@end example +@caption{Example @dapi{} definition} +@end float diff --git a/doc/design.texi b/doc/design.texi index 7bde80d..d605fde 100644 --- a/doc/design.texi +++ b/doc/design.texi @@ -34,10 +34,6 @@ The main components of the system are: and provides hooks that drive the rest of the system. @xref{Bucket}. - @cindex Calculated Value - @item Calculated Values - A small sub-system for calculating bucket values from other values. - @cindex Client @item Client Basic logic for navigating between steps, @@ -47,6 +43,12 @@ The main components of the system are: etc. @xref{Client}. + @cindex Data API + @item Data API + Declarative abstraction for accessing and processing remote data + (e.g. a RESTful service). + @xref{Data API}. + @cindex Developer Dialog @item Developer Dialog Renders information about the system for debugging the client. @@ -91,6 +93,7 @@ The main components of the system are: and other types of processing. Code is shared with the client, ensuring identical behavior for appropriate behaviors. + @xref{Server}. @cindex Type Validation @cindex Validation, Type @@ -102,441 +105,4 @@ The main components of the system are: @xref{Validation}. @end table -More information about each can be found in their respective section. - - -@menu -* Assertions:: -* Bucket:: -* Client:: -* Predicate System:: -* Program:: -* Program UI:: -* Program XML:: -* Validation:: -@end menu - - - -@node Assertions -@section Assertions -@maintstart -@footnote{ - Assertions are compiled from the Program XML (@pxref{Program XML}). - Rather than using a library, - it compiles a mess of largely duplicate code inline. - This system needs to be @emph{replaced}, - not modified. - - A replacement can either be in the form of a library (removing most - if not all code generation from the Program XML compiler), - or possibly compile into classifications and use the classification - system. - @emph{The latter option is preferred, - and would be more powerful with less maintenance.}} -@maintend - -@helpwanted - - - -@node Bucket -@section Bucket -@helpwanted - -@menu -* Value Assignment:Bucket Assignment. Writing data to the Bucket. -@end menu - - -@c TODO -@node Bucket Assignment -@subsection Bucket Value Assignment -@helpwanted - - - -@node Client -@section Client -@maintstart -@footnote{ - The client is largely managed by a single class, - @srcref{src/client/Client.js, Client}, - which has grown out of control. - @code{Client} mediates essentially the entire system. - Code is to be extracted out of this class as it is touched. - - The other system mammoth is @code{Ui} (@pxref{Program UI}).} -@maintend - -@helpwanted - -@menu -* Error Handling:: -@end menu - - -@node Error Handling -@subsection Error Handling - -@maintenance{ - The complexity of this system and integration into legacy - layers has caused maintenance trouble in the past. - Each of the error checking layers need to be integrated - to reduce complexity.} - -@cindex Error -@cindex Failure -There are three layers of error checking:@footnote{ - Primarily for legacy reasons. - They are being consolodated as the system is touched.} - -@enumerate - @item Required field checking@mdash{ - }whether all required questions have been answered. - @item Type Validation@mdash{ - }verify that questions contain valid data according to their - declared type. @ref{Validation}. - @item Assertions@mdash{ - }arbitrary checks on data. - @ref{Assertions}. -@end enumerate - -@cindex Required Field -@cindex Field, Required -@cindex Field, Fixed -@cindex Error, Fixed -@cindex Error, Required -@cindex Fixed, Error -Required fields fail serially@mdash{ - }the system will notify the user of the required field, - and direct him/her to it (usually through scrolling). -A field is marked as @dfn{fixed} according to the rules - in @ref{Managing Error State}. - - -@menu -* Managing Error State:: Determining when failures should be marked as ``fixed'' -@end menu - - -@node Managing Error State -@subsubsection Managing Error State - -@cindex Failure Stack -@cindex Error Stack -Each failure caused by assertions is associated with a - @dfn{failure stack}. -The stack represents the trail of assertions that have run, - containing the ids of all values asserted against. -When any field or classification changes that is represented on the - failure stack, - the failure for the failed field associated with that failure stack - is cleared. - -@exnotice{ - If an assertion for some question @var{foo} first checked the value - of bucket field @var{bar}, - and within its failure checked the value @var{c:predicate}, - the failure stack would contain both of those ids. - If either @var{bar} or the @var{predicate} classification changed, - the question @var{foo} would have its error cleared.} - -Error state is managed by - @srcref{src/validate/ValidStateMonitor.js, ValidStateMonitor}. - - - -@node Predicate System -@section Predicate System -@maintstart -@footnote{ - New programs (using the old incarnation of TAME) use the classifier - embedded into the rater by TAME. - Old ones, however, still use the @dfn{Global Classifier}. - This system isn't as well tested as TAME's@mdash{ - }which needs to work properly for the sake of calculating premium@mdash{ - }and has suffered from a number of bugs in the past. - - The solution is to migrate all programs to TAME and remove that old - code.} -@maintend - -@tip{ - For a practical application of these concepts, see its use in the - Program@tie{}XML (@pxref{Specifying Predicates}).} - -@cindex Predicate -@cindex Classifier -@cindex Applicability -@cindex Domain of discourse, Predicate -The @dfn{predicate system} determines the @dfn{applicability} of - certain objects (like questions and assertions) by associating them - with predicates. -The domain of discourse (variables which may be quantified) is listed - in @ref{t:predicate-dod}. - -What it means for some object to be applicable depends on the context. - -@float Table, t:predicate-dod -@multitable @columnfractions 0.25 0.10 0.65 - @headitem Type @tab Prefix @tab Description - - @item Classifications - @tab @emph{None} - @tab Results of applying an external @dfn{classifier} to the bucket - (@pxref{Bucket}). - - @item Bucket Truth Predicate - @tab @code{q:} - @tab - Whether the given name in the bucket (@pxref{Bucket}) is - non-empty and non-zero. - The prefix @samp{q:} refers to its most common use - case---questions (@pxref{Program UI,,Program@tie{}UI}). -@end multitable -@caption{Predicate system domain of discourse} -@end float - -This system is limited to universal quantification over the domain of - discourse. -For other quantifiers and higher-order logic, - defer to one of the systems that contributes to the domain of - discourse, - like the classifier.@footnote{ - This is usually TAME. - The Program XML also supports inline classifications - with TAME's syntax (@pxref{Specifying Predicates}).} - -Predicates are usually specified in the Program XML - (@pxref{Specifying Predicates}) and compiled into the program - (@pxref{Program}). - - - -@node Program -@section Program -@maintstart -@footnote{ - The @code{Program} class was one of the first prototypes created, - and has evolved poorly with the rest of the system. - It is the base class for all compiled programs, - and it glues together too many other systems with a terrible - API and little to no encapsulation. - - With that said, it is one of the least touched classes (thus its - state); developers rarely have the need to touch @code{Program}.} -@maintend - - -@cindex Program -The @dfn{Program} is a declarative representation of an entire system. -It is the highest level of abstraction from a data perspective. -The user observes and interacts with a Program using the - @ref{Program UI,,Program@tie{}UI}. - -@cindex Program, XML -Programs contain a lot of metadata that is not in a convenience - human-readable (or modifiable) format, - some of which are redundant. -Programs are ideally compiled from a @ref{Program XML,,Program@tie{}XML} - document. - - - -@node Program UI -@section Program UI -@maintenance{ - The @code{Ui} class, - in addition to @srcref{src/client/Client,Client} (@pxref{Client}), - represent the two monoliths of the system. - This mediates all UI-related tasks, - and still has far too many concerns with far too many - dependencies. - Code is to be extracted out of this class as it is touched. -} - - -@cindex Program, User Interface -@cindex User Interface, Program -The @dfn{Program UI} renders a @ref{Program} as a form. - -@cindex Step -@cindex Group -At the highest level, - steps are rendered in a tab-like manner, - above the main form content. -A step contains groups, - which in turn contain elements such as questions. -Groups are delimited in some manner defined by their style - (@pxref{Group Styles}). - -@cindex Question -@cindex Question, Value Formatting -@cindex Bucket, Updating -Questions are rendered as form fields. -Any time the respective @ref{Bucket} field is changed, - the form field is updated to reflect those changes, - after having first been formatted with the appropriate validator - (@pxref{Formatting Values}). -When a question is changed by the user, - the value is expected to be propagated to the Bucket - (@pxref{Bucket Assignment}). - -@cindex Navigation Bar -@cindex User Interface, Navigation Bar -@cindex User Interface, Button Navigation -Navigation between steps can be done via the - @dfn{Navigation Bar} above the step@tie{}content, - or using ``Go@tie{}Back'' and ``Continue'' buttons at the foot of the - step@tie{}content. - -@cindex Sidebar -A @dfn{Sidebar} is rendered adjacent to the step content. -It displays the name of the Program, - as well as configurable metadata (usually through the @samp{sidebar} - node of the @ref{Program XML,,Program@tie{}XML}). -It also displays question help text (also configured through the XML) - and any error messages (@pxref{Error Handling}). - -@menu -* Group Styles:: Different ways of displaying groups of questions to - the user. -@end menu - - -@node Group Styles -@subsection Group Styles -@refactor{ - Some group styles still use jQuery; - they should be modified to use modern formatters and Liza DOM - abstractions (see @srcrefraw{src/ui/field} - and @srcrefraw{src/ui/styler}).} - -@cindex Group, Styling -Groups support a number of @dfn{group styles} that determine how - they are delimited from other groups; - how the elements they contain are rendered and laid out; - and how multiple indexes are displayed, added, and removed. -A list of available styles is detailed in @ref{t:group-styles}. - -@float Table, t:group-styles -@multitable @columnfractions 0.15 0.65 0.10 0.10 - @headitem Name @tab Description @tab Multi-Index? @tab Add/Remove Index? - - @item @samp{default} - @tab - Groups are unstyled by default@mdash{ - }they render elements as flat fields like a traditional form. - Only the first index of elements is rendered. - @tab@center N - @tab@center N - - @item @samp{collapsetable} - @tab - Renders element label in the leftmost column like @samp{sidetable}. - Indexes are groups of rows delimited by headings, - which collapse the respective group of rows when clicked. - @tab@center Y - @tab@center Add - - @item @samp{sidetable} - @tab - Renders elements as rows with label in the leftmost column rather - than the top row. - Each index is rendered as a column. - @tab@center Y - @tab@center Add - - @item @samp{tabbedblock} - @tab - Each group is rendered as a block, - with each index rendered as a tab to the right of it. - Clicking a tab toggles the body content to the associated index. - Elements are rendered within the box. - @tab@center Y - @tab@center N - - @item @samp{tabbed} - @tab - Like @samp{default}, - but each index has a tab at the top of the group. - Clicking a tab toggles the body content to the associated index. - @tab@center Y - @tab@center Y - - @item @samp{table} - @tab - A vanilla table with elements as columns, - their labels across the top row. - Each index is rendered in its own row. - @tab@center Y - @tab@center Y -@end multitable -@caption{Group styles and index support} -@end float - - - -@node Program XML -@section Program XML -@helpwanted - -@menu -* Specifying Predicates:: -@end menu - - -@node Specifying Predicates -@subsection Specifying Predicates - -Object predicates (@pxref{Predicate System}) are specified using the - @xmlattr{when} attribute of certain nodes. -It must contain a string of references understood by the system - (see domain of discourse, @ref{Predicate System}), - all of which much match for the predicate to be true. - -@float Figure, f:pred-when -@example - <question id="describe" type="noyes" - label="Any special notes for this location?" /> - - <question id="vacant_desc" type="textarea" - when="q:describe vacant property" - label="Show only when a vacant property with the - question 'describe' non-empty and non-zero" /> -@end example -@caption{Using the @xmlattr{when} attribute} -@end float - -In @ref{f:pred-when} above, - question @samp{vacant_desc} will be applicable when @emph{all} of - the values of @samp{vacant}, @samp{property}, - and@tie{}@samp{q:describe} are true.@footnote{ - @xref{Predicate System} for what ``true'' means for a particular - variable in the domain of discourse.} -Within the context of the @progxml, - this concretely means that the classifications - @samp{vacant} and@tie{}@samp{property} are true, - and that the question @samp{describe} is answered ``yes''. -It reads as a sentence: - ``@samp{vacant_desc}'' is applicable when we should @tie{}``describe - a vacant property''. - - - -@node Validation -@section Validation -@helpwanted - -@menu -* Formatting Values:: -@end menu - - -@node Formatting Values -@subsection Formatting Values - -@cindex Question -@cindex Question, Value Formatting -@helpwanted +More information about each can be found in their respective chapter/section. diff --git a/doc/liza.texi b/doc/liza.texi index d091cd9..c8bd87e 100644 --- a/doc/liza.texi +++ b/doc/liza.texi @@ -76,6 +76,14 @@ Free Documentation License''. @menu * Design and Architecture:Design. Design and structure of framework +* Assertions:: +* Bucket:: +* Client:: +* Data API:: +* Predicate System:: +* Program:: +* Server:: +* Validation:: * License:: Document License * Concept Index:: @ifset DEVNOTES @@ -89,6 +97,14 @@ Free Documentation License''. @end ifnottex @include design.texi +@include assert.texi +@include bucket.texi +@include client.texi +@include dapi.texi +@include pred.texi +@include program.texi +@include server.texi +@include validation.texi @include license.texi @node Concept Index diff --git a/doc/macros.texi b/doc/macros.texi index db725ba..e5a283c 100644 --- a/doc/macros.texi +++ b/doc/macros.texi @@ -66,12 +66,17 @@ @end ifset @end macro +@c documentation TODO +@macro todo{text} +@devnotice{TODO: \text\} +@end macro + @c indicate that help is needed to produce docs @macro helpwanted{} @cindex TODO, Missing Docs @dnindex Missing Docs -@notice{There's nothing here yet. Maybe you can help?} +@notice{There isn't much here yet. Maybe you can help?} @end macro @@ -140,11 +145,15 @@ This system has maintenance concerns. @c the path to the file @ifset SRCURI @macro srcref{path, display} -@url{@value{SRCURI}/\path\, @code{\display\}} +@url{@value{SRCURI}/\path\, @file{\display\}} @end macro @macro srcrefraw{path} -@url{@value{SRCURI}/\path\, @code{\path\}} +@url{@value{SRCURI}/\path\, @file{\path\}} +@end macro + +@macro srcrefjs{base,module} +@srcref{src/\base\/\module\.js, \module\} @end macro @end ifset @@ -156,6 +165,12 @@ This system has maintenance concerns. @macro srcrefraw{path} @file{\path\} @end macro + +@c intended to display a name without JS, +@c so just do that rather than the actual path +@macro srcrefjs{base,path} +@srcrefraw{\base\/\path\} +@end macro @end ifclear @@ -169,7 +184,38 @@ This system has maintenance concerns. @end macro +@c JS formatting +@macro jsmethod{name} +@code{#\name\} +@end macro + + @c text to avoid repeated e.g. ties and other formatting @macro progxml Program@tie{}XML @end macro + +@macro progxmlref +@ref{Program XML,,Program@tie{}XML} +@end macro + +@macro dapi +Data@tie{}API +@end macro + +@macro dapiref +@dapi (@pxref{Data API,,Data@tie{}API}) +@end macro + +@c todo: link to reference directly +@macro proguicref{ref} +`\ref\' @proguicrefsuffix +@end macro + +@macro proguicxref{ref} +See `\ref\' @proguicrefsuffix +@end macro + +@macro proguicrefsuffix{} +in the Liza Program@tie{}UI Compiler manual +@end macro diff --git a/doc/pred.texi b/doc/pred.texi new file mode 100644 index 0000000..8ee01eb --- /dev/null +++ b/doc/pred.texi @@ -0,0 +1,74 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Predicate System +@chapter Predicate System +@maintstart +@footnote{ + New programs (using the old incarnation of TAME) use the classifier + embedded into the rater by TAME. + Old ones, however, still use the @dfn{Global Classifier}. + This system isn't as well tested as TAME's@mdash{ + }which needs to work properly for the sake of calculating premium@mdash{ + }and has suffered from a number of bugs in the past. + + The solution is to migrate all programs to TAME and remove that old + code.} +@maintend + +@tip{ + For a practical application of these concepts, see its use in the + Program@tie{}XML (@pxref{Specifying Predicates}).} + +@cindex Predicate +@cindex Classifier +@cindex Applicability +@cindex Domain of discourse, Predicate +The @dfn{predicate system} determines the @dfn{applicability} of + certain objects (like questions and assertions) by associating them + with predicates. +The domain of discourse (variables which may be quantified) is listed + in @ref{t:predicate-dod}. + +What it means for some object to be applicable depends on the context. + +@float Table, t:predicate-dod +@multitable @columnfractions 0.25 0.10 0.65 + @headitem Type @tab Prefix @tab Description + + @item Classifications + @tab @emph{None} + @tab Results of applying an external @dfn{classifier} to the bucket + (@pxref{Bucket}). + + @item Bucket Truth Predicate + @tab @code{q:} + @tab + Whether the given name in the bucket (@pxref{Bucket}) is + non-empty and non-zero. + The prefix @samp{q:} refers to its most common use + case---questions (@pxref{Program UI,,Program@tie{}UI}). +@end multitable +@caption{Predicate system domain of discourse} +@end float + +This system is limited to universal quantification over the domain of + discourse. +For other quantifiers and higher-order logic, + defer to one of the systems that contributes to the domain of + discourse, + like the classifier.@footnote{ + This is usually TAME. + The Program XML also supports inline classifications + with TAME's syntax (@pxref{Specifying Predicates}).} + +Predicates are usually specified in the Program XML + (@pxref{Specifying Predicates}) and compiled into the program + (@pxref{Program}). diff --git a/doc/program.texi b/doc/program.texi new file mode 100644 index 0000000..94a1e12 --- /dev/null +++ b/doc/program.texi @@ -0,0 +1,282 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + + +@node Program +@chapter Program +@maintstart +@footnote{ + The @code{Program} class was one of the first prototypes created, + and has evolved poorly with the rest of the system. + It is the base class for all compiled programs, + and it glues together too many other systems with a terrible + API and little to no encapsulation. + + With that said, it is one of the least touched classes (thus its + state); developers rarely have the need to touch @code{Program}.} +@maintend + + +@cindex Program +The @dfn{Program} is a declarative representation of an entire system. +It is the highest level of abstraction from a data perspective. +The user observes and interacts with a Program using the + @ref{Program UI,,Program@tie{}UI}. + +@cindex Program, XML +Programs contain a lot of metadata that is not in a convenience + human-readable (or modifiable) format, + some of which are redundant. +Programs are ideally compiled from a @ref{Program XML,,Program@tie{}XML} + document. + +@menu +* Program UI:: +* Program XML:: +* Document Metadata:: Document-level data that cannot be modified by + the client. +@end menu + + +@node Program UI +@section Program UI +@maintenance{ + The @code{Ui} class, + in addition to @srcref{src/client/Client,Client} (@pxref{Client}), + represent the two monoliths of the system. + This mediates all UI-related tasks, + and still has far too many concerns with far too many + dependencies. + Code is to be extracted out of this class as it is touched. +} + + +@cindex Program, User Interface +@cindex User Interface, Program +The @dfn{Program UI} renders a @ref{Program} as a form. + +@cindex Step +@cindex Group +At the highest level, + steps are rendered in a tab-like manner, + above the main form content. +A step contains groups, + which in turn contain elements such as questions. +Groups are delimited in some manner defined by their style + (@pxref{Group Styles}). + +@cindex Question +@cindex Question, Value Formatting +@cindex Bucket, Updating +Questions are rendered as form fields. +Any time the respective @ref{Bucket} field is changed, + the form field is updated to reflect those changes, + after having first been formatted with the appropriate validator + (@pxref{Formatting Values}). +When a question is changed by the user, + the value is expected to be propagated to the Bucket + (@pxref{Bucket Assignment}). + +@cindex Navigation Bar +@cindex User Interface, Navigation Bar +@cindex User Interface, Button Navigation +Navigation between steps can be done via the + @dfn{Navigation Bar} above the step@tie{}content, + or using ``Go@tie{}Back'' and ``Continue'' buttons at the foot of the + step@tie{}content. + +@cindex Sidebar +A @dfn{Sidebar} is rendered adjacent to the step content. +It displays the name of the Program, + as well as configurable metadata (usually through the @samp{sidebar} + node of the @ref{Program XML,,Program@tie{}XML}). +It also displays question help text (also configured through the XML) + and any error messages (@pxref{Error Handling}). + +@menu +* Group Styles:: Different ways of displaying groups of questions to + the user. +@end menu + + +@node Group Styles +@subsection Group Styles +@refactor{ + Some group styles still use jQuery; + they should be modified to use modern formatters and Liza DOM + abstractions (see @srcrefraw{src/ui/field} + and @srcrefraw{src/ui/styler}).} + +@cindex Group, Styling +Groups support a number of @dfn{group styles} that determine how + they are delimited from other groups; + how the elements they contain are rendered and laid out; + and how multiple indexes are displayed, added, and removed. +A list of available styles is detailed in @ref{t:group-styles}. + +@float Table, t:group-styles +@multitable @columnfractions 0.15 0.65 0.10 0.10 + @headitem Name @tab Description @tab Multi-Index? @tab Add/Remove Index? + + @item @samp{default} + @tab + Groups are unstyled by default@mdash{ + }they render elements as flat fields like a traditional form. + Only the first index of elements is rendered. + @tab@center N + @tab@center N + + @item @samp{collapsetable} + @tab + Renders element label in the leftmost column like @samp{sidetable}. + Indexes are groups of rows delimited by headings, + which collapse the respective group of rows when clicked. + @tab@center Y + @tab@center Add + + @item @samp{sidetable} + @tab + Renders elements as rows with label in the leftmost column rather + than the top row. + Each index is rendered as a column. + @tab@center Y + @tab@center Add + + @item @samp{tabbedblock} + @tab + Each group is rendered as a block, + with each index rendered as a tab to the right of it. + Clicking a tab toggles the body content to the associated index. + Elements are rendered within the box. + @tab@center Y + @tab@center N + + @item @samp{tabbed} + @tab + Like @samp{default}, + but each index has a tab at the top of the group. + Clicking a tab toggles the body content to the associated index. + @tab@center Y + @tab@center Y + + @item @samp{table} + @tab + A vanilla table with elements as columns, + their labels across the top row. + Each index is rendered in its own row. + @tab@center Y + @tab@center Y +@end multitable +@caption{Group styles and index support} +@end float + + + +@node Program XML +@section Program XML +@helpwanted + +@menu +* Specifying Predicates:: +@end menu + + +@node Specifying Predicates +@subsection Specifying Predicates + +Object predicates (@pxref{Predicate System}) are specified using the + @xmlattr{when} attribute of certain nodes. +It must contain a string of references understood by the system + (see domain of discourse, @ref{Predicate System}), + all of which much match for the predicate to be true. + +@float Figure, f:pred-when +@example + <question id="describe" type="noyes" + label="Any special notes for this location?" /> + + <question id="vacant_desc" type="textarea" + when="q:describe vacant property" + label="Show only when a vacant property with the + question 'describe' non-empty and non-zero" /> +@end example +@caption{Using the @xmlattr{when} attribute} +@end float + +In @ref{f:pred-when} above, + question @samp{vacant_desc} will be applicable when @emph{all} of + the values of @samp{vacant}, @samp{property}, + and@tie{}@samp{q:describe} are true.@footnote{ + @xref{Predicate System} for what ``true'' means for a particular + variable in the domain of discourse.} +Within the context of the @progxml, + this concretely means that the classifications + @samp{vacant} and@tie{}@samp{property} are true, + and that the question @samp{describe} is answered ``yes''. +It reads as a sentence: + ``@samp{vacant_desc}'' is applicable when we should @tie{}``describe + a vacant property''. + + + +@node Document Metadata +@section Document Metadata +@dfn{Document metadata} are metadata that describe certain aspects of the document; + they are stored adjacent to the bucket in @samp{meta}@tie{}on the + document root.@footnote{ + Terminology note: ``document'' and ``quote'' are the same thing; + the latter is transitioning to the former for generality.} +They should be used in place of a bucket field any time + the client has no business knowing about the data. +The @samp{meta} record is called the @dfn{Metabucket}. + +@c don't use a dapi xref here; don't want to confuse the reader by +@c directing them away from this section before they continue reading +@tip{Metadata in the Metabucket should@tie{}@emph{not} be + directly populated by external systems@mdash{ + }@dapi integration should be used instead (see below).} + +Metadata can be populated using any@tie{}@dapiref@mdash{ + }return data populate the Metabucket in the same way that they + populate the Bucket. +Definitions are stored in @code{meta.fields}, + as shown in @ref{f:meta-fields}. + +@float Figure, f:meta-fields +@example +"fields":@{ + ["string(name)": @{ + "desc": "string", + "dapi": @{ + "name": "string", + "map": @{ + "string(dest field)": "string(source field)" + @} + @} + @} +@} +@end example +@caption{Format of @code{meta.fields}.} +@end float + +Further, a key-value mapping of all bucket fields that@mdash{ + }when modified, + need to result in a metadata API@tie{}call@mdash{ + }are stored in the @code{mapis}@tie{}object; + this is shown in @ref{f:mapis}. + +@float Figure, f:mapis +@example +"mapis":@{ + ["string(field name)"]: [ "string(dapi name)", ... ] +@} +@end example +@caption{Format of @code{mapis}.} +@end float diff --git a/doc/server.texi b/doc/server.texi new file mode 100644 index 0000000..ca9d777 --- /dev/null +++ b/doc/server.texi @@ -0,0 +1,268 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Server +@chapter Liza Server +@maintenance{The @srcrefjs{server/daemon,Daemon} monolith and + @srcrefjs{server,Server}, + among other things, + need refactoring.} + +@helpwanted{} + +@cindex Server +The @dfn{server}@footnote{ + @cindex Quote Server + Which may also be referenced as ``quote server'' in certain legacy + contexts, + referring to Liza's origin as an insurance rating system.} + is a RESTful service that serves as the HTTP server. +It is designed to run under Node.js, + motivated by the benefits of sharing code with the@tie{}Client + (@pxref{Client}). +The daemon is handled by the abstract @srcrefjs{server/daemon,Daemon} + monolith, + which requires that a concrete @code{#getEncryptionService} + method be defined by a subtype or trait. +An example script to start the server is shown in @ref{f:server-start}. + +@cindex Encryption Service +@tip{For local development, + or to avoid use of any encryption service, + use @srcrefjs{server/daemon,DevDaemon}, + which uses a dummy encryption service.} + +@float Figure, f:server-start +@example + const @{ Daemon @} = require( 'liza' ).server.daemon; + const port = 8080; + const log_priority = 10; + + Daemon( port, log_priority ).start(); +@end example +@caption{Starting the server daemon} +@end float + +@cindex HTTP Server +The HTTP server is managed by + @srcrefjs{server/daemon,http_server}. + + +@menu +* Requests:: Handling HTTP requests. +* Posting Data:: Handling step saves and other posts. +* Server-Side Data API Calls:: Accessing external resources on the server. +* Encryption Service:: Managing sensitive data. +@end menu + + + +@node Requests +@section HTTP Requests +@helpwanted{} + +@cindex Session +@cindex PHPSESSID +@cindex Memcache +Each HTTP request produces a @srcrefjs{server/request,UserRequest} + associated with a @srcrefjs{server/request,UserSession}. +Sessions are tightly coupled with PHP@footnote{ + They don't have to be@mdash{}refactoring is needed.}; + an existing PHP session is expected, + as identified by the @samp{PHPSESSID} cookie. +Sessions are shared via Memcache + (see @srcrefjs{server/cache,ResilientMemcache}).@footnote{ + Via a @url{https://secure.php.net/manual/en/memcached.sessions.php,memcache session handler}.} + If a session is not found (or is invalid), + an HTTP@tie{}@code{500} status code is returned and the + HTTP@tie{}request is aborted. + +@cindex Timeout +@cindex Request timeout +Requests are subject to a 120@tie{}second timeout, + after which the request will be served an HTTP@tie{}@code{408} + status code. +Note that this @emph{does not stop background processing}@mdash{ + }this timeout exists to prevent the user from hanging indefinitely. + +@cindex Long-running requests +@tip{If a process intends to perform background processing for any length + of time (longer than a few seconds), + it should complete the request as quickly as possible and + use some other mechanism to report back progress + (e.g. polling).} + +The @srcrefjs{server/request,UserRequest} exposes raw request data with + minor processing. + +@table @strong + @item Path (@jsmethod{getUri}) + The path component of the URI. The method name is unfortunate. + + @item Query data (@jsmethod{getGetData}) + Query string processed into a key/value object. + Despite the name, + this is also populated if non-GET requests contain query strings. + + @item POST data (@jsmethod{getPostData}) + POST data processed into an object as if it were a query string + (just as @jsmethod{getGetData}). + Since this requires data that is streamed asynchronously, + this method takes a callback that waits for all data to become + available; + if the data are already available, + it is immediately invoked with the processed POST data. + + @item Cookies (@jsmethod{getCookies}) + Cookies parsed into a key/value object. + + @item Remote address (@jsmethod{getRemoteAddr}) + IP address of the origin of the request. + If the server is behind a proxy that sets the + @samp{X-Forwarded-For} header, + it is used instead. + + @item Host address (@jsmethod{getHostAddr}) + Hostname of the server. + If the server is behind a proxy that sets the + @samp{X-Forwarded-Host} header, + it is used instead. + + @item Origin (@jsmethod{getOrigin}) + Origin of request. + Only available if at lease one of the @samp{Origin} or + @samp{Referer} headers are set. + This is useful mainly for determining the protocol and host while + behind a proxy. + + @item User agent (@jsmethod{getUserAgent}) + The user agent string of the request. + + @item Session ID (@jsmethod{getSessionId}) + The user's unique session id (@samp{PHPSESSID}). + + @item Session ID name (@jsmethod{getSessionIdName}) + The name of the cookie from which the session ID originated + (hard-coded to @samp{PHPSESSID}). +@end table + +@todo{Document return format and writing response data.} + + + +@node Posting Data +@section Posting Data +@cindex Post +@cindex Bucket diff +@cindex Step save +A diff of the bucket data (@pxref{Bucket Diff}) is posted to the + server on step@tie{}save. +This operation is performed asynchronously@mdash{ + }the client need not wait for the step to save before the next can + be requested. + +Since validations are shared between the server and the client + (@pxref{Validation}), + saving should only fail in exception situations. +Should a failure occur, + the server will instruct the client to kick the user back to the + previous step (@dfn{kickback}). + +A step cannot be saved if it is locked; + such attempts will result in an error. + +To prevent a user from skipping steps, + the client may post only one step past the last step that has + successfully saved; + otherwise, the user is kicked back to the last step that was saved. + +Once those basic checks have passed, + the document is updated: + +@enumerate + @item + @cindex Data sanitization + The diff is first @dfn{sanitized} to strip out unknown fields, + internal fields posted by non-internal users, + and to filter fields on permitted characters; + + @item + The sanitized diff is then applied to the existing bucket on the + document; + + @item + @cindex Calculated values, server-side + Calculated values marked for storage (@pxref{Calculated Values}) are + re-calculated on the server (the values posted by the client have + already been discarded by the first step in this list); + + @item + Server-side @dapi{} calls (@pxref{Data API}) are triggered using the + diff as input data and an empty bucket for response storage + (@pxref{Server-Side Data API Calls}); + + @item + @cindex Premium calculation date + The last premium calculation date is cleared (indicating that + premiums are no longer valid);@footnote{ + This concept is tightly coupled with insurance; + it should be factored out at some point.} + + @item + @cindex Encryption + Data marked as sensitive is encrypted and the ciphertext written to + the bucket in place of the plaintext (@pxref{Encryption Service}); + + @item + @cindex Top visited step + The current step is incremented and the @dfn{top visited + step}@tie{} is set to the larger of the incremented step or the + existing top visited step id; and then + + @item + The new document state and bucket data are written to the database. +@end enumerate + + + +@node Server-Side Data API Calls +@section Server-Side Data API Calls +@maintenance{This makes use of @srcrefjs{server/meta,DapiMetaSource} + to encapsulate the horrible API of @srcrefjs{dapi,DataApiManager}; + the latter needs cleanup to remove the former.} + +@cindex Data API +@cindex Document metadata +Server-side @dapi{} calls (@pxref{Data API}) are triggered on + step save (@pxref{Posting Data}) and are handled much like they are + on the client. +Such calls are made automatically only for document metadata. +Results of sever-side calls are @emph{not} written to the bucket + and are therefore useful for data that the client should not be + permitted to modify; + it also allows data to be kept secret from the client.@footnote{ + All bucket data is served to the client, + with the exception of internal fields if the user is non-internal.} + +@dapi{} results on the client can be mapped back to multiple bucket values; + the server, however, has serious concerns with how data are + propagated for data integrity and security reasons. +Further, + document metadata can be structured, + unlike the Bucket which has a rigid matrix format (@pxref{Bucket}). +Therefore, + the entire response is mapped into the parent field; + defined return values are used only for filtering. + + + +@node Encryption Service +@section Encryption Service +@helpwanted diff --git a/doc/validation.texi b/doc/validation.texi new file mode 100644 index 0000000..8ad0e29 --- /dev/null +++ b/doc/validation.texi @@ -0,0 +1,25 @@ +@c This document is part of the Liza Data Collection Framework manual. +@c Copyright (C) 2017 R-T Specialty, LLC. +@c +@c Permission is granted to copy, distribute and/or modify this document +@c under the terms of the GNU Free Documentation License, Version 1.3 +@c or any later version published by the Free Software Foundation; +@c with no Invariant Sections, no Front-Cover Texts, and no Back-Cover +@c Texts. A copy of the license is included in the section entitled ``GNU +@c Free Documentation License''. + +@node Validation +@chapter Validation +@helpwanted + +@menu +* Formatting Values:: +@end menu + + +@node Formatting Values +@section Formatting Values + +@cindex Question +@cindex Question, Value Formatting +@helpwanted |